Identifying indolent and aggressive prostate cancers is a critical problem for optimal treatment. The existing approaches of prostate cancer detection are facing challenges as the techniques rely on ground truth labels with limited accuracy, and histological similarity, and do not consider the disease pathology characteristics, and indefinite differences in appearance between the cancerous and healthy tissue lead to many false positive and false negative interpretations. Hence, this research introduces a comprehensive framework designed to achieve accurate identification and localization of prostate cancers, irrespective of their aggressiveness. This is accomplished through the utilization of a sophisticated multilevel bidirectional long short-term memory (Bi-LSTM) model. The pre-processed images are subjected to multilevel feature map-based U-Net segmentation, bolstered by ResNet-101 and a channel-based attention module that improves the performance. Subsequently, segmented images undergo feature extraction, encompassing various feature types, including statistical features, a global hybrid-based feature map, and a ResNet-101 feature map that enhances the detection accuracy. The extracted features are fed to the multilevel Bi-LSTM model, further optimized through channel and spatial attention mechanisms that offer the effective localization and recognition of complex structures of cancer. Further, the framework represents a promising approach for enhancing the diagnosis and localization of prostate cancers, encompassing both indolent and aggressive cases. Rigorous testing on a distinct dataset demonstrates the model’s effectiveness, with performance evaluated through key metrics which are reported as 96.72%, 96.17%, and 96.17% for accuracy, sensitivity, and specificity respectively utilizing the dataset 1. For dataset 2, the model achieves the accuracy, sensitivity, and specificity values of 94.41%, 93.10%, and 94.96% respectively. These results surpass the efficiency of alternative methods.© 2024. The Author(s) under exclusive licence to Society for Imaging Informatics in Medicine.