Objective Colorectal cancer (CRC) is a common malignant tumor of the digestive system with a high incidence rate. It is prone to misdiagnosis or missed diagnosis in clinical practice. Therefore, researching computer-aided diagnostic methods for endoscopic colon disease image classification is of great importance. This study proposes a deep learning-based method for colon disease classification. It utilizes intestinal images or captures from an endoscope camera to achieve intelligent classification of gastrointestinal diseases, providing assistance to doctors in their decision-making process. Methods Firstly, the algorithm is used to preprocess the dataset by removing duplicates and applying enhancement techniques. Two different network architectures, namely A_Vit, MobileNet, are employed. The models are trained using the same parameters and dataset with the Adam optimizer. The training process generates loss curves, accuracy, and recall rates for each of the four network architectures. Results The results indicate that the training with A_Vit has shown better performance, achieving an accuracy rate of 95.76% and an impressive recall rate of 97.21%. Therefore, the model trained using the A_Vit network structure is ultimately selected as the preferred choice. Conclusion This method can improve the efficiency and accuracy of colon disease diagnosis.