Electroencephalography (EEG) is increasingly being used in pediatric neurology and provides opportunities to diagnose various brain illnesses more accurately and precisely. It is thought to be one of the most effective tools for identifying newborn seizures, especially in Neonatal Intensive Care Units (NICUs). However, EEG interpretation is time-consuming and requires specialists with extensive training. It can be challenging and time-consuming to distinguish between seizures since they might have a wide range of clinical characteristics and etiologies. Technological advancements such as the Machine Learning (ML) approach for the rapid and automated diagnosis of newborn seizures have increased in recent years. This work proposes a novel optimized ML framework to eradicate the constraints of conventional seizure detection techniques. Moreover, we modified a novel meta-heuristic optimization algorithm (MHOA), named Aquila Optimization (AO), to develop an optimized model to make our proposed framework more efficient and robust. To conduct a comparison-based study, we also examined the performance of our optimized model with that of other classifiers, including the Decision Tree (DT), Random Forest (RF), and Gradient Boosting Classifier (GBC). This framework was validated on a public dataset of Helsinki University Hospital, where EEG signals were collected from 79 neonates. Our proposed model acquired encouraging results showing a 93.38% Accuracy Score, 93.9% Area Under the Curve (AUC), 92.72% F1 score, 65.17% Kappa, 93.38% sensitivity, and 77.52% specificity. Thus, it outperforms most of the present shallow ML architectures by showing improvements in accuracy and AUC scores. We believe that these results indicate a major advance in the detection of newborn seizures, which will benefit the medical community by increasing the reliability of the detection process.