Selecting suitable parameter configurations in hybrid post-quantum cryptographic systems is not only a matter of security strength but also of performance efficiency under varying system and network conditions. Many existing studies focus mainly on security guarantees, with limited attention to how different configurations behave in real deployment environments. This creates a gap in understanding how to balance security requirements with practical performance. This study explores the use of machine learning as a data-driven approach to analyze and predict performance trends in a hybrid cryptographic system that combines CRYSTALS-Kyber (a lattice-based post-quantum key encapsulation mechanism) with the Advanced Encryption Standard (AES). A controlled environment for the experimental was developed using Python-based tools. Multiple configurations of AES key sizes (128, 192, 256 bits) and Kyber variants (Kyber512, Kyber768, Kyber1024) were evaluated under different network and computational conditions. The dataset result captured key performance metrics such as network latency, encryption latency, decryption latency, throughput, and CPU utilisation. Different machine learning models, including Random Forest, XGBoost, Linear Regression, Decision Tree, LightGBM, and Multilayer Perceptron (MLP), were trained and evaluated using MAE, RMSE, and R² metrics. The results show that while some models, particularly Random Forest and LightGBM, achieved relatively lower prediction errors for certain metrics, the overall predictive strength across tasks remained weak, with most R² values close to zero or negative. This indicates that the performance behaviour of the hybrid cryptographic system is influenced by complex and highly variable interactions that are not easily captured using standard supervised learning models and the current feature set.

Post-Quantum Cryptography CRYSTALS-Kyber Advanced Encryption Standard Performance Analysis Machine Learning Hybrid Cryptographic Systems System Optimization.