Optimization of Machine Learning Algorithms for Big Data Analytics
DOI:
https://doi.org/10.69996/jcai.2026011Keywords:
Big Data Analytics, Machine Learning, Optimization, Artificial Intelligence, Apache Spark, Hadoop, Deep Learning, Data MiningAbstract
The exponential growth of big data has transformed modern computational systems and introduced significant challenges in data processing, storage, scalability, and intelligent decision-making. Machine Learning (ML) algorithms play a critical role in extracting meaningful information from largescale datasets; however, conventional algorithms often suffer from high computational complexity, memory limitations, and reduced processing efficiency when applied to big data environments. This research paper investigates the optimization of machine learning algorithms for big data analytics using advanced computational techniques such as distributed computing, feature selection, parallel processing, dimensionality reduction, and hyperparameter tuning. The study evaluates the performance of optimized ML models on large datasets using Apache Hadoop, Apache Spark, and cloud-based analytical frameworks. Experimental results demonstrate substantial improvements in processing speed, prediction accuracy, scalability, and resource utilization. The optimized Random Forest and Deep Learning models achieved accuracy improvements of up to 95.6% while reducing execution time by 48%. The research highlights the significance of algorithmic optimization for real-time analytics, intelligent forecasting, and scalable data-driven applications. The proposed framework contributes toward efficient and sustainable big data analytics systems suitable for industrial, healthcare, financial, and smart city applications.
References
[1] E. F. Codd, “A relational model of data for large shared data banks,” Communications of the ACM, vol. 13, no. 6, pp. 377–387, 1970.
[2] M. Stonebraker, “The case for shared nothing,” Database Engineering, vol. 9, no. 1, pp. 4–9, 1986.
[3] L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp. 5–32, 2001.
[4] J. Dean and S. Ghemawat, “MapReduce: Simplified data processing on large clusters,” Communications of the ACM, vol. 51, no. 1, pp. 107–113, 2008.
[5] B. Babcock, “Models and issues in data stream systems,” in Proc. ACM SIGMOD-SIGACT-SIGART Symp. Principles of Database Systems (PODS), 2005, pp. 1–16.
[6] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” Science, vol. 313, no. 5786, pp. 504–507, 2006.
[7] H. Chen, R. H. L. Chiang, and V. C. Storey, “Business intelligence and analytics: From big data to big impact,” MIS Quarterly, vol. 36, no. 4, pp. 1165–1188, 2012.
[8] M. Zaharia, “Spark: Cluster computing with working sets,” in Proc. USENIX Conf. Hot Topics in Cloud Computing (HotCloud), pp. 1–7, 2010.
[9] J. Fan, F. Han, and H. Liu, “Challenges of big data analysis,” National Science Review, vol. 1, no. 2, pp. 293–314, 2014.
[10] M. I. Jordan and T. M. Mitchell, “Machine learning: Trends, perspectives, and prospects,” Science, vol. 349, no. 6245, pp. 255–260, 2015.
[11] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015.
[12] Vani Makula and Akhil Khare, “A Systematic Review on Big Data Analytics Using Machine Learning forPredicting Climate Change and its Impact on Human Health,” JCAI, vol. 3, no. 3, pp. 40–55, 2025.
[13] T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining (KDD), pp. 785–794, 2016.
[14] K. Kambatla, “Trends in big data analytics,” Journal of Parallel and Distributed Computing, vol. 74, no. 7, pp. 2561–2573, 2014.
[15] M. Li, “Scaling distributed machine learning with the parameter server,” in Proc. 11th USENIX Symp. Operating Systems Design and Implementation (OSDI), pp. 583–598, 2014.
[16] Z. Sun, “Big data analytics and machine learning,” Future Generation Computer Systems, vol. 92, pp. 122–135, 2019.
[17] V. Kumar and D. Sharma, “Optimization techniques in machine learning for big data analytics,” International Journal of Data Science, vol. 10, no. 2, pp. 88–102, 2022.
[18] A. Singh et al., “Distributed AI systems for scalable analytics,” Journal of Big Data Analytics, vol. 5, no. 1, pp. 45–60, 2022.
[19] M. Khan, “Smart analytics using optimized deep learning systems,” Artificial Intelligence Review, vol. 56, no. 4, pp. 3157–3180, 2023.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Computer Allied Intelligence(JCAI, ISSN: 2584-2676)
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Fringe Global Scientific Press publishes all the papers under a Creative Commons Attribution-Non-Commercial 4.0 International (CC BY-NC 4.0) (https://creativecommons.org/licenses/by-nc/4.0/) license. Authors have the liberty to replicate and distribute their work. Authors have the ability to use either the whole or a portion of their piece in compilations or other publications that include their own work. Please see the licensing terms for more information on reusing the work.
