The week's pick
Reseach Article
Optimizing Data Storage for AI, Generative AI, and Machine Learning: Challenges, Architectures, and Future Direction
| International Journal of Computer Applications |
| Foundation of Computer Science (FCS), NY, USA |
| Volume 186 - Number 73 |
| Year of Publication: 2025 |
| Authors: Ankush Ramprakash Gautam |
10.5120/ijca2025924597
|
Ankush Ramprakash Gautam . Optimizing Data Storage for AI, Generative AI, and Machine Learning: Challenges, Architectures, and Future Direction. International Journal of Computer Applications. 186, 73 ( Mar 2025), 29-33. DOI=10.5120/ijca2025924597
Abstract
In rapidly evolving fields of study such as [1] Artificial Intelligence (AI), [2] Generative AI, [3] Retrieval-Augmented Generation (RAG), and [4] Machine Learning (ML), it is crucial to store data efficiently. The ability to store, manage and retrieve large datasets has a direct impact on the performance, scalability and reliability of these applications. AI and ML depend on large amounts of data for training and inference, therefore, it needs storage solutions that are high-throughput, low-latency and cost-effective. This article aims to explore the role of data storage in AI and ML, its advantages and limitations, and presents insights from recent scholarly research. The paper also discusses various storage architectures such as cloud, hybrid, and on-premise and how they are applicable to different AI workload.
References
- Artificial Intelligence definition [Online] https://en.wikipedia.org/wiki/Artificial_intelligence
- Generative AI definition [Online] https://en.wikipedia.org/wiki/Generative_artificial_intelligence
- Retrieval-augmented generation definition [Online] https://en.wikipedia.org/wiki/Retrieval-augmented_generation
- Machine Learning definition [Online] https://en.wikipedia.org/wiki/Machine_learning
- Vector database definition [Online] https://en.wikipedia.org/wiki/Vector_database
- HDFS definition [Online] https://en.wikipedia.org/wiki/Apache_Hadoop#HDFS
- Ceph definition [Online] https://en.wikipedia.org/wiki/Ceph_(software)
- Liu, Yu, et al. A survey on AI for storage. CCF Transactions on High Performance Computing, vol. 4, 2022, pp. 233–264.
- van Ooijen, P. M. A., Erfan Darzidehkalani, and Andre Dekker. AI Technical Considerations: Data Storage, Cloud usage and AI Pipeline. arXiv preprint arXiv:2201.08356, 2022.
- Sriramoju, Sumalatha. A Comprehensive Review on Data Storage. International Journal of Scientific Research in Science and Technology, vol. 6, no. 5, 2019.
- Zhao, Mark, et al. Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training. arXiv preprint arXiv:2108.09373, 2021.
- Aizman, Alex, Gavin Maltby, and Thomas Breuel. High Performance I/O For Large Scale Deep Learning. arXiv preprint arXiv:2001.01858, 2020.
- Lian, Xiang, and Xiaofei Zhang. Learning-Based Data Storage [Vision]. arXiv preprint arXiv:2206.05778, 2022.
- Gu, Albert. Mamba: A New Model Design for AI Efficiency. Time, 2024.
- Hooker, Sara. Enhancing Model Efficiency and Data Quality in AI. Time, 2024.
- AI Will Force a Transformation of Tech Infrastructure. The Wall Street Journal, 2024.
- Scientists develop DNA technology in data storage breakthrough. Financial Times, 2024.
Index Terms
Keywords