CFP last date
20 March 2025
Reseach Article

Standardization of System Integrated Data Engineering Architecture

by Purvesh Jadhav
International Journal of Computer Applications
Foundation of Computer Science (FCS), NY, USA
Volume 186 - Number 60
Year of Publication: 2025
Authors: Purvesh Jadhav
10.5120/ijca2025924363

Purvesh Jadhav . Standardization of System Integrated Data Engineering Architecture. International Journal of Computer Applications. 186, 60 ( Jan 2025), 45-49. DOI=10.5120/ijca2025924363

@article{ 10.5120/ijca2025924363,
author = { Purvesh Jadhav },
title = { Standardization of System Integrated Data Engineering Architecture },
journal = { International Journal of Computer Applications },
issue_date = { Jan 2025 },
volume = { 186 },
number = { 60 },
month = { Jan },
year = { 2025 },
issn = { 0975-8887 },
pages = { 45-49 },
numpages = {9},
url = { https://ijcaonline.org/archives/volume186/number60/standardization-of-system-integrated-data-engineering-architecture/ },
doi = { 10.5120/ijca2025924363 },
publisher = {Foundation of Computer Science (FCS), NY, USA},
address = {New York, USA}
}
%0 Journal Article
%1 2025-01-28T19:06:50.786622+05:30
%A Purvesh Jadhav
%T Standardization of System Integrated Data Engineering Architecture
%J International Journal of Computer Applications
%@ 0975-8887
%V 186
%N 60
%P 45-49
%D 2025
%I Foundation of Computer Science (FCS), NY, USA
Abstract

Data Engineering plays an important role in the Data Science and Analytics industry to ensure that the data availability is seamless, efficient and accurate for further analysis. Rapid evolution of data engineering facilitates the need for robust and standardized approaches of system integration in data engineering. With increase demands for scalability, reliability, efficiency and inconsistent methodologies can lead to fragmented solutions, operational inefficiencies and high maintenance cost, thus standardization of system integrated data engineering solution helps to mitigate these challenges achieve the timely and effective data consumption, storage and availability. Also, managing these system integration solutions becomes challenging if business runs under different industries, and different platforms. Hence it is very important for data engineering architects to standardize the system integrated solutions when it comes to data engineering project implementation. This article explains feasibility of standardizing the system integrated data engineering architecture and proposing a framework that emphasizes uniform practices in areas such as data ingestion, transformation, storage, and retrieval across diverse platforms by conducting a case study of leading Global Online Retail Company. As more data sources and more business problems are involved in data engineering, the architecture becomes more and more complex to manage. The core and integral part of data engineering is to ensure ingestion and storage of data

References
  1. Sultan Yerbulatov, “Integration of Big Data and Data Engineering in Modern Organizations”, International Journal of Scientific Engineering and Science, Volume 8, Issue 6, pp. 11-14, 2024. https://ijses.com/wp-content/uploads/2024/06/36-IJSES-V8N3.pdf
  2. Professor Marco Iansiti (Harvard University), "Data Governance, Interoperability and Standardization: Organizational Adaptation to Privacy Regulation.", No. 21-122, May 2021. (Revised November 2023.) https://www.hbs.edu/ris/Publication%20Files/21-122_77bc83c9-3aec-44ad-8bec-0c0fa181c8a8.pdf
  3. Professor V.J. Reddi, “Data Engineering for everyone” (Harvard University), “arXiv:2102.11447v1 [cs.LG] 23 Feb 2021. https://arxiv.org/abs/2102.11447
  4. Professor Michael Stonebraker (MIT), “One Size Fits All: An Idea Whose Time Has Come and Gone” https://cs.brown.edu/~ugur/fits_all.pdf
  5. AWS Cloud Services: Leading cloud provider in industry https://aws.amazon.com/
  6. AWS managed Airflow (MWAA): Orchestration tool developed by Apache. https://airflow.apache.org/
Index Terms

Computer Science
Information Sciences

Keywords

AWS (Amazon Web Services) System Integration ETL pipelines Cloud Native Solution Data Engineering.