Design and implementation of an application-layer data throughput prediction and optimization service for many-task computing in widely distributed environments. This service uses multiple parallel TCP streams to improve the end-to-end throughput of data transfers. A novel mathematical model is developed to determine the number of parallel streams, required to achieve the best network performance. This model can predict the optimal number of parallel streams with as few as three prediction points. We implement this new service in the Stork Data Scheduler. We propose Stork data transfer jobs with optimization service can be completed much earlier compared to non-optimized data transfer jobs.

1. Allcock . W, Bresnahan . J, Kettimuthu . R, Link . M, Dumitrescu . C, Raicu . I, and Foster . I, "The Globus Striped Gridftp Framework and Server," Proc. 2005 ACM/IEEE Conf. Supercomputing (SC'05), p. 54, 2005.
2. Altman . E, Barman . D, Tuffin . B, and Vojnovic . M, "Parallel TCP Sockets: Simple Model, Throughput and Validation," Proc. IEEE INFOCOM '06, pp. 1-12, Apr. 2006.
3. Karrer . R. P, Park . J, and Kim . J, "TCP-ROME: Performance and Fairness in Parallel Downloads for Web and Real Time Multimedia Streaming Applications," technical report, Deutsche Telekom Labs,2006.
4. Kosar . T and Livny . M, "Stork: Making Data Placement a First Class Citizen in the Grid," Proc. IEEE Int'l Conf. Distributed Computing Systems (ICDCS '04), pp. 342-349, 2004.
5. Raicu . I, Foster . I , and Zhao . Y, "Many-Task Computing for Grids and Supercomputers," Proc. IEEE Workshop Many-Task Computing on Grids and Supercomputers (MTAGS), 2008.
6. Timothy G. Armstrong, "Scheduling Many-Task Workloads on Supercomputers: Dealing with Trailing Tasks," Zhao Zhang Department of Computer Science University of Chicago.
7. Yildirim . E, Yin . D, and Kosar . T, "Prediction of Optimal Parallelism Level in Wide Area Data Transfers," to be published in IEEE Trans. Parallel and Distributed Systems, 2010.

Many Task Computing Scheduling