As a drop-in replacement for an ordinary shell that orchestrates steps, but does not reorder or split them, PaSh provides a no-hassle way to improve the performance of big data-processing jobs,” adds Douglas McIlroy, adjunct professor in the Department of Computer Science at Dartmouth College, who previously led the Computing Techniques Research Department at Bell Laboratories (which was the birthplace of the Unix operating system). “Hand optimization to exploit parallelism must be done at a level for which ordinary programming languages (including shells) don’t offer clean abstractions. The resulting code intermixes matters of logic and efficiency. It’s hard to read and hard to maintain in the face of evolving requirements. PaSh cleverly steps in at this level, preserving the original logic on the surface while achieving efficiency when the program is run.” Reference: “Practically Correct, Just-in-Time Shell Script Parallelization” by Konstantinos Kallas, Tammam Mustafa, Jan Bielak, Dimitris Karnikis, Thurston H.Y. Dang, Michael Greenberg and Nikos Vasilakis. PDF This work was supported, in part, by Defense Advanced Research Projects Agency and the National Science Foundation.Our system is the first that shows this type of fully correct transformation, but there is an indirect benefit, too. The way our system is designed allows other researchers and users in industry to build on top of this work,” Vasilakis says. He is excited to get additional feedback from users and see how they enhance the system. The open-source project joined the Linux Foundation last year, making it widely available for users in industry and academia. Moving forward, Vasilakis wants to use PaSh to tackle the problem of distribution — dividing a program to run on many computers, rather than many processors within one computer. He is also looking to improve the annotation scheme so it is more user-friendly and can better describe complex program components. “Unix shell scripts play a key role in data analytics and software engineering tasks. These scripts could run faster by making the diverse programs they invoke utilize the multiple processing units available in modern CPUs. However, the shell’s dynamic nature makes it difficult to devise parallel execution plans ahead of time,” says Diomidis Spinellis, a professor of software engineering at Athens University of Economics and Business and professor of software analytics at Delft Technical University, who was not involved with this research. “Through just-in-time analysis, PaSh-JIT succeeds in conquering the shell’s dynamic complexity and thus reduces script execution times while maintaining the correctness of the corresponding results.”To overcome this problem, uses a preprocessing step that inserts simple annotations onto program components that it thinks could be Then attempts to parallelize those parts of the script while the program is running, at the exact moment it reaches each component. This avoids another problem in shell programming — it is impossible to predict the behavior of a program ahead of time. By parallelizing program components “just in time,” the system avoids this issue. It is able to effectively speed up many more components than traditional methods that try to perform parallelization in advance. Just-in-time parallelization also ensures the accelerated program still returns accurate results. If at a program component that cannot be parallelized (perhaps it is dependent on a component that has not run yet), it simply runs the original version and avoids causing an error.
Top Computer scientists developed a new system that can make computer programs run faster while guaranteeing accuracy.
About Author
You must be logged in to post a comment.