The Power of Many: Running Many Simulations on Many : Dr. Shantenu Jha, Rutgers University (57 mins, ~28 MB)
Listen now
Description
There are several important science and engineering problems that require the coordinated execution of multiple high-performance simulations. Some common scenarios include but are not limited to, "an ensemble of tasks", "loosely-coupled simulations of tightly-coupled simulations" or "multi-component multi-physics simulations". However, historically supercomputing centers, have supported and priortised the execution of single "jobs" on supercomputers. Not suprisingly, the tools and capabilities to support coordinated multiple simulations are limited. A promising way to overcome this common limitation is the use of a Pilot-Job --- which can be defined as a container or placeholder job to provide multi-level scheduling via an application-level scheduling overlay over the system scheduler. We discuss both the theory and practise of Pilot-Jobs: Specifically, we introduce the P* Model of Pilot-Jobs and present "BigJob" as a SAGA-based extensible, interopable and scalable implementation of the P* Model. We then discuss several science problems that have/are using BigJob to execute multiple simulations at unprecedented scales on a range of supercomputers and distributed supercomputing infrastructure such as XSEDE. This talk was given as part of our MSc in HPC's 'HPC Ecosystem' course. Talk slides
More Episodes
Bioinformatics and more widely Computational Biology is a largely data-driven Science. The array of high-throughput technology platforms in the last 10 years mean that the amount of data being generated in this field is likely to enter into Exabytes by 2020. The challenges associated with this...
Published 03/21/14
Performing complex solar shading analysis to take into account the sun's path and solar penetration on large buildings has historically consumed very many CPU cycles for IES "Virtual Environment" (3D building physics) simulation users. One particularly complex model took almost 2 weeks to...
Published 03/14/14
Intel will provide an insight into future HPC technology development looking at hardware trends, ecosystem support and the challenges around ExaScale computing. The talk will also touch upon the convergence of High Performance Computing and High Performance Data Analytics, examining where the...
Published 02/28/14