Blue Waters Symposium 2019

The Convergence of Big Data and Large-scale Simulation

Motivations abound for the convergence of large-scale simulation and big data: (1) scientific and engineering advances, (2) computational and data storage efficiency, (3) economy of data center operations, and (4) the development of a competitive workforce. To take advantage of advances in analytics and learning, large-scale simulations should incorporate these technologies in-situ, rather than as forms of post-processing. This potentially reduces IO, may obviate significant computation in unfruitful regions of physical parameter space, offers smart data compression, and potentially improves the results of the simulation, itself, since many simulations incorporate empirical relationships currently tuned by human experts. Flipping the perspective, simulation potentially provides significant benefits to analytics and learning workflows. Theory-guided data science is an emerging paradigm that aims to improve the effectiveness of data science models, as a form of regularization, wherein non-unique candidates are penalized by physical constraint. Simulation can also provide training data for machine learning. Finally, much software has been developed for large-scale simulation, particularly in data-sparse linear algebra and second-order optimization, that promises to expand the practical reach of analytics; while the complex parameterized codes of large-scale simulation may in turn be tuned for computational performance by machine learning applying machine learning to the machine.