The Convergence of Big Data and Large-scale Simulation
Motivations abound for the convergence of large-scale simulation and big data: (1) scientific and engineering advances, (2) computational and data storage efficiency, (3) economy of data center operations, and (4) the development of a competitive workforce. To take advantage of advances in analytics and learning, large-scale simulations should incorporate these technologies in-situ, rather than as forms of post-processing. This potentially reduces IO, may obviate significant computation in unfruitful regions of physical parameter space, offers smart data compression, and potentially improves the results of the simulation, itself, since many simulations incorporate empirical relationships currently tuned by human experts. Flipping the perspective, simulation potentially provides significant benefits to analytics and learning workflows. Theory-guided data science is an emerging paradigm that aims to improve the effectiveness of data science models, as a form of regularization, wherein non-unique candidates are penalized by physical constraint. Simulation can also provide training data for machine learning. Finally, much software has been developed for large-scale simulation, particularly in data-sparse linear algebra and second-order optimization, that promises to expand the practical reach of analytics; while the complex parameterized codes of large-scale simulation may in turn be tuned for computational performance by machine learning applying machine learning to the machine.
Monday June 3, 2019 9:00am - 10:00am PDT
Homestead
COMPASS probes proton substructure by scattering high-energy pion- and muon-beams on nuclear targets at CERN. The experiment explores the momentum and coordinate phase space of quarks inside the proton. Observing correlations between proton spin and the intrinsic transverse momentum of quarks will shed light on the quark dynamics inside the proton and will provide a critical test of fundamental predictions derived from Quantum Chromo Dynamics, the quantum field theory describing the strong nuclear force. The measurements produce 10 petabytes of experimental and simulated data. Blue Waters' balance of processing capabilities and data storage and handling is well suited for the analysis of the large COMPASS data samples as these require significant algorithmic processing per pion/muon-proton scattering event. In addition to raw data processing and physics-level analysis, COMPASS carries out extensive studies of systematic effects as Blue Waters allows for a detailed simulation of COMPASS detector and environmental properties.
Monday June 3, 2019 10:30am - 10:50am PDT
Heritage
High-resolution simulations using the Community Earth System Model, version 1.3 (CESM1.3) were conducted on Blue Waters for the purpose of advancing the study of climate change and its potential impacts on our planet, and more specifically for the purposes of examining climate extremes. This presentation will serve to update the Blue Waters community on the important role this project plays in a much larger effort to provide the climate community with ensembles of climate simulations of varying ocean and atmosphere resolution and complexity. Current research on midlatitude storms and storm tracks in both hemispheres will also be discussed. This research utilizes a 0.25° atmosphere and 0.1° ocean, the highest resolution possible with the CESM. Discussion will include: dependence on model resolution of processes involved in future change of these storms, namely atmospheric stability and steering, and sensitivity to model coupling and modelled atmospheric physics on storm track strength and location.
Monday June 3, 2019 10:30am - 10:50am PDT
Homestead
Parallel numerical linear algebra libraries are a key component of the software stack on HPC systems. We are pursuing development of novel communication avoiding algorithms and libraries that encapsulate them. This talk will focus on advances in parallel QR factorization, one of the most widely used matrix factorizations and a critical component of linear least squares and eigenvalue problems. Our new communication-avoiding CholeskyQR2 algorithm is the first practical algorithm that is communication optimalgiven any amount of memory and matrix dimensions. The algorithm requires slightly more computation but achieves a much higher arithmetic intensity, making it especially promising on emerging HPC architectures. We demonstrate its benefit via strong and weak scaling studies on Blue Waters and Stampede2, achieving speed-ups of up to 3.3x when using 65,000 processors.
Monday June 3, 2019 10:30am - 10:50am PDT
Landmark
There is strong evidence for the existence of dark matter, but most of its properties are unknown. Experiments around the world are therefore searching for signals of atomic nuclei recoiling from collisions with dark matter. To constrain theories of dark matter with existing null results and reliably interpret future experimental discoveries, the interactions between nuclei and possible types of dark matter must be understood and accurately modeled. Nuclear structure effects on these interactions can be calculated from the Standard Model of particle physics using lattice QCD, and our first calculations predict unexpectedly large nuclear structure effects in light nuclei. These exploratory calculations used unphysically heavy particle masses to reduce computational cost, and we are now using Blue Waters HPC resources and GPU optimization techniques including JIT compilation to embark on calculations with physical particle masses in order to definitively calculate the interaction strengths of light nuclei with dark matter.
Monday June 3, 2019 10:50am - 11:10am PDT
Heritage
In the coming decades, models predict an increase in climate extremes in Southwest North America (SWNA), with both more severe droughts and more intense precipitation events. However, confidence in model projections of SWNA climate are limited by an incomplete understanding of the complex interactions driving regional climate variability and the unprecedented nature of future climate perturbation. By studying the past, we can improve our understanding of the mechanisms driving current and future hydroclimate change in SWNA. This region has been particularly well surveyed, with some of the densest and most diverse coverage of paleoclimate proxy records available. Nevertheless, comparisons between climate model outputs and proxy records remain difficult because 1) the resolution of the models is often too low to resolve the topography of the sample sites and 2) the models do not directly simulate the quantities measured in the proxy records. Here, we overcome these limitations by employing a 0.25 resolution, water isotope-enabled Earth system model for simulations of the Last Glacial Maximum and preindustrial. These simulations are computationally expensive, requiring the use of the Blue Waters supercomputer.
Monday June 3, 2019 10:50am - 11:10am PDT
Homestead
Over the last years, the availability of genomic sequence data from thousands of different species has led to hopes that a phylogenetic (i.e., evolutionary) tree of all life might be achievable. Yet, the most accurate methods for estimating phylogenies are heuristics for NP-hard optimization problems, many of which are too computationally intensive to use on large datasets. In research supported by Blue Waters, we have developed new divide-and-conquer approaches for scaling the leading species tree estimation methods to large datasets. In this talk, I will describe these strategies and their theoretical properties, present open problems, and discuss opportunities for impact in large-scale phylogenetic estimation using these and similar approaches.
Monday June 3, 2019 10:50am - 11:10am PDT
Landmark
The BATS-R-US magnetohydrodynamic (MHD) code can solve various forms of the MHD equations on block-adaptive grids with a variety of spatial and temporal discretizations. The code consists of over 200k lines of Fortran 90+ code. Scaling to very large problems has been limited to about 16k to 32k cores on Blue Waters due to memory constraints: the global block tree information is stored on every MPI process, and the tree becomes larger as the number of grid blocks is increased. Using OpenMP can mitigate this issue, and allow scaling to 500k cores. Our strategy of OpenMP parallelization exploits the grid blocks: the loops over the blocks are multi-threaded instead of individual grid cells. With relatively modest modifications of the source code, a lot of testing and debugging, all done by graduate student Hongyang Zhou, we achieved our goal. I will discuss issues, solutions, and show some extremely promising results.
Monday June 3, 2019 11:10am - 11:30am PDT
Heritage
Global circulation models operate at scales requiring parameterizations to represent the effects of convective storms that occur over distances too small to be directly resolved. These parameterizations lack sufficient representation of the primary convective components. Of particular interest here is the cold pool, produced by the melting or evaporation/sublimation of ice/raindrops falling from the storm. Improving parameterizations requires a better understanding of these microscale processes contributing to the larger-scale cold pool, requiring a set of high-resolution simulations over large domains with various modifications of precipitation processes, a feat that is best accomplished with petascale computing on Blue Waters. Results from the 12 simulations show that (i) the sublimation of graupel within downdrafts contributes the most to the cold pool; (ii) evaporation of rain better explains its depth and propagation speed; and (iii) the importance of the warm rain process in creating the earliest precipitation that modulates its first appearance.
Monday June 3, 2019 11:10am - 11:30am PDT
Homestead
The focus of this talk is on improving simulation performance by exposing algorithms to machine layout, as found in modern machines such at Blue Waters. The solution to large sparse linear systems, for example, often demands a high amount of communication. By reorganizing the communication in a node-centric manner, the number of costly intranode messages can be reduced. The talk will highlight the benefits of using node aware communication in the context of a multigrid solver. The solve times can be reduced in comparison to conventional approaches. The results also extend to a wider range of sparse matrix operations and a general communication package will be presented.
Monday June 3, 2019 11:10am - 11:30am PDT
Landmark
The phase diagram of high pressure hydrogen is of great interest both for fundamental research,such as in astrophysicsand for energy applications. The existence and precise location of a phase transition in the liquidphase between a molecular insulating fluid and a monoatomic metallic fluid is relevant for planetary models. Recentexperiments reported contrasting results about itslocationandresultsbased onDensity Functional Theory are very scattered. We performedCoupledElectron-Ion Quantum Monte Carlo calculations of this transition, finding resultsthat laybetween the twoexperimental predictions. The transition is signaled by a discontinuity inthe specificvolume, a sudden dissociation of themolecules, a jump in electrical conductivity and in electronlocalization.During the past year our prediction of a transition was verified by a new experiment and our recent results of its optical properties reinterpret the other experiments.
Monday June 3, 2019 11:30am - 11:50am PDT
Heritage
Past studies have shown that projected climate change could affect air quality, but little is known about resulting effects on health - this study examines exceedances for upper limits of exposure set by environmental policy. We use a fully coupled model of the Earth's climate system with interactive atmospheric chemistry; Blue Waters is one of the few computers that can be used for such a complex study. Exceedances for surface ozone and particulate matter (PM) concentrations are examined for two different climate projections. We especially focus the United States, India and China. Along with an overall increase in ozone exceedances, the study also projects a significant shift in seasonality of the events. PM showed a significant increasing trend in all the regions in the future with an overall increase of number of exceedance events annually. We also show that a switch to clean energy by mid-century would greatly improve air quality.
Monday June 3, 2019 11:30am - 11:50am PDT
Homestead
Air travel has been identified as a leading factor in the spread of infections. We had earlier shown that changes to boarding and deplaning procedures could substantially reduce the risk of transmitting Ebola and SARS on flights. This was accomplished through a fine-scaled model that tracks the movement of passengers in airplanes. Our approach incurs a large computational cost to deal with inherent uncertainties in human behavior, which is addressed through massive parallelization. We will discuss new techniques, such as low dimensional parameter sweep, to reduce the above computational cost. In addition, optimizations using number-theoretic properties of low-discrepancy sequences reduce load imbalance on Blue Waters. These optimizations enable our simulations to produce timely results as needed in emergencies that threaten public health.
Monday June 3, 2019 11:30am - 11:50am PDT
Landmark
The violent collapse of cavitation bubbles can damage its surroundings and causing cavitation erosion on material surfaces. To understand this phenomenon, we have developed a novel numerical model to conduct efficient, high-fidelity simulations to investigate the collapse of single and multiple bubbles near (i) semi-infinite rigid and (ii) finite-size elastic objects. The numerical model solves the three-dimensional compressible Navier-Stokes equations for a multi-component system and includes visco(elasticity) to investigate the interactions between an object and the bubble collapse fluid flow. The Blue Waters system's computational power and uniqueness was essential to conduct the high-resolution simulations to quantify the non-spherical bubble morphology, maximum pressure/temperature fields and elastic components of stress therein produced. As a result, these fundamental simulations have furthered our ability to quantify the damage mechanisms and lead to strategies to better control cavitation-induced erosion in a wide-range of naval and medical engineering and energy science applications.
Binary neutron-star mergers belong to the most energetic transients in the universe. They are key targets for time-domain astronomy surveys and advanced LIGO in the era of multimessenger observations. Magnetic fields play a crucial role in the merger and postmerger and are the key ingredient to determine the lifetime of the hypermassive neutron star remnant, the composition of ejected material, and the formation of a sGRB engine. I will discuss the unique challenges in both input physics and computational modeling for these systems involving all four fundamental forces and present results from recent breakthrough3D simulations performed on BlueWaters.
The Amber empirical force field is the major workhorse of modern biomolecular simulation packages. However, tuning its parameters to the properties of biological materials represents a major methodological and computational challenge, due to the size of the required petascale-level simulations. This work employs a combination of experimental data, quantum chemistry calculations of finite systems, and large-scale RMG DFT-level simulations of a crystal of Guanine to generate an extended set of data for refinement of the empirical force field. Simultaneous charge fitting to the electrostatic potential of multiple clusters incorporates the condensed phase effect into the empirical parameters. The created extensive dataset of dimer, trimer and tetramer clusters includes configurations that probe the curvature of the potential energy profile of the crystal in the vicinity of its thermodynamic minimum that are not directly accessible from experiment. These runs, along with the Lennard-Jones parameter sweep, require petascale compute capability.
Understanding the physics of the pressure fluctuations induced by turbulent boundary layers is of major theoretical and practical importance. From a practical point of view, an in-depth knowledge of the nature of boundary-layer-induced pressure fluctuations is essential to the structural design of launch vehicles and to the generation of background noise in wind-tunnel facilities. From a theoretical point of view, pressure fluctuations are an important ingredient in turbulence as they appear in statistical correlations such as the pressure-strain correlation terms that redistribute turbulence among different components of fluctuating velocity. In the presentation, I will talk about our use of the Blue Waters supercomputer to conduct direct numerical simulations that provide the basis for an in-depth understanding of the global pressure field induced by turbulent boundary layers at supersonic speeds. I will also discuss about our effort to maximize node-level parallelism on the Blue Waters.
When matter orbits around a black hole obliquely with respect to the holes spin axis, a relativistic torque causes the orbits to precess. Forty years ago, astrophysicists proposed that the orbital angular momentum in accreting gas should align with the mass's spin near the hole. The location of the alignment front is determined by a balance between the torque, the resulting differential precession, and warp-induced inward mixing of misaligned angular momentum from the outer to the inner disk. Analytic approaches are restricted to linear one-dimensional analysis, using viscosity for dissipation, despite the fact that accretion disks are not viscous systems, but MHD turbulent. Numerical simulations are required, but three-dimensional high-resolution simulations required have only recently become possible. We are investigating alignment through MHD and hydrodynamic simulations of mis-aligned disks. Our approach has been to use a semi-Newtonian method, using idealized disks that can probe the physical process in detail.
Unidirectional pulmonary airflow, a condition where lung gases travel in the same direction through most of the airways throughout the respiratory cycle, has long been of interest to comparative physiologists. Recent work has discovered this phenomenon beyond birds and raised questions about the underlying fluid dynamical phenomena occurring in unidirectional lungs. Computational fluid dynamics, which simulates patterns of flow from prescribed boundary conditions and the laws of fluid motion, provide a powerful tool to study airflow through these complex structures. Here, computed tomography scans were segmented into a detailed computational mesh, representing the major and minor airways of monitor lizards, Varanidae. The surface of the computational meshes expanded and contracted to simulate lung motion during ventilation and provided the boundary conditions for flow. Our models show unidirectional flow in many regions of the lung and reveals airflow patterns in chambers that are too small or are inaccessible with traditional techniques.
A robust and computationally efficient transition prediction methodology for hypersonic boundary layers is being developed, verified and validated at the University of Kentucky. This framework includes the Adaptive Mesh Refinement Wave-Packet Tracking (AMR-WPT) technique which is used for tracking linear and nonlinear instabilities in hypersonic boundary layers and, subsequently, for predicting the transition to turbulence. As opposed to classical fixed mesh approaches, an AMR method is employed for greater computational efficiency. The presentation will show the research that has been conducted using the Blue Waters HPC allocation. It has allowed for the AMR-WPT approach to be validated for tracking strongly nonlinear wavepackets in hypersonic boundary layers. Results from far more computationally costly Direct Numerical Simulations (DNS) were used for validating the AMR-WPT method. Furthermore, by utilizing the Blue Waters HPC allocation, we have been able to optimize the tracking procedure and the algorithms which govern the AMR-WPT method.
Magnetic fields have been observed on galactic scales. However, their coupling to the gas dynamics and turbulence in and around galaxies is still debated, especially concerning small-scale dynamo action and turbulent cascades. Resolving the turbulent cascade in magnetohydrodynamic (MHD) simulations of galaxies requires computational resources much larger than current supercomputers. Simplified models of turbulence are needed to reproduce the effects of magnetized turbulence below the grid resolution. High resolution simulations of MHD turbulence can be used to develop these subgrid models. To efficiently simulate astrophysical plasmas on current and next generation supercomputers, we have developed K-Athena, a performance portable version of Athena++ that also runs on GPUs. Using K-Athena, I have simulated the Taylor-Green vortex in the ideal compressible MHD regime to study transfer of magnetic and kinetic energies between scales in an idealized setup. This work will be used to inform high resolution galaxy simulations and subgrid turbulence models for Enzo-E, a cosmology code built for exascale.
We employ all-atom molecular dynamics simulations along with novel enhanced sampling techniques to characterize the large-scale conformational changes of membrane transporters and their coupling to chemical events. We have specifically used Blue Waters to study the conformational landscape of a bacterial membrane transporter involved in peptide transport. Proton-coupled oligopeptide transporters (POTs) couple the inwardly directed proton flow to the transport of small peptides and peptide-like molecules. The human POT transporters PepT1 and PepT2 provide the main route through which the body absorbs and retains dietary proteins. Employing enhanced sampling techniques and path-finding algorithms within an ensemble-based molecular dynamics framework, we have been able to characterize the conformational free energy landscape of GkPOT, a homolog of human POTs, and its dynamic behavior at the millisecond timescale. A detailed description of GkPOT conformational landscape achieved by these simulations sheds light on the structure-function relationships in POTs and other membrane transporters.
Turbulence in fluid flow is extremely complex, of vital importance in many problems of nature and engineering, and --- even in its simplest forms ---- a grand challenge problem for high performance computing. Precious resources on Blue Waters made available through two successive PRAC grants have enabled simulations at state-of-the-art grid resolution in the order of 1 trillion grid points. The computational approach is primarily based on Fourier pseudo-spectral methods, which are of high accuracy but communication intensive, especially at large problem sizes where massive parallelism is essential. Outstanding support from the Blue Waters staff have also allowed us to perform simulations efficiently up to 8192 Cray XE nodes on Blue Waters. These simulations have led to significant advances in fundamental physical understanding, of the fluid flow (velocity field), of turbulent mixing especially of substances of low molecular diffusivity, and the dispersion of collections of infinitesimal fluid elements both forwards and backwards in time. In this talk I plan to review the lessons we learned through code development and optimization on Blue Waters, and to discuss some of the new insights that we have obtained through the analysis of large datasets produced from this work. Specific topics will include the dynamics of four-particle clusters known as tetrads, and the statistics of locally averaged energy dissipation rates as well as circulation around closed contours of linear size in the so-called inertial range of high Reynolds number turbulence.
In our continuing quest for more comprehensive models of the intergalactic medium including resolved galaxies, we are developing Enzo-P/Cello (now renamed Enzo-E), the petascale/exascale fork of the popular Enzo cosmological AMR code. Enzo-E employs an array-of-octrees AMR mesh for improved scalability, and is parallelized using Charm++. We report on progress implementing a scalable Poisson solver for Enzo-E for cosmological applications. The solver consists of two steps: in a first step, the global gravitational potential on the uniform based grid is computed using geometric multigrid. In a second step, the local gravitational potential in each octree of the array is solved using a biconjugate gradient method subject to boundary conditions taken from the global mesh. We report on the weak and strong scaling properties of the DD solver, and compare its speed and accuracy to the original Enzo code.
Induction of potentially deadly abnormal cardiac rhythms is one of the most common and dangerous risks for drugs in development and clinical use. It has been tightly associated with the conduction block of the cardiac potassium channel hERG, which leads to delayed action potential repolarization, manifesting in the prolongation of the QT interval on the ECG. However, not all hERG blocking and QT prolonging drugs cause cardiac arrhythmias. This leads to withdrawal from the development of safe and effective pharmaceuticals. We have developed a multi-scale computational pipeline for safety pharmacology, which allows to determine drug proclivity for arrhythmogenesis based on its chemical structure and fundamental mode of interaction with hERG channel. All-atom enhanced sampling molecular dynamics simulations of hERG drug interactions, simultaneously running on multiple Blue Waters nodes, allowed us to determine state-dependent drug binding affinities and rates, which we used to predict emergent cardiac arrhythmias on cardiac tissue scale.
Detecting and understanding the first galaxies and supermassive black holes are one of the major challenges in galaxy formation. Our team has led the development of cosmological codes optimized to Petascale/Bluewaters and used these resources to understand how supermassive BHs and galaxies formed, from the smallest to the rarest and most luminous. With close to one trillion particles we have carried out the BlueTides simulation on BlueWaters. BlueTides has been run successfully on the entire set of compute nodes on Blue Waters using the latest version of our MP-Gadget code. With its large volume and high resolution, the BlueTides simulation is the only one that probes directly the first and rarest observed quasars in the early universe. With the increasing rate of discovery of galaxies and quasars at this early epoch, the BT dataset continues to increases its tremendous value as we evolve into the later Universe. We discuss the BT predictions for the first quasars counterparts in the regime at current and upcoming frontier of observational facilities. BT can probe the host galaxies of the first quasars, as well as the gas inflows/outflow that regulate the growth and regulation processes around the first supermassive black holes.
Birth of Planet Earth tells the twisted tale of our planet’s origins. Scientists now believe that our galaxy is filled with solar systems, including up to a billion planets roughly the size of our own. The film employs advanced, data-driven, cinematic-quality visualizations to explore some of the greatest questions in science today: How did Earth become a living planet in the wake of our solar system’s violent birth? What does its history tell us about our chances of finding other worlds that are truly Earth-like? Donna J. Cox and Robert Patterson will present a flat screen version of the digital fulldome show and provide an inside look at creating some of the scientific visualizations. Project is partially funded by NSF.
Zachary Andalman - Evanston Township High School and Northwestern University How does Stellar Spaghetti Circularize?
Bryton Anderson - Boise State University Computational Investigations of Surface Adsorption of Ethane on M1 Catalyst (Mo-V-Nb-Te-O)
Elaad Applebaum - Rutgers University Looking Out for the Little Guy: A Comprehensive Study of Star Formation in Dwarf Galaxies
Mark Avara - Rochester Institute of Technology 3D-GRMHD Simulations of Binary Black Hole Accretion: Beyond 30 Orbits
Katelyn Barber - University of North Dakota Characteristics of tropical and midlatitude out-of-cloud convectively-induced turbulence from high resolution convective simulations
Koushik Chatterjee - University of Amsterdam Simulating a Tilted Black Hole Shadow
Shanna Chu - Stanford University Source Processes of Intermediate-Depth Earthquakes
Robert L. Cieri - The University of Utah Computational fluid dynamics modeling reveals a unique net-unidirectional pulmonary airflow patterns in monitor lizards (Varanidae)
Abigail C. Dommer - University of California San Diego Microphysical climate-relevant properties of model marine aerosols explored with all-atom molecular dynamics simulations
Andrew Emerick - Columbia University Enabling Exascale Astrophysics: Galaxy-Scale Simulations with Enzo-E
Lakshay Gautam - University of Illinois at Urbana-Champaign Phonon Dispersion calculations of Graphene, Graphite and AB stacked Bi-layer Graphene
Forrest Glines - Michigan State University Magnetohydrodynamic Simulations of Turbulence in the Compressible Magnetized Taylor-Green Vortex
Alex Gurvich - Northwestern University GPU Accelerated Time-Dependent Chemistry in the Context of Galaxy Formation with WIND
Joshua Lansford - University of Delaware Electron Density-Based Machine Learning for Accelerating Quantum Calculations
Hui Li - Yale University Impact of air-sea coupling on the simulated global tropical cyclone activity in the high-resolution Community Earth System Model (CESM)
Matthew Liska - University of Amsterdam Pre-Exascale General Relativistic MHD Simulations of Ultra-thin Accretion Disks
Riccardo Longo - University of Illinois at Urbana-Champaign Unravelling Nucleon Structure using Petascale Computing Resources
John McGarigal - University of Arkansas Performance Analysis and Enhancements for Rayleigh-Taylor Instability Simulations
Lauren Pounds - Mississippi State University Analyzing the Importance of Model Physics in Simulating Environments Associated with Tornadogenesis Within Tropical Cyclones
Nicole Rosato - Rochester Institute of Technology Improved Trumpet Initial Lapse and Shift for Binary Black Hole Simulations
Swarnali Sanyal - University of Illinois at Urbana-Champaign Ozone and PM2.5 exceedance events: How air quality is evolving under changing climate
Micheline Soley - Harvard University Escaping from an ultracold inferno: computational chemistry at a new frontier in the ultracold KRb dimer reaction
Migle Surblyte - New Jersey Institute of Technology Using High Performance Computing to Investigate Blood Clot Formation
Ruji Darius Teo - Duke University Unraveling functional hole hopping pathways in the [4Fe4S]-containing DNA primase
Walter Torres - Duke University The transport and dynamics of wave-driven reef jets under the influence of stratification and rotation
Samuel H. R. Whitman - University of Colorado Boulder Simulation of Bluff-Body-Stabilized Flames Using PeleC, a Combustion Code for Exascale Computing
In this talk, I will describe an emerging initiative at Argonne National Laboratory to advance the concept of Artificial Intelligence (AI) aimed at addressing challenge problems in science. We call this initiative “AI for Science”. The basic concept is threefold: (1) to identify those scientific problems where existing AI and machine learning methods can have an immediate impact (and organize teams and efforts to realize that impact); (2) identify areas of where new AI methods are needed to meet the unique needs of science research (frame the problems, develop test cases, and outline work needed to make progress); and (3) to develop the means to automate scientific experiments, observations, and data generation to accelerate the overall scientific enterprise. Science offers plenty of hard problems to motivate and drive AI research, from complex multimodal data analysis, to integration of symbolic and data intensive methods, to coupling large-scale simulation and machine learning to drive improved training to control and accelerate simulations. A major sub-theme is the idea of working toward the automation of scientific discovery through integration of machine learning (active learning and reinforcement learning) with simulation and automated high-throughput experimental laboratories. I will provide some examples of projects underway and lay out a set of long-term driver problems.
Monday June 3, 2019 7:00pm - 9:00pm PDT
Great Hall
The National Geospatial-Intelligence Agency and its Work
The National Geospatial-Intelligence Agency is both a combat support agency under the United States Department of Defense and an intelligence agency of the United States Intelligence Community, with the primary mission of collecting, analyzing, and distributing geospatial intelligence (GEOINT) in support of national security.
In addition to using GEOINT for U.S. military and intelligence efforts, the NGA provides assistance during natural and man-made disasters, and security planning for major events such as the Olympic Games and Hurricane Katrina. NGA also engages with research and innovation to improve being able to address their missions, including a vision to evolve from maps to models of the world.
Tuesday June 4, 2019 8:30am - 9:30am PDT
Homestead
DNA double-strand breaks make up a vast majority of lesions a normal human cell undergoes every day. Here, we propose to employ graphene nanopore transistor to detect nicks in the DNA backbone efficiently using electronic sheet currents obtained across the membrane. For this purpose, we use molecular dynamics simulations accompanied with electronic transport calculations, data denoising and signal detection. While the ionic currents do not show any distinct signature from the nicked-site in the signal, a clear dip is seen in the transverse sheet current signal corresponding to the location of the breakage, thereby enabling us to detect and map these damages, electronically. For this research work, we rely greatly on supercomputers such as Blue Waters nodes to run our MD simulations and electronic transport calculations. We strongly believe such a detection mechanism enables the development of versatile sensor for early cancer detection caused by structural modification of the genome.
Tuesday June 4, 2019 10:30am - 10:50am PDT
Heritage
Modeling galaxies and the Intra-Cluster Medium (ICM) in galaxy clusters is a formidable challenge. The morphology of the galaxies, and the energy and metal content of the gas is ultimately controlled by star formation processes that happen on molecular cloud scales of less than a million solar masses. On the other, total cluster masses exceed 10^15 solar masses; hence a dynamic range in mass of over a billion is necessary for consistently modeling galaxies within this context. With Blue Waters we have tackled this challenge and produced the highest resolution hydrodynamic simulations of galaxy clusters run to date. The simulations have given us new insight into the connected evolution of the cluster galaxies, the ICM and the active galactic nuclei within the cluster.
Tuesday June 4, 2019 10:30am - 10:50am PDT
Homestead
Atomistic structures of protein capsids have been resolved for many viral species, but structural organization of their genomes still remains largely unclear. One such viral species is HK97 bacteriophage, for which experiments have characterized the packaging mechanism and resolved its protein capsid with atomistic resolution. Here, we report a computational reconstruction of the HK97 genome organization at atomistic resolution, obtained from a series of simulations gradually increasing in resolution. An all-atom explicit solvent molecular dynamics simulation was performed using an implicit representation for the capsid, allowing the internal pressure and ions to reach equilibrium. The simulation was continued with the explicit atomistic protein capsid, revealing the pattern of DNA-protein capsid interactions. Measurements of internal pressure throughout the coarse-grained simulation were consistent with experimental data, and the SAXS profile derived from the resulting structures matched experiment. The complete atomistic structure of a packaged virus particle uncovers the detailed genome-capsid interactions, offering exciting new avenues for the development of antiviral drugs.
Tuesday June 4, 2019 10:30am - 10:50am PDT
Landmark
Recent proof-of-concept demonstrations of artificial and bio-hybrid (partly synthetic, partly biological) miniaturized swimming robots have underscored their potential as diagnostic and therapeutic vehicles, as well as the lack and need of rigorous engineering methods for design and flow control. A brief survey reveals that most prototypes operate in flow regimes where viscous streaming can be leveraged. Streaming generates steady flows in response to body-fluid oscillations and offers powerful control options for transport, mixing, drug delivery, assembly. Here we present a novel approach to co-design bio-hybrid compliant mini robots and associated flow fields so as to control and manipulate the surrounding environment.
Tuesday June 4, 2019 10:50am - 11:10am PDT
Heritage
Developing codes that can take advantage of future generations of computing environments is essential to advancing our understanding of the Universe. The size and scope of simulations possible with current generations of Astrophysical codes are often hampered by substantial scaling limitations beyond 1,000 or 10,000 cores. The Enzo-E / Cello project seeks to push beyond these restrictions to develop an astrophysical hydrodynamics code capable of taking advantage of exascale systems. As part of a Blue Waters Graduate Fellowship I helped to develop necessary physics modules in Enzo-E (including star formation and stellar feedback) with the long-term goal of running large-scale cosmological simulations of galactic evolution. I will discuss this development, present initial scaling results for simulations of an isolated Milky Way mass galaxy, and discuss future plans and development work.
Tuesday June 4, 2019 10:50am - 11:10am PDT
Homestead
The rapid expansion of exotic saltcedar along riparian corridors has drastically altered landscape structures and ecosystem functions throughout the United States. Conducting the continental-scale monitoring of spatio-temporal dynamics of this invasive species over the past 40 years is essentially critical to understand its invasion mechanism. Previous studies indicated that the leaf senescence stage is the optimal time window to remotely monitor saltcedar distributions. However, the computational complexity in predicting the leaf senescence timing, along with the massive volume of satellite data in both spatial and temporal dimensions, makes large-scale invasive species monitoring prohibitive. This project aims to develop a parallel computational algorithm on Blue Waters that can accommodate the spatio-temporal variation in plant phenology to facilitate the continental-scale monitoring of invasive species dynamics. Results indicates that the Blue Waters provides unprecedented opportunities to achieve continental-scale monitoring of saltcedar distribution to revolutionize our understanding of saltcedar invasion processes.
Tuesday June 4, 2019 10:50am - 11:10am PDT
Landmark
Flame stabilization is important for maintaining steady combustion in gas turbines. However, computational combustion models do not capture the dynamic complexity of stabilized flames with sufficient generality, limiting attempts to improve turbine performance. To understand this complexity and provide validation data, we adapt PeleC, an exascale compressible combustion code, to Blue Waters to develop direct numerical simulations of stabilized reacting and non-reacting flows surrounding a triangular bluff body, currently under experimental investigation. Adaptive mesh refinement (AMR) is incorporated via the AMReX block-structured framework to locally resolve the physics of interest at reduced cost compared to static mesh approaches. We show results of a convergence study for the non-reacting case and preliminary findings from the reacting case, in which heat release interacts with shear-layer instabilities and vortex formation in the wake. AMR is necessary to accurately resolve the interactions among shear-layer, vorticity dynamics, reaction-rate chemistry, and flame formation at a computationally manageable cost.
Tuesday June 4, 2019 11:10am - 11:30am PDT
Heritage
I present new developments at the interface of deep learning, high performance computing, numerical relativity and electromagnetic surveys to push the frontiers of Multi-Messenger Astrophysics. I discuss the application of deep learning at scale to characterize binary black hole mergers through gravitational wave observations, developing the first generation of neural network models that are trained with over a trillion waveforms. It is shown that once fully trained, these state-of-the-art neural network models enable gravitational wave parameter estimation within a few milliseconds using a single V100 GPU. The combination of these algorithms with the construction of galaxy catalogs using deep learning algorithms is discussed in the contest of gravitational wave cosmology.
Tuesday June 4, 2019 11:10am - 11:30am PDT
Homestead
The hippocampus provides the basis for episodic memory in the brain, remembering events experienced in the past and linking them with their spatio-temporal context. Our project combines experimental data with computational modeling to elucidate how neurons in the hippocampus generate episodic memory sequences. We have constructed 1:1 scale computational models of two hippocampal subregions with detailed neuronal morphology and biophysics for most neuron types. Input to these models is provided by an idealized representation of the environmental information coded by cortical neurons projecting into the hippocampus, which matches the statistical features of experimental observations. Model parameters have been constrained so that simulations of neural activity during locomotion show highly sparse coding of position information that is specific to the hippocampus. Our simulations provide a complete picture of the known neurons within the hippocampus and allow the tracking of subpopulations representing individual trajectories and sensory cues associated with them.
Tuesday June 4, 2019 11:10am - 11:30am PDT
Landmark
Thunderstorms are responsible for causing severe winds resulting from tornadoes and strong downdrafts called downbursts. Over the past five years our group has conducted simulations of supercell thunderstorms at resolutions in which fully resolved tornadoes form. A file system utilizing lossy floating point compression of 3D arrays was developed in order to accommodate the large amount of data produced by these simulations. Due to the large data volume, new analysis techniques have been developed that leverage Graphics Processing Units (GPUs) to conduct fast quantitative analysis. One such tool for analyzing saved model data in a Lagrangian particle framework has been developed, allowing parcels of air to be tracked from simulation data in which data has been saved every model time step of 1/6s. In this talk, recent analysis of tornado and downburst producing thunderstorms will be presented, along with details on data analysis on the GPU.
Tuesday June 4, 2019 11:30am - 11:50am PDT
Heritage
Using our state-of-the-art code Fornax on Blue Waters we have simulated the collapse and explosion of the cores of many massive-star models in three spatial dimensions. This is the most comprehensive set of realistic 3D core-collapse supernova simulations yet performed and has provided very important insights into the mechanism and character of this 50-year-old astrophysical puzzle. I will present detailed results from this suite of runs and the novel conclusions derived from our new capacity to simulate many 3D, as opposed to 2D and 1D, full physics models every year. This new capability, enabled by this new algorithm and modern HPC assets such as Blue Waters, is destined to transform our understanding of this central phenomenon.
Tuesday June 4, 2019 11:30am - 11:50am PDT
Homestead
Multi-structured biomolecular systems play crucial roles in many cellular processes but have resisted traditional methods of structure determination, which often resolve only a few low-energy states. High-resolution structure determination of conformational ensembles using experimental methods that report on heterogeneity, such as double electron-electron resonance (DEER), remains extremely challenging. We have therefore developed a computational method to integrate these sparse, non-parametric experimental data to obtain estimates of conformational ensembles using molecular dynamics (MD) simulation. Existing methods of incorporation capture side-chain fluctuation rather than backbone conformational change; our method is particularly designed to address this outstanding challenge. Using Blue Waters, we have tested our method by incorporating DEER data obtained on the SNARE protein syntaxin-1a into MD simulation. We have found that our method of integration substantially outperforms existing state-of-the-art methods in capturing syntaxins open/closed conformational equilibrium. Our improved refinement method will greatly accelerate the structural understanding of such systems.
Tuesday June 4, 2019 11:30am - 11:50am PDT
Landmark
The project explores the response of global tropical cyclone activity to increased atmospheric carbon dioxide in the coupled Community Earth System Model (CESM), which in previous Blue Waters experiments has been shown capable of simulating realistic tropical cyclone circulations and global cyclone activity metrics (number, intensity distribution, seasonality, etc.). We performed two thirty-year CESM simulations, using the 25 km atmosphere coupled to the nominal 1-degree ocean model. In one simulation, atmospheric carbon dioxide is held fixed at pre-industrial levels, while the other simulation prescribes carbon dioxide levels equal to 4 times pre-industrial values. Here we outline key differences TC activity between model simulations, and we highlight the effects of enhanced temperature on large scale dynamics and tropical cyclones for different regions. Results shed important light on how tropical cyclones may change in the future under global warming using a state-of-the-art fully-coupled climate model.
Galaxy formation is quite numerically challenging, due to the wide range of physics and physical scales involved. In this talk I present two results from our collaboration’s simulation campaign to explore galaxy collaboration over the entire age of the universe on Blue Waters, which has spanned three PRAC grants, one GLCPC allocation, and the entirety of the Blue Waters lifecycle. The first result is the discovery of a new mechanism for the formation of supermassive black holes in the early universe (Wise et al. 2019, Nature), which unlike much previous work provides a natural explanation for the abundance of incredible massive black holes observed in the universe roughly a billion years after the Big Bang. The second result explores the diffuse circumgalactic plasma around Milky Way galaxies using incredibly high spatial resolution, and shows that this increased (and expensive!) resolution is crucial to resolving hydrothermal instabilities in this gas, thus understanding observations of present-day galaxies. These results are exemplars of the type of theoretical studies that are only possible on capability-scale supercomputers, and I close the talk by highlighting two codes, Enzo-E and K-Athena, that build on lessons learned with Blue Waters in order to be both scalable and highly performant on future exascale supercomputers.
Tuesday June 4, 2019 1:30pm - 1:50pm PDT
Homestead
We develop an Evolutionary Markov Chain Monte Carlo (EMCMC) algorithm for sampling from large, idiosyncratic, and multi-modal state spaces. Our algorithm combines the advantages of evolutionary algorithms (EAs) as optimization heuristics for state space traversal and the theoretical convergence properties of Markov Chain Monte Carlo algorithms for sampling from unknown distributions. We encompass these two algorithms within the framework of a Metropolis-Try Markov Chain with a generalized Metropolis-Hastingratio. We harness the computational power of massively parallel architecture by integrating a parallel EA framework that guides Markov chains running in parallel. Our algorithm has applications in many different fields of science.
Convectively-induced turbulence is an aviation hazard that is a forecasting challenge because operational weather models are too coarse to resolve turbulence. Turbulence indices are commonly used to aid pilots in avoiding turbulence through nowcasting and limited forecasting systems. Over tropical oceans, turbulence prediction systems rely heavily upon nowcasting where satellite observations identify hazardous regions near convection and diagnostics calibrated for midlatitude turbulence. In this study, seven high resolution simulations of tropical oceanic and midlatitude continental convection are performed to characterize the turbulent environment near various convective types. This study addresses the current shortcomings in turbulence diagnostics and determines the limitations of midlatitude continental diagnostics currently used for turbulence prediction over tropical oceanic regions. The environmental characteristics including stability, vertical wind shear, velocity in the vertical and horizontal, and temperature around convection during the developing and mature stages are analyzed to determine if stage specific thunderstorm guidelines are needed.
Plasma turbulence is ubiquitous in the Universe. In most space and astrophysical systems, turbulence dissipates energy and momentum via microscopic collective processes operating at small (kinetic) scales. Correct description of these processes is only possible via 6D Vlasov equation, whose high dimensionality coupled with large scale separation prohibits the use of many standard numerical tools for simulations with parameters typical for space environment. We describe a new method for the solution of the kinetic equations for a magnetized plasma. It is based on spectral expansion of the velocity space with Asymmetrically Weighted Hermite Polynomials (AWHP), together with discontinuous Galerkin (DG) approximation for coordinate space. The spectral AWHP expansion allows significant reduction in number of degrees of freedom required to represent velocity space, while still retaining kinetic effects. Moreover, the spectral expansion is isomorphic to classical fluid moment expansion, hence providing a fine control over fluid/kinetic regime transition with number of used polynomials/moments. DG approximation guarantees high accuracy (arbitrary order) required for turbulence simulation. At the same time, DG high locality ensures parallel efficiency leading to optimal scalability, which is desirable for large scale kinetic simulations on modern HPC architectures. Here, we demonstrate properties and capabilities of Spectral Plasma Solver framework based on DG approximation (SPS-DG) on classical example of decaying magnetized plasma turbulence, the Orszag-Tang vortex test. We further compare SPS-DG with fully more standard tools, such as Particle-In-Cell codes.
Tuesday June 4, 2019 1:50pm - 2:10pm PDT
Homestead
There are significant uncertainties in the climate and economic system. Integrated Assessment Models (IAMs) of climate and economy aim to analyze the impact and efficacies of policy responses to climate change. We develop and solve new computational IAMs with more than ten-dimensional continuous state space, which incorporate spatial temperature system, climate tipping points, economic risks, carbon capture and storage, and regional economic activities. We then analyze the optimal policy under uncertainty and risks and how such a policy impacts the economic activities. We find that tipping points and sea level rise significantly increase social cost of carbon (SCC), but efficient adaptation and carbon capture and storage can significantly decrease SCC, whileignoring spatial heat transfer leads to nonnegligible bias. Moreover, we solve dynamic stochastic cooperative and non-cooperative equilibrium and find that non-cooperation leads to much lower carbon tax and then much higher temperature in future.
I will discuss our implementation and then recent use of the Model for the Prediction Across Scales (MPAS). MPAS is a new, non-hydrostatic weather and climate model that allows for local grid refinement. Because MPAS is also a global model, it is well suited for extended range predictions, as will be illustrated with the MPAS predictions made during the RELAMPAGO field campaign. A configuration detail of particular relevance during RELAMPAGO was the specification of 3-km gridpoint spacing over the entirety of South America, with 15-km gridpoint spacing elsewhere around the globe. The 3-km spacing is considered to be "convection allowing", and thus we were effectively able to resolve thunderstorms over large domains.
Black holes are responsible for a wide variety of astrophysical phenomena. They devour stars, eject relativistic jets, affect star formation and galaxy evolution, and enrich the Universe with heavy elements. In many of these situations, the angular momentum axis of the infalling gas is tilted relative to that of the black hole to some degree. Particularly poorly understood is the physics of bright tilted accretion flows, such as those that power active galaxies, or quasars. Such disks are very thin, with height-to-radius ratio, h/r ~ 0.01, which makes simulating them a long-standing computational challenge. The unique combination of Blue Waters GPUs and our recently developed GPU-accelerated general relativistic magnetohydrodynamic code H-AMR allowed us to carry out the simulations of the thinnest disks to date, with h/r ~ 0.015 - 0.03. We show that contrary to the standard expectations, such disks break up into multiple individually precessing sub-disks, upending the standard picture of luminous black hole accretion. We discuss the observational and physical consequences of the disk breaking.
Tuesday June 4, 2019 2:10pm - 2:30pm PDT
Homestead
The coastal ocean includes diverse ecosystems encompasing both terrestrial and marine habitats that support approximately a third of the worlds population1.We are only beginning to understand the economic and environmental valueof these resources and how to protect these ecosystems in the face of climate change, sea level rise, extreme storm events, and increased human impact and pollution associated with population growth. These highly nonlinear systems and are difficult to observe, however the advent of numerical models and increased computational resources has made predicting the dynamics of these systems more accessible. An open question this research address is the required resolution to capture the shallow water dynamics, and how to couple these higher resolution models to coarser regional and global models. We employ a validated high-resolution numerical model of a New Hampshire estuary toestimate the nutrient loading from sediments in the estuary to the coastal ocean.
Arctic sea ice is an important component of the global climate system, which is highly sensitive to any internal and external forcings associated with and feeds back to amplify global climate changes. During recent decades, significant shrinking of Arctic sea ice extent and thinning of sea ice thickness have occurred. However, climate models have shown large discrepancies in simulating Arctic sea ice properties, which therefore hinders understanding and prediction of sea ice changes. In this study, we employed the sea ice-ocean component model of the Community Earth System Model (CESM 1.2) with the Elastic-Viscous-Plastic (EVP) rheology, forced by a repeating climatological forcing constructed using the ERA-Interim reanalysis dataset. We conducted a series of model sensitivity experiments and found that sea ice thickness distribution and associated sea ice motion and extent are highly sensitive to nonlinear interplays between intensity of wind stress and treatments of sea ice internal dynamics
Massive outflows of gas from galaxies known as galactic winds are crucial in regulating galactic star formation rates. These outflows are observed to consist of multiple phases, including both hot (~million K) and cold, molecular, gas. Previous simulations either lacked the resolution or the chemistry treatment necessary to capture these phases. This limits the ability of previous work to produce powerful galactic winds and also accurately predict observations of their chemical diagnostics. In response, we present WIND-CHIMES, a new GPU-accelerated version of the time-dependent chemistry code CHIMES that enables us to run high resolution simulations that, for the first time, will capture the full range of phases of galactic winds. We will show initial results of applying WIND-CHIMES to the context of galaxy formation, including convergence tests and speed comparisons along with a description of the new aspects of the code and its key features.
Tuesday June 4, 2019 2:30pm - 2:50pm PDT
Homestead
Current computing trends make the ensemble computational model highly relevant; it has the ability to overcome limitations of single task applications, and to achieve significant performance gains on large-scale parallel machines. Not surprisingly, the concept of running ensembles on large-scale HPC systems is thus gaining in importance. We discuss some of the challenges in executing ensembles at scale. We discuss the abstractions (pilot-systems), software systems (RADICAL-Cybertools) and execution models that we have developed to address many of these challenges. We will discuss how RADICAL-Cybertools, along with advances in statistical and adaptive algorithms have enabled ensemble-based applications to overcome limitations oftraditional single task applications. In spite of order(s) of magnitude efficiency gains, much greater improvements are still needed. We will close with a brief mention of novel application architectures that hold promise to provide the needed improvements.
Binary systems of supermassive black holes hold the key to resolving open questions regarding galactic evolution and massive black hole growth. Here, we present the first credible calculations of the light emitted by nearby gas as two supermassive black holes approach merger. By so doing, we will give observers strong clues about how to find such systems, potentially identifying examples well before their gravitational wave emission can be detected.The galactic environment of supermassive black hole binaries should, in many cases, supply ample gas to be accreted, liberating large amounts of energy in the process. This should produce photons spanning the spectrum from visible light to hard X-rays; the power output can be great enough that they can be seen from across the Universe. The magnetohydrodynamical equations of gas flows near binary supermassive black holes are extremely complicated. Fortunately, we have developed novel codes in the past few years on Blue Waters enabling us to realistically simulate these systems in 3-d using general relativity for the first time. Our simulations on BW provide the first astrophysically-accurate predictions of the time- and energy-dependence of the electromagnetic precursor to supermassive black hole binarie merger.
Tuesday June 4, 2019 3:30pm - 3:50pm PDT
Homestead
The study of the quantum properties of materials is all about understanding how collective behavior arises in systems with a large number of interacting particles (electrons and nuclei). In some materials these interactions give rise to unique correlated quantum states which are behind fascinating macroscopic phenomena such as magnetism or superconductivity. We will discuss a novel efficient way of probing correlated materials which relies on quantifying how electrons in a material respond to each other. In particular, we will see how known high-temperature unconventional superconductors such as the copper oxides, iron-pnictides and iron-chalcogenides have distinctive electronic correlations that set them apart from most other materials. Such observation allows for the construction of a data-based prediction model that might help in the search for new high-temperature unconventional superconductors.
Cosmological hydrodynamic simulations pose unique computational challenges owing to processes spanning many orders of magnitude in both space and time. In addition to solving the equations of gravity and hydrodynamics, these codes use analytic prescriptions to model processes taking place below the resolution of the simulation. Though implementation details differ, in large galaxies the results tend to converge. However, these prescriptions have not been tested in the low-mass limit. In new simulations capable of resolving the faintest known galaxies, we test how different star formation recipes affect the resulting galaxy distributions. Using Blue Waters we run different models, then compare galaxies across simulations. We find that the robustness of the models depends on the galaxy environment, with results being model-dependent in isolated regions but converged in dense environments. These results highlight the need for simulators to use caution when interpreting the results of their simulations in faint galaxies.
The investigation of ultracold chemical reactions requires a combination of experiment, theory, and computation. Computational simulation of many ultracold chemical reactions is currently beyond reach with time-dependent quantum mechanics. The memory requirements scale exponentially with the size of the chemical system. This scaling has been termed the "curse of dimensionality." Blue Waters offers the opportunity to break this curse. Parallelization allows a unique view into how the reaction proceeds by enabling simulation of thousands of trajectories with classical mechanics and diagonalization of large random matrices in time-independent quantum mechanics. By using insight from classical mechanics and R-matrix theory, we are employing Blue Waters to predict and analyze the results of the ultracold KRb dimer reaction. This research offers opportunities for applications from development of quantum computers to investigation of fundamental constants of nature.
Tuesday June 4, 2019 3:50pm - 4:10pm PDT
Homestead
Many ongoing and future experiments utilize light and medium mass atomic nuclei as laboratories for exploring the fundamental symmetries of nature and searching for signals of new physics beyond the Standard Model. Determining structure of these nuclei from first principles (or ab initio), that is, using realistic internucleon interactions derived from Quantum Chromodynamics considerations, represents a great computational challenge. We utilized the computational power of the Blue Waters to carry out large-scale first-principle modeling of a range of nuclei previously inaccessible to ab initio theory and experimental studies. We applied a highly scalable hybrid MPI/OpenMP implementation of an innovative many-body technique, that exploits dominant collective symmetries of many-nucleon systems, to provide nuclear structure information for isotopes from oxygen through titanium key to probing physics beyond the Standard model, including exotic unstable isotopes -- focus of next-generation large-scale nuclear experimental facilities for studying details of nucleosynthesis processes.
[Fe4S4]-containing proteins participate in charge transfer and related redox signaling relevant to cell function. The Y345C mutation in the DNA-binding domain of primase (which has been identified in gastric tumors) can affect the binding of the RNA/DNA duplex to the C-terminal domain of primase and alter charge hopping pathways between the [Fe4S4] cluster and the duplex. This in turn impacts the signaling dynamics and endogenous repair function of primase. In this study, we first developed a python module that is able to map and rank hopping pathways in proteins according to the mean residence time of the transferring charge in such pathways. We then parameterized the high-potential [Fe4S4] cluster in its 2+ and 3+ redox states for molecular dynamics simulations. The effects of the mutation on the structure, dynamics, hole hopping pathways between the [Fe4S4] cluster and the duplex, and DNA-binding free energy will be described. We will also examine the electrostatic contributions to the binding free energies of the duplex to primase in both redox states. This analysis aims to examine the molecular origins of the experimentally observed strong dependence of the DNA binding affinity on the charge state of the [Fe4S4] cluster.
Recent observational discoveries of lack of Lyman-alpha emitters near z~6 quasars raised questions about how quasars impact galaxy formation during the cosmic reionization. To accurately model this problem requires both a large cosmological simulation volume and high spatial resolution to resolve galaxies, as well as fully self-consistent radiative transfer (RT). Here I present the first attempt to model quasar proximity zones in full RT hydrodynamic cosmological simulations. We find that a quasar suppresses star formation in low mass halos (M<1e9Msun) through dissolving molecular hydrogen. However, due to the decreased stellar feedback, star formation catches up after ~50 Myr. The effect on the galaxy luminosity function is observable at faint magnitudes, but also depends on the quasar lifetime. This effect can be observed by the JWST in the near future, and can be used as a novel probe of cosmic reionization.
Tuesday June 4, 2019 4:10pm - 4:30pm PDT
Homestead
The quest to understand the fundamental building blocks of nature and their interactions is one of the oldest and most ambitious of human scientific endeavors. As the world's most powerful particle collider, the Large Hadron Collider (LHC) represents a huge step forward in this quest. The 2012 discovery of the Higgs boson demonstrates the great scientific value of the LHC and the Higgs particle itself represents a new tool for discovery. In this talk, we present our work using deep learning techniques on the Blue Waters supercomputing to develop and optimize a novel method of identifying the decays of highly-boosted Higgs bosons produced at the LHC a signature of new particles and/or phenomena at the energy frontier of particle physics. We also discuss our ongoing work using Blue Waters to develop scalable cyberinfrastructure for sustainable and reproducible data analysis workflows through the NSF-funded IRIS-HEP Institute and SCAILFIN project.
In collaboration with Falk Herwig’s team at the University of Victoria, in Canada, we have used Blue Waters for the last 6 years to simulate in 3D brief events in the deep interiors of stars in which new or additional fuel is ingested into a convection zone above a nuclear burning shell, giving rise to violent and aspherical nuclear combustion and associated nucleosynthesis. We have proposed to exploit the increased power of NSF’s follow-on to the Blue Waters system to expand our investigation to the study of interactions between multiple convection zones and their nuclear burning shells. A glimpse of our early test runs in gearing up for this expanded study will be shown and the key new numerical techniques that we will employ will be discussed. Most important among these new techniques is a hybrid advection scheme for concentrations of components of the gas in which composite fluids are advected with the full-blown PPB moment-conserving technique, while the various constituents of each such composite are advected with a less elaborate and less accurate PPM technique. One additional quantity is advected for each composite fluid as a new means of maintaining high accuracy while exclusively employing 32-bit precision arithmetic.
Tuesday June 4, 2019 4:30pm - 4:50pm PDT
Homestead
In microfluidic flows, fluctuations in stress and heat flux continuously drive the fluid out of local thermodynamic equilibrium (LTE). The stochastic Landau-Lifshitz Navier-Stokes (LLNS) equations have helped reveal how such fluctuations, amplified by nonlinearities, underlie unique phenomena such as giant fluctuations, enhanced diffusive transport, and nanojet droplet breakup. At yet finer spatial discretization, fluids appear granular and, with very few particles per grid cell, leveraging particle-based simulations seems simpler than justifying the usual constitutive laws (Newton's law of viscosity and Fourier's law), the continuum approximation, and LTE. Yet our work suggests that a fluctuating hydrodynamics model can not only be computationally viable and economical, but also physically preferable because its structure explicitly encodes the underlying transport physics. Our numerical model, HERMESHD, naturally extends LLNS and is seen to ultimately emerge from higher-moment approximations to the Boltzmann transport equation.
Frontera: The Next NSF Leadership Computing Resource.
Abstract: In this talk I will describe the new NSF-funded “Frontera” system, including the hardware specifications, software capabilities, and an overview of the entire project. I will also provide new updates on the system deployment, benchmarking to date, and (very) early science results. A discussion will follow of how users can get access to the new system, as well as contribute to planning for new system capabilities and support activities, as well as to requirements for the design of future NSF computing resources.
Wednesday June 5, 2019 9:00am - 10:00am PDT
Homestead
Controlled particle migration in a microchannel has important applications in separation technologies like filtration, cell sorting, and fractionation. It also has implications for physical processes like the margination of cells in the blood stream and for multiphase flows in geological formations (enhanced oil recovery). We investigated the migration of a small fluid droplet in a parallel-plate microchannel using large-scale dissipative particle dynamics computer simulations. In a Newtonian solvent, the droplet moved toward the channel walls, in agreement with theoretical predictions and recent simulations. However, the droplet focused onto the channel centerline in a viscoelastic complex fluid (dilute polymer solution). Focusing was typically enhanced for longer polymers and higher polymer concentrations, with a nontrivial flow-rate dependence due to droplet and polymer deformability, but the droplet position fluctuated because of Brownian motion. Such viscoelastic focusing is shown to be a viable method to manipulate small fluid droplets in microchannels.
Wednesday June 5, 2019 10:30am - 10:50am PDT
Heritage
The theory of plate tectonics has become the foundation of modern Earth sciences. However, many questions remain on the fate of subducted plates and their effects on the Earth's surface geology. Answers to this question begs for a quantitative understanding of the internal dynamics of the solid Earth, a process only becoming tangible in the past decade due to the fast development of supercomputing. At the University of Illinois, my group has been using Blue Waters to construct such quantitative and deterministic geodynamic models. By now, we have developed the largest in-scale models of its kind, with up to 10,000 MPI cores for a single simulation. Most of our models use data assimilation that merges data from geology, tectonics and geophysics into the same geodynamic framework. Notable examples include our simulation of subduction and convection on a global scale, mantle processes below Yellowstone, and the long-term evolution of continental lithosphere.
Wednesday June 5, 2019 10:30am - 10:50am PDT
Homestead
The renaissance in artificial intelligence and robotics combined with the rise of high-throughput omics and high-performance computing have put us on the verge of a paradigm shift in biological research. In this new paradigm, omics-based self-learning agents not only act as powerful predictors but also identify next set of experiments to be done by the robot. In this talk, we first present the Genetic Neural Network (GNN), an artificial neural network for predicting genome-wide gene expression given gene knockouts and master regulator perturbations. Then we introduce optimal experimental design framework to identify set of experiments that maximize prediction performance given gene expression data. Our methods result in approximately 40% improvement both in terms of prediction accuracy and number of data points necessary to achive same accuracy compared to common baselines. Validation of our methods was enabled by HPC parallel comuting platform using in vitro data sets and in sillico simulations.
Wednesday June 5, 2019 10:30am - 10:50am PDT
Landmark
The electronic density distribution completely specifies a chemical system's state and can be calculated using density functional theory (DFT). We are developing an atom-centered machine learning (ML) algorithm, trained on electron density, which can generate catalytic reaction mechanisms and kinetic models at reduced computational cost. The ML-based model will accelerate the three most computationally intensive components of the reaction energy profile: transition states, global minima, and entropy. This methodology will be the first to combine structural and density data to enhance ML convergence. Due to the atom-centered nature, the representation of the molecular systems will be invariant to rotations, translations, and reordering of atoms. Because we will use distances, partial charges, and atomic dipoles, the model will be generalizable to any size system.
Wednesday June 5, 2019 10:50am - 11:10am PDT
Heritage
The Southern California Earthquake Center (SCEC) used Blue Waters to perform physics-based probabilistic seismic hazard analysis (PSHA) using the CyberShake method which executes a complex suite of seismological software and scientific workflow middleware. Physics-based PSHA results are used to improve hazard estimates, which are used in building design, urban planning, community earthquake awareness, and disaster preparation. During the last year, we completed CyberShake Study 18.8, the first 3D physics-based PSHA model for the San Francisco Bay region. This study used over 3.7 million Blue Waters node hours to calculate hazard results for 869 locations in California, and produced hundreds of millions of seismograms for hundreds of thousands of individual earthquakes. These data products enable us to investigate the effect of basin structures and rupture directivity on hazard, improve upon standard attenuation-based methods of calculating seismic hazard, and identify research targets to further improve PSHA estimate accuracy.
Wednesday June 5, 2019 10:50am - 11:10am PDT
Homestead
Hepatitis B virus is a major cause of liver disease. Researchers are endeavoring to develop new treatments against the virus that target its capsid, a protein shell that encases the viral genome and drives its delivery to the host cell-nucleus. Numerous drugs that disrupt the capsid have been identiifed, but their mechanisms of action are incompletely understood. The PI has been utilizing Blue Waters as a computational microscope since 2015 to investigate the intact hepatitis B capsid and the effect of drugs on its structure, dynamics, and assembly. Here, the PI will describe her continued efforts to characterize the capsid as a drug target, including important new discoveries regarding drug binding modes, cooperativity in drug-uptake, and mechanisms of drug resistance. Importantly, the results produced by this study were inaccessible to experimental methods and were made possible only through molecular dynamics simulations of multimillion-atom capsid systems enabled by Blue Waters.
Wednesday June 5, 2019 10:50am - 11:10am PDT
Landmark
This project addresses the applied research goal of virtual certification pursued by NCSA Industry program and their academic and industrial collaborators, a key challenge in the design and manufacturing of gas turbine engines, in particular for aerospace applications. We are advancing the state of the art in computationally intensive implicit finite element analysis (FEA) by studying real-world jet engine models with LS-DYNA software, having as many as 200 million Degrees of Freedom (DOF), and identifying and removing performance scaling barriers. This is a continuation of our previous work on Blue Waters that demonstrated the feasibility of efficiently solving extreme-size real-world multiphysics problems at peta- and potentially exa-scale levels thus adding novel value to engineering research by enabling the creation of large-scale, high-fidelity models that yield accurate and detailed insight into the performance and safety of a proposed engineering design.
Wednesday June 5, 2019 11:10am - 11:30am PDT
Heritage
Earth's intrinsic magnetic field, called the geomagnetic field, is observed to vary on broad spatial and temporal scales, and is generated by vigorous convection in the Earth's liquid outer core (geodynamo). Understanding and prediction of such variations are very important for fundamental scientific research and for societal applications. This project, under the collaboration of NASA and NGA, aims at investigating the properties of covariances of large ensembles of geodynamo model solutions, and their implications to the forecast accuracies of geomagnetic secular variations on time scales of 5 to 20 years. The model used in this project is the GEMS (Geomagnetic Ensemble Modeling System) developed in NASA GSFC. It includes three components: a 3D geodynamo model to provide forecasts, the EnKF model to provide analysis with model forecast covariances, and the driver to manage interactions between the other two components. This project requires large amount of computing resources: for a single test with the ensemble size Ne and the spatial resolution N3, thus the total flops required are ~ 5 X 106 x Ne x N3. We plan to carry out three ensemble tests initially with Ne≥ 200 and N ~ 160. More tests will be carried out if needed. The final tested system will be used to generate candidate model to International Geomagnetic Reference Model (IGRF) and World Magnetic Model (WMM).
Wednesday June 5, 2019 11:10am - 11:30am PDT
Homestead
The membranes that surround cells and organelles play a vital role in biological function, acting as semipermeable barriers that separate and protect cells and organelles from their surroundings. Molecular dynamics (MD) simulations provide the means to investigate the fundamental molecular mechanisms that give rise to membranes' biological functions, but it is exceedingly difficult to build MD models that capture cellular membranes' immense scales and complex geometries. To address this challenge, we previously introduced software called xMAS (Experimentally-Derived Membranes of Arbitrary Shape) Builder, and we used it to generate a ~4.5 billion atom (~1.9) model of a piece of the endoplasmic reticulum with a helicoidal shape called a Terasaki ramp. We also ran a long-term equilibrium MD simulation on a smaller test system to demonstrate that xMAS Builder produces stable, high quality models. Used to develop and test the base capabilities of xMAS Builder, these previous models were somewhat limited in that they were composed entirely of lipids. We now introduce an updated xMAS Builder capable of incorporating proteins into its membrane models.
Wednesday June 5, 2019 11:10am - 11:30am PDT
Landmark
Continuous casting is a mature and sophisticated technological process, used to manufacture over 96% of the world's steel. Thus, even small improvements to understanding of defect formation in the process can have huge impact on the steel industry. Due to the harsh environment of the process and many process-variables, computational modeling is an ideal tool to quantify the complex defect-related phenomena. This project aims to develop sophisticated and accurate multiphase models of the process for more realistic simulations and scientific investigations of the formation of various defects. Transport and capture of over 5-million particles and micro-scale steel-solidification structures are simulated together with multiphase turbulent flow in a huge caster-domain (>2m3), using both the multi-GPU in-house code, CUFLOW on Blue Waters XK-node and the commercial CFD-code ANSYS-Fluent with numerous user-defined functions on XE-node. Speed-up breakthrough (>3000X) on Blue Waters is greatly contributing to conducting very high-resolution simulations of numerous cases simultaneously.
Wednesday June 5, 2019 11:30am - 11:50am PDT
Heritage
The physical mechanisms that generate intermediate-depth earthquakes (70-300 km) withinsubducting slabs are not well understood, because the processes that enable shallow earthquakesare prohibited at extremely high temperatures and pressures. Because there can be large intraslab earthquakes in regions without mapped faults, it is important to better elucidate their mechanisms to better understand ground motions and hazards. Seismological studies of intermediate-depth earthquakes primarily focus on analyzing data assuming a very simple sourcemodel, but the physics of these models are based on shallow earthquake physics and intermediate-depth earthquakes have shown signatures of very complex ruptures. We attempt to create better physical models of intermediate-depth earthquakes using the SORD dynamic rupture code on Blue Waters. We show the results from source inversions of large intermediate depth earthquakes in Japan, as well as the effects of modeling these earthquakes with alternate source models, such as pulse-like ruptures.
Wednesday June 5, 2019 11:30am - 11:50am PDT
Homestead
The human immunodeficiency virus (HIV) uses enclosing capsid (CA) proteins to package and release its genetic information into newly infected host cells. CA forms fullerene cones that compartmentalize viral RNA. TRIM5 is an important antiretroviral restriction factor found in rhesus macaques that blocks HIV infection. Recent cryo-electron tomography analysis demonstrate cytoplasmic TRIM5 self-assembles into hexagonally patterned nets that encage mature capsids, with a symmetry and spacing that matches the underlying CA lattice. In this work, we use coarse-grained and atomistic simulations to probe the structural and thermodynamic properties TRIM5 assembly and the molecular mechanisms of how TRIM5 recognizes and restricts viral capsid.
Wednesday June 5, 2019 11:30am - 11:50am PDT
Landmark
Controlling fluid transport at nanoscale is crucial to enhance the performance of several applications such as water treatment and bio-electronic devices. Understanding this transport requires multiscale modeling to allow better integration of the associated multi-physics. In this talk, we focus on two different systems. First, we present a new power-law relation between the ionic conductance and concentration in CNT. This system is modeled using both continuum and molecular dynamics approaches in OpenFOAM and LAMMPS, respectively. The obtained results are validated against experimental measurements. Second, we present ab initio molecular dynamic modeling, using CP2K package, of water transport in graphene nanochannel with a slit gap equivalent to the carbon inter-atom distance (~3.4 ). We show how the chemistry of graphene surface plays a role in determine the water properties. In each part, we show the scaling of the computational tools and the unique calculations that were performed using Blue Waters.
Wednesday June 5, 2019 1:30pm - 1:50pm PDT
Heritage
The challenge of our project is to perform multi-scale simulations of the solar wind (SW) from the solar surface to the boundary of the heliosphere, and its interaction with the local interstellar medium (LISM). Our simulations have both fundamental and practical applications: (1) they involve discontinuous flows of partially ionized, collisional and collisionless plasmas, and (2) help interpret spacecraft observations and predict hazardous conditions affecting humans and electronics in space. Our accomplishments: (1) we removed the heliospheric contribution to the TeV cosmic ray (CR) anisotropy observed in the Tibet air shower experiment and identified a source of the original anisotropy; (2) developed a data-driven model for coronal mass ejections; and (3) studied the SW-LISM interaction effect on CRs. MS-FLUKSS solves self-consistently the system of MHD equations for the ion-electron mixture, kinetic Boltzmann equation for neutral atoms, and two separate systems for PUIs and turbulence generated by them.
Wednesday June 5, 2019 1:30pm - 1:50pm PDT
Homestead
From 2013-2018, molecular dynamics and enhancing sampling simulations of nucleic acids have allowed us to assess, validate, and improve the force fields for nucleic acids. Using various model systems, we have demonstrated through various ensemble simulations and replica exchange molecular dynamics (REMD), including multidimensional replica exchange (with temperature and Hamiltonian exchanges), the ability to reproducibly converge, from different initial conditions, the conformational ensembles of various DNA duplexes and mini-dumbells and of various RNA dinucleotides, tetranucleotides and tetraloops. We have also better refined NMR models to their published experimental data, explored salt dependence of DNA duplexes and their interactions with various ligands, and demonstrated that the use of minimal experimental restraints can improve fidelity with experiment. All of this has been enabled by advancements in the AMBER biomolecular simulation codes including excellent performance on the Blue Waters GPUs.
Wednesday June 5, 2019 1:30pm - 1:50pm PDT
Landmark
The physics of a plume or jet entering a quiescent fluid is a classical problem in fluid dynamics at both small and geophysical scales. In the coastal ocean, jets are particularly ecologically and physically important; they facilitate the exchange of heat, salt, nutrients, and larvae between the nearshore environment and open ocean. Here, we investigate the behavior of a momentum jet driven by the interaction of surface gravity waves with coral reef topography using a 3D coupled wave-circulation numerical model of an idealized island domain. Simulations conducted over a range of background density stratification and rotational (Coriolis force) conditions are used to characterize possible flow regimes and dynamical balances. The results highlight the role of stratification and rotation in significantly modifying the spatial structure of reef jets, as well as identifying the leading-order importance of wave-current interaction for jet dynamics in the near-surface.
Wednesday June 5, 2019 1:50pm - 2:10pm PDT
Heritage
Our understanding of the star formation process has reached the point where advanced observational capabilities and high-resolution simulations that include the appropriate two-fluid physics of giant molecular clouds (GMCs) are required. We have shown that while magnetic fields grow exponentially in a fully ionized plasma, they grow more slowly in a partially ionized. Another part of our research consists of realizing that most astrophysical systems are spherical and should, therefore, be simulated on meshes that are optimal for such a mesh. We have developed high order MHD for such meshes. Several large-scale simulations have been run or are running on Blue Waters, and key results have been published. The present work is a substantial improvement over previous work in terms of resolution and in the details of input physics and accuracy. Developing geomesh MHD and publishing it was also a prime accomplishment. GPU ready codes have also been developed via an attendance at an NCSA Hackathon.
Wednesday June 5, 2019 1:50pm - 2:10pm PDT
Homestead
The essential conundrum of modern biology, namely the question of how life emerges from myriad molecules whose behavior is governed by physical law alone, is embodied within a single cell — the quantum of life. The rise of scientific supercomputing has allowed for the study of the living cell in unparalleled detail, from the scale of the atom to a whole organism and at all levels in between. In particular, the past three decades have witnessed the evolution of molecular dynamics (MD) simulations as a "computational microscope", which has provided a unique framework for the study of the phenomena of cell biology in atomic (or near-atomic) detail. Here I present an overview of our use of Blue Waters, starting from the early science days until the present date, to determine the molecular details during the life-cycle of multiple infectious diseases.
Wednesday June 5, 2019 1:50pm - 2:10pm PDT
Landmark
Numerical simulations of turbulent mixing play an important role in understanding the chaotic mixing dynamics driven by the fluid interface instabilities and in predicting the growth rate that describes the outer edge of the mixing zone. The essential features of the algorithm used in our numerical studies are front tracking, to achieve resolution of a steep and sharp discontinuity in density gradients and Large Eddy Simulations with subgrid scale to model the diffusive transport corrections to the mesh averaged Navier-Stokes equations. The focus of this project is to improve the scalability of the front tracking code for turbulent mixing simulations on HPC systems. The performance measurements and analysis using TAU and CPMAT allow us to observe the performance on Blue Waters, identify the most computationally expensive parts and enhance the performance of collective operations and high order accurate weighted essentially non-oscillatory numerical scheme.
Wednesday June 5, 2019 2:10pm - 2:30pm PDT
Heritage
For years, extreme scale High-Performance Computing has been used to investigate cosmology --the origin and development of the universe via simulations.Optical Astronomical Surveys, such as the Dark Energy Survey, collect data used to compare cosmological simulations with observation based measurements. The Dark Energy survey recently completed six years of observations primarily targeted at the phenomena known as dark energy, the continuing acceleration of the universe. Blue Waters is a crucial infrastructure used for the most ongoing demanding tasks to derive physical measurements from the observations of the Dark EnergySurvey. My goal for attendance --to continue to work/ understand issues with HPC machines in the context of experimental, rather than simulation science.
Wednesday June 5, 2019 2:10pm - 2:30pm PDT
Homestead
Extending the size of systems for which electronic structure can be computed has been an ongoing effort at different levels of electronic structure theory. Within tight-binding, system sizes are limited by the linear algebraic problem of computing the density matrix whose rank increases with the number of electrons. In recent work, an implicit algorithm that scales linearly with system size, and relies entirely on sparse matrix-vector multiplications (SpMVs) was presented. Now, using the PETSC matrix library optimized for Blue Waters, it is possible to simulate 10^6 atoms in a few minutes (<100 nodes). Simulations approaching 10^7atoms have been demonstrated, and further increase in system sizes by one or two orders of magnitude is also possible. Electronic structure simulations of this size allow us to simulate microscale systems with atomic resolution in particular, we are interested in microscale plasma generators, which have several technological applications.
Wednesday June 5, 2019 2:10pm - 2:30pm PDT
Landmark
In past research studies, rotation of turbulent flows has been shown to provide a turbulence suppression. The axially rotating pipe is an exemplary prototypical model problem that exhibits these complex turbulent flow physics. For this flow, the rotation causes a region of turbulence suppression which is particularly sensitive to the rotation rate and Reynolds number. The physical mechanisms causing turbulence suppression are not well-understood, and a deeper understanding of these mechanisms is of great value for many practical examples involving swirling or rotating flows, e.g. swirl generators, wing-tips, axial compressors, hurricanes, etc. DNS at moderate Reynolds numbers and several rotation rates were conducted. Turbulence statistics are used to quantify the effects of rotation on turbulence. Finally, an overview of the second phase of this research project is provided where the entire reverse transition process is simulated on considering ~30 billion points as part of the recently awarded NSF-PRAC 2019 allocation.
Wednesday June 5, 2019 2:30pm - 2:50pm PDT
Heritage
Antiferromagnetic materials (AFM) have fascinating properties like robust properties against external stimulations and absence of stray fields, which makes them interesting candidates for future memory devices. To understand the properties of AFMs, two different studies are implemented involving Blue Waters: First, voltage-induced switching of antiferromagnetically ordered topological orthorhombic CuMnAs is explored. Model Hamiltonian theoretical studies indicate that a chemical potential change can induce the reorientation of the Néel vector and transition from semi-metallic to semiconducting. First-principles density functional theory (DFT) is performed to verify model Hamiltonian analysis in orthorhombic CuMnAs. Second, optical properties and heat capacity of metallic antiferromagnetic Fe2As are analyzed. In collaboration with experiment, the magneto-optic response of Fe2As to ultrafast temperature excursions is investigated. Using electron and phonon dispersion from DFT, the electron and phonon heat capacity was obtained and subtracted from the total heat capacity from experiment, to obtain the magnon heat capacity.
Wednesday June 5, 2019 2:30pm - 2:50pm PDT
Landmark
We present new initial lapse and shift by constructing analytic formulas that mimic their expected settled behavior in the moving puncture gauge. These initial values allow for improvements in the initial shape of both the lapse and the shift. Current simulations set initial β = 0 and initial lapse to be isotropic asymptotically; our new initial construction of α sets these initial values to be closer to the settled shape of α and β for boosted, spinning black holes. To evaluate the effects of this initial gauge choice on evolutions, the Hamiltonian and Momentum constraint violations are monitored. This method shows improvements in a simulation of an unequal-mass, q = 3, nonspinning system and also in an equal-mass system with spins a = 0.8. We expect that this method will reduce the amount of time required for the settling of the lapse and shift, and will make simulations of both high mass-ratio and near-extremally spinning systems more efficient.
Wednesday June 5, 2019 2:30pm - 3:00pm PDT
Homestead
I will present the results of our PRAC work on Blue Waters evolving the collapse and explosion of massive stars as core-collapse supernovae using large, long-running, three-dimensional, radiation hydrodynamics simulations. I will focus on the diversity of outcomes and how they are tied to the variety of input conditions that represent stars of different masses and compositions that experience core collapse throughout the Galaxy and Universe, and also show how these simulations fit into a long-term simulation program and can be used for years to come.
Wednesday June 5, 2019 2:50pm - 3:10pm PDT
Homestead
