Performance Analysis and Prediction

Performance and cost are the two primary characteristics for comparing different approaches to solving a computing problem.  Both characteristics can be considered along many dimensions, such as computation throughput and communication latency (performance), and power draw and price (cost). Performance and cost are cross-cutting characteristics that apply to all research directions in computer science.

Understanding the performance and cost of a particular approach is the first step towards improving it, but understanding performance and cost is often challenging.  For approaches that have been implemented on existing systems, obtaining such understanding may require measurement and analysis.  For scenarios where the hardware or software under consideration does not yet exist, performance prediction using analytical modeling or simulation may be necessary.

The ORNL Future Technologies Group includes a team of members investigating techniques for understanding performance and cost, with special focus on high-performance computing environments and on emerging computing technologies. Our projects include both empirical approaches for performance measurement and analysis, and performance prediction using modeling and simulation.

Projects

Performance analysis and prediction impacts all of our group's work, from early evalution studies to parallel I/O.  For work that deals specifically with performance analysis and prediction techniques, we are pursuing several directions.  For some projects, we have publications and even code for download.  For other projects that aren't yet that far along, we encourage you to check back for project updates.   Some of the work described here has been done as part of a series of projects funded through the Department of Energy Office of Science's Scientific Discovery through Advanced Computing (SciDAC) program, namely the Performance Evaluation Research Center (PERC, SciDAC-I), the Performance Engineering Research Institute (PERI, SciDAC-II), and the Institute for Sustained Performance, Energy, and Resilience (SUPER, SciDAC-III).

Publications on our performance-related work are available from our Publications page. 

ASPEN

  • ASPEN is a domain-specific language for performance modeling.
  • ASPEN builds upon ideas from the Modeling Assertions work done previously by members of the Future Technologies Group.  With Modeling Assertions, symbolic expressions that describe the performance of a code are added as annotations to that code, and then checked at run time, similar to the way logical assertions (i.e., the assert macro) are used in a C program.

MIAMI

  • The Machine Independent Application Models for performance Insight (MIAMI) toolkit is a collection of tools for automatic construction of application centric, single node performance models.
  • MIAMI aims to compute detailed and accurate models of the execution behavior of applications. Accurate models of application execution behavior have many uses: 1) identify performance bottlenecks in applications; 2) understand the causes of performance inefficiency and provide hints about code transformations for addressing them; 3) estimate the potential for performance improvement from code or architectural transformations; and 4) predict execution behavior on proposed future architectures.
  • At its lowest level, MIAMI uses static and dynamic analysis of fully optimized x86-64 application binaries. Working on application binaries, as opposed to source code, offers several advantages: 1) language independence: tools can naturally work with applications written in different languages; 2) code coverage: working on binaries avoids running into blind spots when applications are linked against third party or system libraries available only in binary format; 3) capture optimization effects: tools can analyze and model the performance of optimized code without perturbing the optimization process.

Scalable Tools

  • To be effective for understanding the performance of applications and systems software on large-scale computing systems, a tool must scale at least as well as the system itself. This scalability must be present in all aspects of the tool, from data collection to data analysis to its presentation of results.
  • We are exploring not only the use of overlay networks like the MRNet tool infrastructure software for implementing scalable tool communication and data manipulation functionality, but also the impact of such networks in high-performance computing environments where scalable tools are most needed. This work also considers techniques for visualizing performance data from high-end computing systems.
  • For details on some of our work in this direction, see the PPoPP 2006 paper "On-line Automated Performance Diagnosis on Thousands of Processes" and "Exploring Tree-Based Overlay Networks on the Cray XT", presented at the 2007 Dagstuhl seminar on Code Instrumentation and Modeling for Parallel Performance Analysis.
  • We also help maintain the mpiP lightweight MPI profiling tool, with special focus on making mpiP effective on the leadership computing platforms at ORNL.

Questions?

To ask questions about our research into performance analysis and prediction, ask us using our contact web page.