Extra-P—Empirical performance modeling made easy

1 Introduction

High-performance computing (HPC) systems are designed to solve big problems fast. Scaling laws are essential to understanding the extent to which an application can solve a big problem and/or exploit a big system. For example, suppose the runtime of an application scales quadratically with the input size [i.e., O(n²)]. In that case, input scaling will soon hit the limits even of the largest systems available, rendering the power of HPC little more than a drop in the ocean. Similarly, if the execution time of a weak-scaling experiment keeps growing significantly as new compute nodes are added to the execution configuration, we encounter diminishing returns of parallelism.

Such scaling bottlenecks hinder scientific progress and can waste enormous resources. Unfortunately, they are often discovered only when the critical problem or system size is tried in practice. Until the developers succeed in removing the bottleneck, the expensive system will remain poorly utilized. Since improving scalability may require profound algorithmic changes, if not even substantial algorithmic research, a remedy usually takes time and expertise, a combination not every developer team can afford easily. This is why scaling bottlenecks should be discovered as early as possible—ideally, already during the initial phases of the development.

One instrument for understanding the scaling behavior of HPC applications is a performance model. A performance model is a formula that expresses a critical performance metric, such as runtime, as a function of one or more execution parameters, such as the degree of parallelism and input size. Performance models offer quick insights on a very high level of abstraction, including predictions of future behavior. For example, the function t = c₁+c₂·log(p) characterizes an application whose execution time grows logarithmically on top of a constant base as new processors are added, a benign weak-scaling scenario. Essentially, a performance model is a complexity expression plus constants and coefficients. Deviations in actual performance from what the model predicts can point to unforeseen limitations of the hardware or errors in the model. The main goal of such a model is not to predict the exact runtime at a different scale—although it is one of the most challenging tasks, so it is often a test used in evaluating such models—the main goal is to identify performance bottlenecks, both relative to other parts of the code and compared to the developer expectations.

Performance models can be created in two different ways: analytically or empirically. Analytically means reasoning about the properties of the underlying algorithms, which is challenging for complex applications with thousands of source code lines (Hoefler et al., 2011). In such cases, performance analysts usually restrict themselves to a few selected kernels that they believe will dominate performance at scale. This poses the danger of overlooking some scaling bottlenecks because recognizing dominance at scale is non-trivial. Strictly speaking, identifying the right kernels requires the performance model one is supposed to come up with. Even the creation of the model for a selected kernel, is at least error-prone, and can sometimes even prove impossible. For example, the performance behavior of library calls often remains opaque. Finally, reasoning alone can hardly determine constants and coefficients. Some experimentation is still needed. In contrast, empirical performance modeling, the process of learning such models from performance data, offers a convenient alternative because it can be largely automated and does not require intimate knowledge of the source code.

In this paper, we review the status of the Extra-P project¹, which has been launched to develop a user-friendly tool that implements this method (Calotoiu et al., 2013). Our workflow is summarized in Figure 1. When initiating this effort around 2012, tools for measuring and analyzing measured performance were widely available (Shende and Malony, 2006; Knüpfer et al., 2008; Adhianto et al., 2010; Geimer et al., 2010), while performance modeling was still confined to a community of experts (Kerbyson et al., 2001; Petrini et al., 2003; Pllana et al., 2007). Our goal has therefore been to democratize performance modeling via automation. However, as we continued, we encountered several challenges along the way. The two most prominent are the cost of the experiments needed to generate the underlying data and noise, which can severely affect the accuracy of the model. On the other hand, we also found novel ways to apply this technique to previously disregarded use cases. Overall, our survey covers the following contributions:

• Introduction to Extra-P and its software architecture.
• Explaining the performance model normal form and other underlying core principles of Extra-P.
• Strategies to lower the cost of the required experiments.
• Strategies to counter the effects of noise.
• Non-standard use cases.

Despite being a mature tool by now, we conclude the paper with an outlook on how Extra-P could further evolve, especially given the increasing diversification of hardware and the rapid advances of AI technology.

2 Software architecture

Extra-P is a comprehensive and flexible performance modeling tool built entirely in Python. This enables the usage across platforms, thereby supporting Linux, Windows, and macOS. Extra-P consists of six core components (see Figure 2) that work together to provide performance modeling capabilities in different scenarios. The Extra-P experiment is the central component, which describes all data relevant to one particular experiment: the parameters, call tree, measurements, and the resulting models, as well as the modeler configurations. To get measurements into the experiment, one uses one of several file readers that load data from a variety of formats. Once measurements are input, one can apply one of the modelers, which generates models based on the input measurements. Depending on the exact requirements, one can choose and configure the individual modeler. For example, if one expects a segmented model (see Section 3.4), it is helpful to choose a modeler that is capable of producing segmented models. The complete experiments can be stored as .extra-p files using the experiment IO module, so that they can be reused at a later time. The interaction with all of these modules happens through either a command-line interface (CLI) or a graphical user interface (GUI) based on Qt for Python. The CLI serves as the primary interface for scripting and direct use on the cluster, where GUIs are typically hard to use. In contrast, the GUI offers a user-friendly front-end that allows the interactive exploration of models across the call tree. Extra-P captures the aggregate effects of both computation and communication, including architecture-dependent factors, implicitly through the measurement data. Thus, while the model itself does not explicitly encode architecture features such as cache size or topology, these effects are reflected in the empirical data used for model generation.

2.1 File import

Extra-P supports a variety of input formats. The most important ones, in practice, are the CUBE profile format (Geimer et al., 2010) produced by Score-P (Score-P developer community, 2023) and the JSON lines format. To load data from CUBE profiles, one needs to organize the individual profiles per measurement point into a directory structure that indicates the coordinates of the different parameters. The folder names must follow this pattern: <PREFIX>.<PARAM1><VALUE1>.⋯ .<PARAMn><VALUEn>.r<REPETITION> so that Extra-P gets to know the parameters and the location of the measurement points. In contrast, the JSON lines format contains all information (measurement values, call paths, parameters, and coordinates) in one file. Besides these two formats, we support the following other formats:

• text files,
• JSON files,
• experiments from Extra-P version 3 and older.

Support for more formats can be easily added by creating a new class that implements the FileReader interface, reads the data from the new format, and returns an Extra-P experiment.

2.2 Model generators

Extra-P's model generators are based on the methods and techniques described in Sections 3, 4, and 5. They provide the core functionality of the modeling process: translating measurement data into performance models. We distinguish two cases: single-parameter and multi-parameter experiments. The multi-parameter modeling process works by dividing the modeling problem into a single-parameter modeling problem per parameter. After applying a single-parameter model generator to construct one model per parameter, the resulting models are then combined to generate a multi-parameter model. This approach is the implementation of the multi-parameter search in Section 3.2, which additionally enables a modular construction of performance models using specialized single-parameter modelers.

In contrast to using just one multi-parameter modeler, we have multiple single-parameter modelers that implement the individual modeling approaches described in Section 3. The most important one is the basic modeler, which implements the basic strategy for generating single-parameter models. Based on this modeler, a number of additional modelers were created that improve on the methodology: The refining modeler implements the iterative refinement (see Section 3.3). The segmented modeler implements segmented modeling as described in Section 3.4. The adaptive modeler uses DNNs to generate good models under noisy conditions, as explained in Section 5.1.

2.3 Command-line interface

The command-line interface (CLI) of Extra-P is mainly intended for scripting and usage directly on the cluster. It provides text-based access to Extra-P's functionality: importing measurements, generating models, and outputting models in the form of various text-based reports or an .extra-p file for later usage in the GUI. The minimal example for using the Extra-P CLI is the following command:

extrap <FURTHER_OPTIONS> (--cube | --text | --talpas | --json | --extra-p-3 | --experiment) <PATH_TO_MEASUREMENT_DATA>

As part of the further options, one can specify the modeler to use via the --modeler <MODELER> option. The modeler can then be configured by passing a list of key–value pairs --options <KEY=VALUE,...>. The output of the tool can also be adjusted via the --print option; one can select which part of the experiment to output: everything, just call paths, just metrics, the functions, etc. All configuration options can be found using extrap --help.

Overall, the CLI provides most of Extra-P's functionality; compared to the GUI, it is mainly missing the abilities of interactive exploration (e.g., interactive visualizations of the model) that are available in the graphical user interface.

2.4 Graphical user interface

Extra-P's graphical user interface provides users with an interactive environment for the generation and exploration of performance models. The GUI based on Qt for Python works on all supported platforms and is designed to support novice users as well as experts by providing useful defaults and detailed advanced configuration possibilities. As shown in Figure 3, the core of the interface is a modular layout that integrates various tools and views for generating and analyzing performance models based on measurement data. The process begins with the import of measurement data via an import dialog, which is then automatically followed by generating performance models with the default configuration. Once these two steps are completed, the user can explore the performance models in the experiment.

The metric selection allows the users to choose for which metric (such as execution time or FLOPs) they want to see the performance models. The call-tree view offers a hierarchical representation of the program's calling structure. This view enables users to navigate through different routines and code paths, providing access to performance models at each node of the call tree. This structure is essential for browsing the experiment and locating performance issues. On the left of the view, the severity indicators give a quick estimation of how much the call path impacts the current effort metric. The specific execution configuration—for example, a target number of processes or threads—for which the severity should be displayed can be set using the severity target configuration in the lower left.

Once a metric and a call path are selected, the corresponding performance model can be visualized in the form of various graphs in the model visualization area in the middle. The graphs show how performance changes with varying parameters (e.g., numbers of processes), making it easy to observe trends such as superlinear growth. The available graphs vary from line and contour plots to surface plots and heatmaps; their plotting range can be adjusted using the graph limits controls at the bottom. Of course, graphs can be exported for use in presentations and reports.

If the default modeling settings do not achieve the desired results, the modeling configuration panel on the right allows the tuning of the modeling process, from simply choosing another model generator via the modeler selection up to advanced configurations, like a custom exponent search space. As part of the uniform software architecture, all options for the model generator are available in the GUI.

A particularly helpful feature of the interface is the measurement-point suggestions panel, which implements the GPR experiment selection strategy explained in Section 4.3. This component analyzes the current model and data coverage and offers recommendations for additional measurement points that could improve model accuracy. For instance, if the current performance model has high uncertainty in a particular region of the configuration space (e.g., for very high or very low process counts), the system may suggest additional runs at those configurations. These suggestions guide users in efficiently allocating measurement resources, reducing the need for trial-and-error experimentation, and ensuring more robust performance modeling.

3 The performance model normal form

Analytical performance modeling is a powerful method of determining the scalability of applications; unfortunately, between the expertise required and the time needed to craft such models manually, this approach is limited to small kernels that have already been identified as being performance-critical.² This can easily lead to scalability bottlenecks developing and remaining undiscovered for a long time, as the focus remains on kernels considered performance critical (Calotoiu et al., 2013). To address this issue and simplify the creation of performance models, we have developed a tool that generates empirical models from a small set of experiments (as few as five experiments!).

There are two underlying insights that we leverage in our approach:

• The space of function classes that are commonly found to represent both communication and computation times as a function of program execution parameters, such as the number of processes or the size of the program input, is limited to polynomial and logarithmic terms, or combinations thereof.
• Analyzing the application at a fine-grained level, such as individual function calls, allows the individual performance contributions of each input parameter to be measured even when only a few measurements are used.

Codifying the above insights into a mathematical expression, we introduce the Performance Model Normal Form (PMNF), as described in the Equation 1:

f(p)=∑k=1nck·pik·log2jk(p)(1)

The default values for these models are n = 5, I={02,12,22,32,42,52,62}, and J = {0, 1, 2}. Scalability worse than p³·log²(p) is not distinguished. This choice reflects common behavior encountered in parallel programs, but the search space can and should be tailored if expectations regarding application behavior are already known.

The PMNF is the core of our method, as it provides us with a sufficiently expressive search space to capture scalability behaviors, but can nevertheless be traversed in a reasonable time. We will now go into detail as to how performance models are generated, starting from the collection of performance measurements to presenting the user with a ranked set of kernels. The process is summarized in Figure 4. We start with the collection of performance profiles, such as with Score-P (Score-P developer community, 2023). Score-P is a profiling and tracing tool that instruments a parallel code at compilation and measures its performance during execution. We gather at least five data points by varying a parameter of interest while keeping everything else constant. For each data point, measurements should be repeated depending on system noise until the confidence interval is within an acceptable range (Hoefler and Belli, 2015). We then generate for each kernel in the call tree an individual model by trying to find coefficients for all possible complexity expressions spanned by our PMNF using linear regression. We select the best model, using cross-validation to avoid over-fitting, and then rank the kernels according to their asymptotic complexity.

The computational cost of performance modeling using this approach is divided into the cost of gathering performance experiments and the cost of model generation. Throughout our studies, the cost of model generation has consistently been negligible, and is at most measured in seconds, even when using a single core. The cost of gathering performance experiments depends on the application. Since our method should provide the same result regardless of the value of the parameters being varied, users can choose a set of values that provides the smallest overall costs—such as running the smallest five possible parallel configurations.

3.1 Evaluation

To evaluate our approach, we have first reproduced the analytical models by Hoisie et al. (1999) and Wasserman et al. (2000) of the Sweep3D neutron transport problem. Parallelism is expressed by a wavefront computation along diagonal lines through the input grid. This induces a waiting time in the communication of the application, which scales with the square root of the size of the input grid, thus explaining our empirical models, confirming the previously computed analytical ones, and showing how precise Extra-P modeling can be, as can be seen in Figure 5. Table 1 shows that the prediction errors, even at a scale of more than 100x, remain small. We can thus create powerful models using a small number of small experiments that remain useful even at much larger scales. Another insight we gained by comparing the predictive power of performance models trained using runs up to 2,048 processes with models trained with data up to 8,192 processes is that while the absolute error of the models slightly improves, it is only the coefficients of the terms and not the terms being selected that slightly change. The only exception is the behavior of MPI_Send, whose small dependence on process count is not captured using 2,048 processes. This is crucial, as it means Extra-P can correctly capture the asymptotic behavior.

Table 1. The most time-consuming Sweep3D kernels (i.e., call paths) ranked by their predicted execution time at the target scale p_*t* = 262, 144 processes.

The values and models reflect exclusive execution times without callees. The predictive error applies only to p_*t*. On the left we used training data with up to 2,048 processes (p_*t* = 128·p_max), on the right with up to 8,192 processes (p_*t* = 32·p_max). The overall prediction error for the application as a whole is within the confidence interval of those measurements.

We then explore the performance and scalability of the HOMME dynamical core of the Community Atmospheric Model (CAM). This is an important scientific application developed at the National Center for Atmospheric Research (NCAR), which uses spectral element and discontinuous Galerkin methods on a cubed sphere. The code is written with scalability as a primary goal, and was confirmed to work well up to 100,000 processes.

Table 2 lists the most time-consuming kernels of HOMME. A surprising discovery is that the MPI_Reduce call scales significantly worse than linearly with the number of processes. The fact that this particular routine is part of the initialization of HOMME—and thus outside the code that was even considered by the developers to be performance critical underscores the need to examine the entirety of scientific applications for scalability bottlenecks. Our model and predictions are confirmed by large-scale performance experiments shown in Figure 6. Subsequent analysis by the application developers revealed the scalability bug was caused by not adapting the number of MPI processes assigned to the initial I/O to the total number of MPI processes used by the HOMME. The HOMME case study shows the utility of empirical performance modeling as a tool to uncover latent scalability bottlenecks without requiring a time-consuming analytical modeling effort that only a small number of experts are qualified to perform.

Table 2. Models of the kernels of HOMME derived from smaller and larger-scale input configurations.

The predictive error refers to the target scale of p_*t* = 130k. The overall prediction error for the application as a whole is within the confidence interval of those measurements.

3.2 Multi-parameter search

A natural expansion of the original performance model normal form is to include multiple parameters, as shown in Calotoiu et al. (2016).³

f(x1,…,xm)=∑k=1nck·∏l=1mxlikl·log2jkl(xl)(2)

While it would be possible to consider separate search spaces for each individual parameter, in practice, we default to using a shared search space for all parameters. Thus, the exponents i_{k_*l*} and j_{k_*l*} are chosen from sets that can be defined as in the one-parameter case. Unfortunately, the combinatorial explosion of the search space would lead to over four billion model hypotheses being evaluated for our default set of exponents and only three parameters being considered. On commodity compute devices, a rate of 300,000 hypotheses being evaluated per second would mean each function and metric in the call tree would need multiple hours to produce a model. This motivates us to explore the use of heuristics to reduce the search space to something practical and useful. A first insight that we leverage is that for each parameter, one term is sufficient to describe the behavior. Furthermore, unless many data points can be provided (over a hundred in our experience), allowing for more than one term can be detrimental, as it leads to overfitting more often than not.

3.2.1 Golden section search

The heuristic we use to speed up the search is a modified golden section search. We base our choice on the following insight: The powerful combination of linear regression and cross-validation will find the best coefficients to “bend” any polynomial and logarithmic curve to fit a set of data points. However, the closer the curve is to the actual behavior, the smaller the error will become. Therefore, we can order all hypotheses by the magnitude of their first derivative at the observation with the largest input value, and then traverse this space looking to minimize the error. An established approach toward traversing such spaces is the golden section search, which selects two points in the search space using the golden ratio ϕ = 1.618 between the start and end of the search interval, and then restricts the search interval to the one between the two points with the smallest error as seen in Figure 7. While Bayesian optimization (Onorato, 2024) could also be used for this purpose, the monotonicity of the space makes golden section search a simpler and more efficient path to the solution.

Extending this approach to multiple parameters then becomes as simple as determining the best single parameter models and then combining them to decide whether an additive or multiplicative combination better fits the measurements. These two simplifications reduce the search space from billions of measurements to less than a hundred for a three-parameter model and thus make it eminently tractable.

3.2.2 Evaluation

We have evaluated our heuristics using synthetic functions by providing 100,000 samples of different complexities and comparing the accuracy of the heuristic and exhaustive approaches in asymptotic exactness, as can be seen in Table 3. To note, the ground truth samples are generated using terms available to the modeling approach. The evaluation shows that the heuristic differs from the exhaustive approach in less than 0.6% of cases, and even then, both approaches at the very least identify the correct lead-order term. Therefore, we conclude that this approach offers a powerful solution for fast multi-parameter modeling.

Table 3. Evaluation of multi-parameter modeling heuristics using synthetic functions.

3.3 Iterative refinement

In this subsection, we explore an evolution of the single- and multi-parameter search discussed earlier. The idea is to iteratively refine the terms of the model until a fit is found (Reisert et al., 2017) instead of searching across a given space of models defined by a fixed set of terms. A previous study by Calotoiu et al. (2016) has shown that finding multi-parameter models can be reduced to combining the best single-parameter models; therefore, we focus below on iteratively refining single-parameter models. One significant benefit of this approach is that it can be used as a drop-in replacement in the multi-parameter scenario.

3.3.1 The SMAPE metric

The default quality metric we used earlier to compare the quality of generated models and model hypotheses is the Residual Sum of Squares (RSS). Given N experiments with measurements y_*i* (1 ≤ i ≤ N) for the parameter values x_*i*, the RSS of a model f(x_*i*) is calculated as ∑i=1N(yi-f(xi))2. However, the resulting RSS value depends on the scale of the data being modeled and the squaring of the residuals amplifies this problem. The Symmetric Mean Absolute Percentage Error (SMAPE) metric (Kreinovich et al., 2014) offers a way to mitigate this problem and facilitate a better evaluation metric for the iterative refinement approach.

SMAPE is a scale-independent, relative error metric that overcomes the shortcomings of the RSS. It originates from time series forecasting and is defined as:

1N·∑i=1N|yi-f(xi)|(|yi|+|f(xi)|)/2·100%.

Essentially, it is the mean of a ratio, expressed as a percentage that is always in a range of 0% to 200%, where 0% means no error at all and 200% means the prediction is infinitely far off compared to the actual value. This means it can be easily interpreted by users and compared across models with different scales. Unlike the slightly simpler MAPE metric, SMAPE does not break down when any y_*i* = 0.

3.3.2 Revisiting the PMNF

Previous work that we conducted (Calotoiu et al., 2013; Iwainsky et al., 2015; Shudler et al., 2015; Vogel et al., 2015) led us to the observation that the PMNF can be simplified to fit the common use case better. In doing so, we also increase its resilience to noise. We suggest, therefore, to use the following simplified PMNF:

f(x)=c0+c1·xα·log2β(x)

The main challenge is to select the best exponents α and β, where α = 0 and/or β = 0 is allowed. The reason is that once α and β have been fixed, optimal values for c₀ and c₁ are determined by regression. Observing real applications in numerous studies (Calotoiu et al., 2013; Iwainsky et al., 2015; Shudler et al., 2015; Vogel et al., 2015) led us to the conclusion that we can restrict α < 6 and β < 3. Further analysis by Reisert et al. (2017) helped us narrow this further and identify four one dimensional slices β = 0, β = 1, β = 2 and α = 0.

Along each of these slices, we can now search for appropriate values of α (for the slices where β is fixed) or β (where α is fixed), respectively. In this context, appropriate, does not necessarily mean optimal. What we aim for is to find exponents that are representable by preferably simple fractions. Focusing on irreducible fractions, a fraction is simpler than the other whenever it has a smaller denominator (Guthery, 2011, p. 163). Below, we present an algorithm to find such exponents.

3.3.3 Iterative refinement

The algorithm works by starting with integer exponents (i.e., fractions with denominator 1) and then iteratively refining the search space by increasing the denominators while approximating the true minimum. In a somewhat simpler form, refinement of the search space was previously proposed by Shudler et al. (2015) who suggested repeated halving of an initial interval. That led to denominators that were always powers of two, which were insufficient in some cases. For example, a computational kernel simulating a three-dimensional process can actually require an exponent of n3 to model its complexity.

To understand the algorithm, let us look at the slice β = 0, where no logarithmic term is involved and the algorithm tries to find an appropriate value for α. Later, we shall expand on how the algorithm deals with multiple slices. First, all hypotheses with integer exponents α = 0, …, 5 are computed and compared. The exponent leading to the best model is stored, and its successor and predecessor are used as initial upper and lower bounds, respectively. Following this initialization step, the actual iterative refinement process, presented in Figure 8, starts. In essence, it is a variant of the golden section search (Press et al., 2007), but uses the mediant rather than the golden section to determine new candidate exponents. The mediant of two fractions n1d1 and n2d2 is defined as n1+n2d1+d2 and has the property that it always lies in between the two original fractions (Guthery, 2011). For example, the mediant of 12 and 1 (i.e., 11) is 23.

In every iteration, two new hypotheses are computed from the two mediants in between the currently best hypothesis and each of the two bounds, and their errors are compared to the best hypothesis. If the left mediant has the smallest error, we cut off the right part of the search space by using the left mediant for the new best hypothesis. If the right mediant is the winner, we do the same on the other side. If none of the mediants has a smaller error, then the best hypothesis remains the same, and we use the two mediants as new upper and lower bounds.

We aim to keep the number of iterations small (i.e., 1 to 3) so that the exponents remain readable and more intuitive. At this stage, the SMAPE metric comes to our help and offers a way to make decisions based on the relative improvement of its value. This means that the termination criterion in our algorithm is the rate of the SMAPE improvement. The outline of the full algorithm is as follows

Step 1. For each slice, find and remember the best integer exponent.

Step 2. Refine each slice according to the previously described method. However, the iterative refinement iterations need to proceed in tandem across all slices. It means all slices execute one iteration of the algorithm before the next iteration is started. After each iteration the best hypothesis across all slices is determined and when the rate of the SMAPE improvement sufficiently decreases (termination threshold) we terminate the iterations.

Step 3. The last step is to determine whether the winner hypothesis from the previous step is sufficiently better (non-constancy threshold) than the constant model. If it is not, the hypothesis is rejected in favor of the simpler constant model.

3.4 Segmented modeling

One fundamental underlying assumption of Extra-P is that there exists only one behavior in the data that is to be modeled. Unfortunately, from MPI implementations that change their underlying algorithm depending on process count or message size or scientific applications that use different numerical algorithms for certain ranges of inputs, this assumption is not always true in practice.⁴ This can, unfortunately, result in incorrect models that can mislead users, as can be seen in Figure 9.

3.4.1 Detecting segmentation

We start from the observation that models generated from input data representing a single behavior will differ from models created from input data representing multiple behaviors. We thus focus on subsets of the input data and compare the models derived—if they are mostly identical apart from some variation in the coefficients, this suggests there exists only one underlying behavior. If the models derived from different subsets vary wildly, this suggests either that the behavior cannot be expressed by the PMNF or that there is more than one underlying behavior.

In this work by Ilyas et al. (2017), we focus only on identifying behaviors that are piecewise continuous, therefore creating segments of the input space that are explainable by the PMNF. Therefore, we move a sliding window across the data points and use the differences between models to signal the change in behavior. We also define a specialized error value for the subset models, derived from the Residual Sum of Squares (RSS). We call this error metric the normalized residual sum of squares (nRSS), and compute it by dividing the square root of the RSS by the mean of the points used to generate the models:

nRSS=RSSȳ

For the data from Figure 9, we get six models which are shown in Table 4. The models combining data points from different behaviors have a significantly higher nRSS. To identify the change point we simply look for the center point(s) within the range of models with a higher nRSS than their neighbors, as can be seen in Figure 10. While the detection of segmented models is a niche application of Extra-P, it is a useful tool when modeling MPI functions, as can be seen in the next section.

Table 4. Subsets created for the data from Figure 9, their respective models, and their nRSS values.

Heterogeneous subsets are highlighted.

3.5 Expectation validation

In this subsection, we explore a use case for the PMNF-based single-parameter modeling algorithm. The problem this use case addresses is making sure the scalability of HPC libraries, as modeled from real performance data, corresponds to user expectations. We, therefore, propose a lightweight framework that allows systematically validating performance expectations (Shudler et al., 2015, 2019). The output is a set of divergence models that explain discrepancies between expected and actual performance as process counts grow. Users need only provide asymptotic growth rates as expectations. The rest is either fully automated or partially automated by the framework, depending on the library being validated and the availability of an accompanying profiling layer. The framework targets both library development and algorithm engineering by bridging the gap between theoretical design and practical implementation, and has been successfully applied to MPI libraries, the MAFIA algorithm, and parallel sorting benchmarks, showing its broad utility and scalability.

The methodology for using the proposed framework is illustrated in Figure 11 and comprises the following four steps: (i) define expectations; (ii) design benchmark; (iii) generate scaling models; and (iv) validate expectations. The first two are manual because they involve user decisions, while the second two are automatic. We describe each of them in detail below.

3.5.1 Define expectations

The goal is to keep the method simple yet effective enough to provide valuable insights to the users. By asking users to define expectations in big O notation (aka Landau notation) we make it easier for users to apply the framework on libraries and algorithms they do not know very deeply, but are still familiar with. For example, many users will be able to specify that a floating point sorting library has a runtime expectation of O(nlogn), but will not be familiar with the implementation details. Even if users have only a hypothetical expectation for some function it is still good enough. The empirical model produced by the framework can then become the new expectation. This is especially useful when the user has little knowledge of the library or during regression testing when the main task is to prevent later modifications from introducing scalability bugs.

Part of the expectations definition step is choosing the actual functions that the framework will benchmark. The more functions the user selects the more expensive it will become to construct the benchmark, which is why it can make sense to restrict the selection to those deemed most relevant. However, making too narrow a choice poses the risk of overlooking hidden scalability issues. Another concern is choosing the relevant scaling metric. In many cases, the runtime or floating-point operations is the metric of choice. It is important to remember that some metrics are more prone to jitter, which will result in noisy results and less accurate models.

3.5.2 Design benchmark

In this step, the user needs to either develop a thin benchmarking layer for the selected functions in the previous step, or, more likely, use Score-P (Score-P developer community, 2023). The important point is that the benchmark (or the profiler) needs to measure the selected performance metrics for the selected functions in various execution configurations (e.g., different numbers of processes or input sizes).

In some circumstances, in the face of unexpected architectural constraints for example, the user will be advised to re-implement one or more representative library functions in a way that has been proven to show the expected behavior under ideal conditions. One way to think about it is like a calibration process that establishes a baseline (Shudler et al., 2015).

3.5.3 Generate scaling models

For the model generation, we use the generator, along with the iterative refinement algorithm discussed earlier in this paper. The input to the model generator is a set of measurements whose precise nature depends on the scaling objective (e.g., number of processes vs. input size, weak vs. strong scaling). The output is a set of empirical models for each of the functions the user selected in the first step.

3.5.4 Validate expectations

The expectations are provided in the big O notation. This means that we need to transform the generated models accordingly by first isolating the leading-order term in a model and then stripping off its coefficient.

To help users understand the results, we propose to use a divergence model that is defined as δ(x) = G(x)/E(x), where G(x) is the generated model and E(x) is the expectation provided by the user. This model characterizes the degree of divergence between the expectation and the observed behavior. It can also be used to visualize the severity of the deviation.

Thus, the output we present to the user consists of G(x), δ(x), and a match rank with three possible indications: total match [meaning G(x) corresponds to E(x)], approximate match (G(x) is within the deviation limits), and no match (G(x) is outside the deviation limits).

Extreme divergence points either to a potential issue in the algorithm, an implementation bug, a constraint of the underlying architecture, an unrealistic expectation, or a combination of several factors. Furthermore, undersubscription, when available computational resources are purposely underutilized, is not considered in this study. It has been seen (Prichard and Strasser, 2024) that this phenomenon can have a significant impact on scaling behavior. It can therefore be difficult to identify the root cause of such divergent behavior.

3.5.5 Evaluation

Table 5 presents an excerpt from the MPI collective operations evaluation by Shudler et al. (2015). The table shows models on three different HPC systems with the match criteria in the brackets right after the models.

Table 5. Generated (empirical) runtime models of MPI collective operations on various HPC systems alongside their theoretical expectations.

Excerpt from earlier studies by Shudler et al. (2015, 2019).

4 Experiment selection

The largest cost in empirical performance modeling remains gathering empirical measurements. Even allowing for small-scale experiments and needing only a few data points per parameter being studied, the cost of parallel execution reliably overshadows the cost of modeling by multiple orders of magnitude. As Extra-P has grown, we have explored different avenues toward reducing the number of experiments the user needs to provide. In this section, we will explore the different approaches proposed to reduce this burden.

4.1 Taint modeling for experiment selection

HPC applications can involve many different execution parameters related to the problem size, parallelism size, and application-specific configuration.⁵ This variety and number of parameters present a fundamental challenge for performance modeling: researchers must determine which input variables actually impact the performance, and which are merely numerical parameters affecting the result but not influencing the speed of execution. Black-box modeling all parameters is infeasible due to the problem of combinatorial explosion: since any two different parameters can potentially influence an execution simultaneously, we need to vary their values independently. For example, to model the LULESH proxy-app with the two most important parameters—parallelism and problem size—we need five values per parameter, leading to 25 different samples. If we consider including one additional parameter in the modeling, such as the number of regions or iterations, the number of samples grows to 125. Thus, without a systematic approach, HPC experts must resort to their own intuition and application knowledge, or accept the very high cost of performance modeling.

4.1.1 Static analysis

The dependencies can be identified through code inspection, but this requires expertise from users and significant manual effort, which limits the applicability of automatic performance modeling. Instead, the influence of parameters on the codebase can be analyzed automatically with the help of compilers. Static analysis can be applied to the application's source code, helping to identify which input parameters influence specific functions and loops. This crucial information can be used in the modeling process to prune false dependencies. However, static analysis encounters fundamental roadblocks both in theory and implementation. First, constructing an analysis that can generalize to all types of computations and data movement is infeasible in practice, as it has been shown in the halting problem (Church, 1936) and Rice's theorem (Rice, 1953). Furthermore, practical HPC applications contain various abstractions and indirections that obstruct analysis. While the introduction of object-oriented programming in C++ helps to design and maintain large projects, it can hide the direct relationship between data allocation, initialization, and later read and write operations. With the arbitrary pointer arithmetic and memory aliasing, large C/C++ applications are often too complex for static analysis to generate precise and accurate results.

4.1.2 Dynamic taint analysis

Dynamic program analysis overcomes the limitations of static tools by conducting an instrumented execution of the application to analyze the program behavior. Taint analysis propagates user-defined taint labels during the execution of the program, allowing for the study of how chosen variables impact the results of computation. Practical implementations rely on allocating shadow memory, which mirrors the actual memory space of the application, and which is used to keep track of the label association with each memory address. Dataflow analysis tracks the propagation of tainted values from the operands of each operation to the result, e.g., the taint label of a multiplication result is a union of the taint labels of both inputs (Figure 12). However, tainted information can be propagated through control-flow dependence, e.g., in the case of loop iteration variables. In such cases, it is also necessary to propagate taint information through conditional operations, such as branch instructions.

4.1.3 Perf-taint

To expand the capabilities of Extra-P with the results of program analysis, we proposed Perf-Taint, a taint analysis tool built to understand the impact of performance parameters on the application (Copik and Hoefler, 2019; Copik et al., 2021). The new tool has two major goals: first, we want to understand which functions and loops of the applications are affected by any of the performance-critical parameters. This information enables the removal of instrumentation and modeling for these parts of the application, which are not expected to change when varying input parameters deemed relevant to the scalability of the HPC application. The second goal is understanding which specific parameters impact the code regions, which can be used by Extra-P to remove models with false dependencies.

Perf-Taint is built on top of LLVM (Lattner and Adve, 2004), a modular compiler framework that uses a common Intermediate Representation (IR) for an optimization infrastructure shared between different frontends, such as Clang (C/C++) and Flang (Fortran). Users define input parameters through a custom tainting API (Figure 13), and they compile their application with LLVM while applying Perf-Taint instrumentation. The instrumented application is executed, resulting in a list of all distinct functions with the effect of specific parameters. Extra-P later consumes this list to improve experimentation and modeling. For example, the automatic generation of performance models can disregard overfitted function candidates that include nonexistent parameter dependencies.

At the compilation time, Perf-Taint removes from analysis all functions that perform only a constant number of operations at runtime, i.e., they have no loops or all loops have a constant iteration space. We use dfsan, the dataflow taint analysis provided in LLVM, and extend it with support for control-flow taint propagation. We examine the remaining functions and instrument loop exit conditions by querying the taint label; the presence of a taint marker in either the iteration variable or boundary value indicates that the number of loop iterations depends on the parameter tied to the taint marker. For each function, we store the full call path with all parameters, which enables the tool to determine when invocations of the same function with different arguments result in different performance behavior.

Perf-Taint also supports libraries like MPI, which propagate the parameter dependency through their internal state that is invisible to the tainting problem. For each call to an MPI function, we introduce parameter dependency in the result and insert taint values when necessary. For example, for collective MPI operations, the performance model of the function call can depend both on the number of MPI ranks and the size of the data argument. Thus, the introduced parametric dependency encompasses both the implicit parameter related to the number of MPI ranks and additional parameters obtained by querying taint labels of the size parameter.

4.1.4 Evaluation

Tainting results allow us to understand which parameters impact the program, which helps users find parameters to use for modeling, given the limited computing budget and inability to consider all input parameters.

Table 6 presents the distributed impact of program parameters on LULESH (Karlin et al., 2013) and MILC (Bernard et al., 1991). With the results, we deduce that program parameters describing problem size and the number of MPI ranks are the best candidates for performance modeling with two parameters.

Table 6. The number of functions and loops whose performance depends on the value of the program parameter; results presented in Copik et al. (2021).

Furthermore, Perf-Taint accumulates parameter dependencies within a function: independent loops with different parameters in the same function show an additive dependency, i.e., the performance model of the function should be composed as a sum of two functions. For such functions, performance models can be created by considering experiment samples that do not consider all combinations of parameter values. On the other hand, nested loops can introduce multiplicative dependency, which requires all combinations to derive an accurate performance model. In the case of LULESH, we detect an instance of the parameter iters that shows a multiplicative dependency on all other parameters in the main computational loop. In this case, the parameter can be removed from modeling since it will likely not create interesting insights into the performance of LULESH, helping to reduce the dimensionality of experiments.

4.2 Sparse modeling

In the following, we introduce a different, more general heuristic to reduce the required number of experiments without relying on code analysis. To create a performance model, Extra-P's PMNF-based modeling approach, which we introduced in Section 3, requires five distinct values per model parameter and all their combinations. Thus, it requires an m-dimensional dense matrix consisting of 5^*m* measurement points, where m is the number of considered parameters (Calotoiu et al., 2016). Figure 14 illustrates such a dense matrix of measurement points for a two-parameter scenario. In this case, Extra-P needs a total of 25 unique measurement points (all points shown in the Figure) to create a model. To counter the effects of run-to-run variations, experiment configurations are often repeated multiple times. Based on our observations, we recommend five repetitions per measurement point (Calotoiu et al., 2013). This raises the amount of points required for modeling to 5^*m*+1.

The reason why such a large number of measurement points is required is that the modeling process begins with single-parameter models. For each parameter value, results are averaged across all measurements where that value occurs. This corresponds to projecting the full parameter space onto a single parameter. To ensure that this projection reflects only the effect of the chosen parameter, the influence of all other parameters must remain constant, which requires identical combinations of their values.

The exponential increase in measurement points required per model parameter leads to several problems. First, in practice, it is often too expensive to conduct such a large number of experiments, especially for three or more model parameters. Second, it can be challenging to gather a set of measurements, where for each parameter value all combinations of all other parameter values are available. Lastly, it is often difficult to identify a range of parameter values such that the smallest problem still provides enough work to efficiently utilize the largest number of processes, while the largest problem still fits into the memory of the smallest number of processes.

To tackle these challenges, we proposed a novel modeling approach that requires only a polynomial number of measurement points 5·m+1 per parameter (Ritter et al., 2020). Since this approach constructs models from only a subset of the dense matrix points, we call it the sparse modeler. The sparse modeler is based on the assumption that each parameter exhibits a single, consistent behavior across the entire measured space. If this assumption holds, the same functional terms describing a parameter's effect should be identified regardless of which sequence of five measurements is selected, provided that the effects of all other parameters are held constant. Under this premise, it is sufficient to select just five measurement points per parameter to construct single-parameter models. We validated this assumption through synthetic performance experiments. Consequently, instead of measuring every combination of the five values per parameter, only a minimal set of five points per parameter needs to be measured. These points vary a single parameter while keeping all others fixed, minimizing measurement cost.

Figure 14 illustrates a set of measurement points sufficient to model the effect of the number of processes p on application performance (black circles) and a set of five points for modeling the effect of problem size (gray squares). However, any line of five points that fulfills the requirements could be chosen. These sequences may also overlap, as shown in the figure. Because Extra-P allows parameter effects to be either additive or multiplicative, at least one additional measurement point outside these sequences is required to determine whether the effects of the two parameters are independent or compounded. The same principles apply if more than two parameters are modeled.

The flexibility in selecting measurement points, when using the sparse modeler, raises the question of which points should be chosen to minimize modeling cost while preserving accuracy. To address this, we trained a reinforcement learning agent to learn cost-effective combinations of measurement points. Using an artificial evaluation environment, the agent was trained on hundreds of thousands of synthetic performance functions. The knowledge acquired by the agent was then used to derive a generic measurement point selection strategy applicable to any modeling problem. The strategy follows this principle: first, measure the cheapest available sequences of five points per parameter, including one additional point to determine whether parameter effects are independent or compounded. These measurements are used to construct an initial model using the sparse modeling technique. Subsequently, additional measurements are performed, starting from the cheapest available points. The previously determined model is used to assess whether its quality is sufficient, has improved, or whether further points are required. The model is then updated with all available points, and the process is repeated until the desired model accuracy is achieved or the measurement budget is exhausted. Because this heuristic always selects the next cheapest point, we refer to it as the cheapest points first (CPF) strategy.

We evaluated the sparse modeling technique combined with the CPF measurement point selection strategy using synthetic analyses and three application case studies. Figure 15 presents the results of the synthetic evaluation, comparing the accuracy of Extra-P's state-of-the-art modeling approach with that of the sparse modeler using the CPF strategy. To assess the sparse modeling approach, we conducted experiments varying the number of model parameters (m = 1, 2, 3, 4) and the number of repetitions per measurement (rep = 2, 4, 6 left, middle, and right bar) across 100,000 synthetic performance functions. The x-axis in Figure 15 shows the number of points used for modeling, while the y-axis indicates the percentage of accurate models. We consider a model as accurate if its prediction for a parameter-value combination outside the training range lies within ±5%, ±10%, ±15%, or ±20% of the expected value. The evaluation points were derived by selecting, for each parameter, the next parameter value outside of the training range.

The results presented in Figure 15 show that for single-parameter models, the sparse modeler preserves the accuracy of the original approach. For two parameters, sparse modeling substantially reduces measurement cost while maintaining satisfactory accuracy: using less than 13% of the original cost, 82% of models remain within ±5% error and 86.5% within ±10%. For three parameters, the cost savings are even greater—using only 15 points of the 125-point dense matrix, 74% of models achieve ±5% accuracy while requiring just 1.8% of the original modeling budget. For four parameters, the effect is even more pronounced: 90% of models reach ±20% accuracy using only 4.74% of the previous budget (250 points). When four or more parameters are considered, noise has a stronger impact, making the effects of some parameters harder to distinguish from run-to-run variations. Increasing the number of repetitions improves accuracy by mitigating these noise effects. Our case study analyses showed similar trends; a detailed discussion can be found in Ritter et al. (2020).

In summary, it is possible to reduce the modeling cost of Extra-P by one or even two degrees of magnitude while only giving up single-digit percentages of accurate models. On average, the sparse modeling approach in combination with the CPF strategy reduces the modeling cost by 85% while retaining 92% of the model accuracy. Furthermore, we recommend repeating each performance measurement four times to counter the effects of system noise (Ritter et al., 2020).

4.3 Gaussian process regression

While the CPF strategy presented in Section 4.2 substantially reduced the average modeling cost compared to a dense matrix of measurement points, our evaluation showed that its generic applicability came at the expense of yielding suboptimal solutions for specific modeling problems, particularly in complex tasks with large search spaces. These suboptimal results arose because CPF consistently prioritized cost-effectiveness by selecting the least expensive execution configurations, rather than exploring the search space or mitigating system noise. Moreover, it did not account for previously measured application configurations or the quality of the resulting models.

To address this limitation, we investigated more sophisticated measurement-point selection strategies such as grid search, random search, Bayesian optimization, and Gaussian process regression to allow Extra-P users to identify suitable sets of measurement points for specific modeling problems. Among the approaches considered, Gaussian process regression (GPR) emerged as the most effective candidate for determining an optimal set of experiment configurations. Unlike the CPF strategy, the GPR-based approach accounts for factors such as noise levels in performance measurements, the number of repetitions per configuration, and the range of parameter values. This enables the selection of application configurations that balance model accuracy and cost within a given modeling budget. Our goal is to replace the CPF strategy with this more efficient and adaptive approach.

At its core, selecting a set of application configurations—including the number of repetitions per configuration—for performance measurements constitutes a classical optimization problem. This problem is illustrated in Figure 14. Beyond deciding between the sets of five points to use in order to model the effects of each parameter on application performance, we can also select which additional points to use that might improve model accuracy. The search space is defined by the model parameters, such as process count and input problem size, together with their respective value ranges, and can comprise thousands of potential measurement points. Each point is associated with a specific cost and a potential accuracy gain, making it essential to capture their trade-off in order to select an optimal subset that balances cost and model accuracy. Beyond cost, external factors such as system noise—caused, for instance, by concurrently running jobs or network bandwidth limitations—must also be considered. For example, on systems with minimal run-to-run variation, repeating each measurement twice may suffice, whereas high-noise environments require more repetitions to ensure reliable accuracy.

S={(x1,x2,…,xm) | lbi≤xi≤ubi, Gi(xi)}(3)

Cw(P(x1,x2,…,xm))=wa·A-wc·C(4)

The size and dimensionality of the search space are determined by the number of model parameters m and their respective value ranges. Since the model parameters are discrete, the resulting measurement points and the associated optimization problem are also discrete. The value range of a parameter x₁, such as the number of processes, is bounded by its lower limit lb₁ and upper limit ub₁, which depend on factors including the system, hardware, application, and available modeling budget. For instance, analyzing an application with fewer than 32 processes (lb₁ = 32) may be impractical, while runs with more than 1,024 processes (ub₁ = 1, 024) may be infeasible due to system size constraints. We formally define the search space for the optimization problem in Equation 3, where (x₁, x₂, …, x_*m*) denote the model parameters, lb_*i* ≤ x_*i* ≤ ub_*i* their bounds, and G_*i*(x_*i*) parameter-specific constraints. Such constraints may, for example, restrict a parameter to cubic integer values.

We define the objective function of the optimization problem in Equation 4 as a reward function that balances the achieved model accuracy A of a set of measurement points P(x₁, x₂, …, x_*m*) against their associated cost C. Model accuracy is defined as the degree to which the constructed model fits the training data, while the cost of a measurement point P(x_*m*) corresponds to the number of core hours required to execute its performance measurement. Optimizing this objective function simultaneously maximizes model accuracy and minimizes cost, resulting in a trade-off that is inherently governed by Pareto efficiency.

Gaussian Process Regression (GPR) is a non-parametric version of Bayesian optimization. The GPR approach to regression is widely used in machine learning. It takes a set of hyperparameters and measurements as input and produces a Gaussian process (GP), defined by a mean function f¯ and a covariance function k (kernel). Together, these specify a normal distribution N(f¯,k) at every point t, such that GP(t)=N(f¯(t),k(t,t′)) (Rasmussen and Williams, 2006). In our case, t represents a measurement point P(x_*m*) within the search space S. The mean function represents the most likely solution to the problem, while the covariance function characterizes the uncertainty, indicating the range in which the true solution is most likely to lie given the hyperparameter-defined assumptions.

k(P(xm),P(xm)′)=kMatérn+kWhite(5)

Cw(P(xm))=C(P(xm))2·wckMatérn(P(xm),P(xm)′)2·wa(6)

The rationale for using GPR to guide measurement-point selection is to estimate the expected improvement in model accuracy when adding an unmeasured application configuration to the existing modeling data. Based on the available measurements, a Gaussian process is constructed, and the maxima of its covariance function are used to identify the most informative additional measurement points. As shown in Equation 5, our approach employs a stationary Matérn covariance function k_Matérn (Rasmussen and Williams, 2006), augmented with a white kernel to account for system noise.

To identify the most informative additional measurement points, we employ the function defined in Equation 6, which rates and ranks all remaining points in the search space (including repetitions) based on their weighted cost. This function represents a practical adaptation of the objective function, balancing the model accuracy achieved by a set of measurement points P(x₁, x₂, …, x_*m*) from the search space S against their associated cost. The weight factors w_*c* and w_*a* allow tuning of this trade-off, enabling the selection of different Pareto-optimal solutions depending on the modeling problem, available budget, and specific requirements.

Based on these definitions, we formalize the measurement selection process of the GPR approach as illustrated in Figure 16. First, the five cheapest measurement points for each model parameter are collected while keeping all other parameters constant (Figure 14). Second, for each parameter, one additional measurement point outside these initial series is selected using a different parameter-value combination, again choosing the cheapest available points. To understand the noise introduced by measurement variability, each measurement is repeated twice. Third, the average noise level is estimated by analyzing the deviation from the arithmetic mean of repeated measurements. Using this initial dataset, we train the GPR and generate a preliminary performance model with Extra-P. If the budget is exhausted, this model becomes the final output. Otherwise, GPR is employed to rank the remaining unexplored measurement points in the search space, and the k highest-ranked points are measured. Although measuring multiple points at once can reduce queue waiting times, the full potential of GPR is realized when points are added iteratively. After each round, the GPR is retrained and a new performance model is created. The accuracy of the updated model is then compared against the previous one to assess improvement. If model accuracy increases and budget remains, the process is repeated. If no improvement is observed for λ consecutive iterations, the model is considered final. The value of λ can be adjusted depending on the budget and problem complexity; we used λ = 3 for two-parameter models.

To evaluate the GPR-based measurement point selection strategy, we conducted a synthetic evaluation, similar to the one presented in Section 4.2, along with six application case studies using common HPC benchmark codes such as LULESH and Kripke. In both settings, we assessed model extrapolation accuracy for configurations outside the training range, as well as modeling cost and budget usage, comparing the GPR approach against the CPF strategy under varying numbers of model parameters and artificially induced noise. For the synthetic evaluation, we additionally included Random and Grid search strategies, as well as a Bayesian approach, as baselines.

Our evaluation showed that the GPR strategy consistently outperformed the CPF strategy in both model accuracy and accuracy-to-cost ratio, improving average accuracy by up to 64.29% (compared to CPF) within the same modeling budget. The GPR approach achieved this by selecting fewer but more informative measurement points, while CPF allocated resources to less impactful points and lacked adaptability. By balancing repetitions of observed points with new configurations in response to noise levels, GPR proved more robust to system noise and run-to-run variations. It also reduced job overhead by requiring fewer runs. Moreover, collecting additional measurement points did not necessarily improve accuracy and could lead to overfitting; often, the best results were obtained with only a fraction of the budget. We found that the most practical strategy is to measure a minimal baseline set with two repetitions per configuration and then apply the GPR strategy to refine point selection if additional budget is available. Overall, the GPR-based measurement point selection strategy enables efficient and cost-effective performance modeling, even in large search spaces (Ritter et al., 2026).

5 How to avoid modeling noise

A constant in all empirical measurements is the presence of noise—whether this is caused by resource contention, system jitter, network communication variance, noise reliably negatively affects our modeling process. Thus, we have developed strategies to counteract or minimize the impact of noise on the performance models Extra-P generates. In this section, we will provide an overview of these strategies.

5.1 Robustness against noise via DNNs

One approach for addressing the issue of noise introduced to the modeling process is to make the modeling process more resilient to perturbations. It has been shown that deep neural networks (DNNs) can learn to adapt to noise when they are trained on noisy data (Seltzer et al., 2013). We leverage this by constructing a methodology that combines regression-based modeling (see Section 3) with deep learning to improve the precision of performance models under noisy conditions (Ritter et al., 2021). To achieve that, our approach intelligently selects which modeling technique to use based on the noise level of the measurements.

The adaptive modeler behaves overall similarly to the other performance modelers: it takes performance measurements as inputs from which it generates and outputs performance models. However, unlike conventional ones relying solely on a single technique like regression-based modelers described earlier, our adaptive version automatically selects the modeling approach (regression vs. DNN) based on an estimated noise level. Figure 17 gives an overview of the inner workings of the adaptive performance modeler. It comprises five main parts: (1) the noise estimator, (2) the preprocessor, (3) the DNN performance modeler, (4) the transfer learning approach, and (5) the existing regression-based modeler.

The first step of the modeling process is to estimate the noise level, so that the modeler can decide whether to use the regression-based modeler, the DNN-based modeler, or both to generate models. To estimate the noise level, we use the range-of-relative-deviation heuristic. This involves calculating the relative deviation for each measurement from the mean of the repeated samples. The range of these relative deviations across all measurements then quantifies the noise level. This heuristic aims to provide a more accurate estimate of the actual noise level compared to individual relative deviations. Once the noise level is estimated, we determine the appropriate modeling technique. Although the DNN modeler produces much better results at high noise levels, the regression modeler achieves better results at low noise levels. To leverage the strengths of both, we adopt an adaptive approach that adjusts to the properties of the modeling task at hand. If the estimated noise level is smaller than a certain threshold, the regression modeler is used in addition to the DNN modeler.

The DNN modeler requires a standardized and uniform input format, necessitating preprocessing of the input data. As execution parameters from applications often vary greatly [e.g., x_1 = (32, 64, 128, 512, 1,024) vs. x_1 = (8, 64, 512, 4,096, 32,768)], normalization is essential. To handle varying input parameter ranges and values, we normalize each measured value v at position P. This is done by dividing v by its corresponding measurement point P, resulting in a normalized value v^=v/P. This process embeds implicit positional information and helps the neural network generalize across different application characteristics. Furthermore, the number of available measurements can vary, so we limit the neural network's input size to a range of [5, 11] entries, masking unused ones with zeros when necessary. This range was chosen because at least five unique values per execution parameter are required—as explained in Section 3—and eleven values per parameter are typically sufficient in practical scenarios. Since the fixed number of inputs does not match the potentially variable number of available measurements, we dynamically group tuples (P,v^) by their positions. The normalized values v^ are then assigned to input neurons, each representing a specific range of measurement points. This grouping resembles nearest-neighbor interpolation with the constraint that each value is used at most once.

Based on the preprocessed input data, the deep neural network of the DNN performance modeler predicts the exponents of the PMNF (see Section 3) for each parameter. For that, we formulated the following classification task: given the pre-processed inputs, which exponent combination describes the performance behavior? The possible 43 classes are based on a predefined function search space of common performance behaviors. After classification, the network's top three results are used to construct performance hypotheses. Their coefficients are then determined via regression based on the available measurement data. For multi-parameter models, single-parameter hypotheses are combined (additive/multiplicative), and the best overall model is chosen.

While the DNN is trained with a variety of synthetic data—imitating realistic application scenarios—the task remains complex, especially due to varying measurement points, their variable number, and the varying noise level. Therefore, one would ideally train a new network for each modeling problem. In practice, that is not feasible, so we employ domain adaptation (Goodfellow et al., 2016) to optimize for the specific modeling tasks. This specialized form of transfer learning allows knowledge learned from general problems to be applied to related problems with different data distributions. In practice, this works by generating a new synthetic dataset similar to the given modeling problem (same parameters, measurement points, and noise level). Then, the pretrained DNN is retrained with this dataset, specializing it for the particular modeling problem.

As mentioned before, the regression modeler (the traditional Extra-P approach) is still utilized for scenarios with low noise levels, where it achieves better results than the DNN modeler. The final step of the modeling process combines these techniques: the best performance model is selected by evaluating the models from the DNN modeler and the regression modeler (if used) against each other using cross-validation and the SMAPE metric.

5.2 Toward better expectations with software counters

Software counters are performance measurements based on the source code, without hardware analysis. A simple example is measuring how often a function is executed when changing its input parameters. However, to obtain a more accurate representation, our approach uses two software counters based on executed LLVM basic blocks (LLVM admin team, 2023) and transferred message sizes. These software measurements are typically resilient to noise; the only exception occurs in functions with busy waiting, where a function or a loop keeps executing until a condition is satisfied.

A key aspect of our approach is the use of measurement tools to collect data dynamically in HPC systems. Therefore, we introduce a software counter plug-in into the Score-P workflow (Score-P developer community, 2023) that inserts instrumentation at the basic block level using the LLVM framework.

Basic blocks are code sequences without internal branches, resulting in a roughly constant execution time. This makes their scaling behavior a robust indicator of asymptotic complexity. Accordingly, we design prior performance modes with Extra-P using basic block measurements for computation functions. While for communication, we use Extra-P to model the scaling behavior of the data transferred among the MPI ranks (in bytes), denoted as B(p, x), where x is the message size and p is the number of MPI ranks. Subsequently, this prior model is integrated into established performance models for point-to-point and collective operations (Chan et al., 2007; Zhang et al., 2017), as shown in Table 7. In the table, the term log₂(p) represents the influence of the number of processes on tree-based communication patterns commonly used for MPI distributing data (Chan et al., 2007).

Table 7. Prior performance models of MPI operations, where f(p, x) denotes runtime, B(p, x) the bytes transferred, with p the number of MPI ranks, and x = {x₁, …, x_*m*} are the input parameters.

The terms α, β, and γ represent latency (including overhead), bandwidth, and computation cost, respectively. Excerpt from earlier studies by de Morais et al. (2025).

The prior equations, based on basic blocks and bytes, are the foundations of our software-counter-based (SWC-based) performance analysis. The exponents of these prior models are retained as functional forms, serving as skeleton equations without coefficients. Following, the coefficient-free equations are combined with noise-sensitive measurements (e.g., wall-clock runtime or energy consumption), where we use linear regression within Extra-P to fit the coefficients. This results in our noise-resilient SWC-based performance model.

Our approach offers several benefits for algorithm engineering. First, our SWC-based models remain closely aligned with the application's theoretical expected behavior. Also, because software counters are inherently robust to noise, we offer noise-resilient performance models. Finally, the experimental cost is significantly reduced, since, unlike classic modeling, our approach does not need multiple repetitions with the same configuration to mitigate the effects of noise. Instead, our SWC-based models require only a single noise-sensitive set of measurements alongside the corresponding software counter measurements to fit the model. The overall framework of our method is represented in Figure 18.

5.3 Avoiding noise via taint modeling

Perf-Taint introduced in Section 4.1 allows us to create more robust performance models in two ways: by removing false parameter dependencies and reducing the impact of instrumentation needed to extract measurements.⁶

5.3.1 False dependencies

Perf-Taint provides Extra-P with detailed parameter dependencies for each function, allowing the modeling process to remove false dependencies and skip modeling of constant functions (Table 8). When comparing models obtained with the taint-enhanced process to results of black-box modeling, we observe that the new models are closer to the ground truths obtained with manual modeling. For example, in the case of MILC, we find MPI utility functions such as MPI_Comm_Rank, which are now processed as constant functions, whereas black-box modeling creates incorrect models due to the presence of measurement noise.

Table 8. Perf-Taint identifies program regions that depend on input parameters; results presented in Copik et al. (2021).

5.3.2 Instrumentation noise

Program instrumentation is necessary to obtain performance measurements, but it can introduce additional overheads—particularly in nested functions. Instrumentation tools like Score-P decrease overheads by instrumenting only a subset of functions and allowing the compiler to apply inlining optimizations. While this reduces overheads, it can create false negative results as some performance-critical functions are never instrumented and modeled.

With Perf-Taint, we can guide instrumentation to only measure functions that cannot statically be shown to be constant. Figure 19 compares the overhead of instrumented measurements with Score-P with the instrumentation filter created by Perf-Taint. In addition to reduced overheads and experiment costs, we find that important functions show different results depending on how invasive the instrumentation is. For example, the performance model of function CalcQForElems from LULESH shows an additive dependency between the two parameters: 3·10⁻³·p^0.5+10⁻⁵·size³. However, the modeling with Perf-Taint leads instead to a multiplicative dependency 2.4·10⁻⁸·p^0.25·size³. The latter result can be validated with previously obtained results (Calotoiu et al., 2018). On the other hand, the default Score-P filter does not instrument this function.

6 Non-standard use cases of Extra-P

While the primary goal of Extra-P is the performance modeling of parallel applications and the identification of scalability bottlenecks, multiple approaches have successfully used the powerful modeling capabilities of Extra-P toward other goals. In this section, we provide an overview of the non-standard use cases of Extra-P.

6.1 Requirements engineering for co-design

Extra-P focuses on identifying scaling issues, which are usually architecture-independent, even if the precise runtime implications may depend on architectural features. For example, an algorithm whose runtime grows with the square of the problem size won't scale on any platform. However, as the needs of scientific applications grow, and as the hardware systems they run on become increasingly diverse, hardware-software co-design becomes ever more important. One approach toward understanding performance portability is to partially execute the code on one platform, and derive hardware counters that can be used to predict the performance on other platforms (Yang et al., 2005). We instead leverage Extra-P to support the co-design process by introducing a series of hardware-independent application requirements, and allowing system architects to better understand how the needs of different scientific codes evolve as their various inputs are scaled up, as shown in Calotoiu et al. (2018). The list of requirements proposed is summarized in Table 9.

Table 9. Hardware-independent requirement metrics.

By modeling the memory footprint, computation, communication, and local data movement, we cover the main resources scientific applications use.

To showcase the utility of this approach, we first collect performance models for a number of scientific applications and benchmarks: Kripke—a deterministic particle transport code, Lulesh—an unstructured Lagrangian explicit shock hydrodynamics proxy-app, MILC—a quantum chromodynamics simulation, Relearn—a brain plasticity simulation, and icoFoam—a transient solver in a computational fluid dynamics framework. We summarize the results in Table 10.

Table 10. Per-process requirements models for five scientific applications and proxy-apps.

p denotes the number of processes and n = N/p the problem size per process obtained by dividing the overall problem size N by the number of processes p, under the assumption that the overall problem size can be divided equally among all processes. For each metric, we show the terms with the largest impact on performance for both problem size per process and number of processes. The coefficient is the sum across the entire program, rounded to the nearest power of ten. We mark potential performance bottlenecks with a warning sign.

While a more detailed discussion of both the applications and their performance bottlenecks, as well as possible co-design scenarios, can be found in Calotoiu et al. (2018). In this work, we focus on LULESH and how different requirements would change if the hardware systems would be modified. However, the different requirements scale at different rates when the problem size and process count are varied.

We consider a potential system upgrade, where the number of racks available is doubled, and see how the per-process requirements change relative to the original system. The result is summarized in Table 11. We can see that while the requirements for computation, communication, and memory accesses for LULESH will slightly grow as the program is scaled up, it would be able to make use of the increased resources successfully. This process offers a quick and automatic technique to augment the traditional back-of-the-envelope calculation often used in co-design and HPC system purchasing decisions.

Table 11. Workflow for determining the requirements of LULESH after doubling the number of racks.

The upgrade is relative and we can therefore omit model coefficients.

6.2 Task-graph modeling

One use case for the multi-parameter modeling approach is the parallelism limitations in task-based parallel programs. Task-based programming paradigms such as Cilk and OpenMP (tasking pragmas) have been a focus of active research for quite some time now. The advantage is that they offload thread management to compilers leaving the users with a clearer way to express parallelism.

Task-based programming models like Cilk and OpenMP are increasingly important as multicore systems grow, offering a way to express parallelism through tasks and offload thread management to compilers. However, achieving scalable efficiency in exascale systems requires not only optimizing algorithms but also increasing input size in line with core count, as fixed input sizes limit speedup due to critical path constraints. This relationship is captured by the isoefficiency concept, which ties efficiency to both processing elements and input size. Yet, theoretical models often fail to reflect real-world limitations, such as resource contention (e.g., memory and cache bottlenecks). To address this, the study introduces a novel, practical method that models empirical efficiency functions using a task replay engine that simulates contention-free runs. By comparing these to normal executions and analyzing task dependency graphs, the method identifies performance bottlenecks and improvement opportunities. This approach supports better application and system design by answering key questions related to scalability, contention, optimization potential, and efficient resource utilization.

6.2.1 Task dependency graphs

A task dependency graph (TDG) is a directed acyclic graph (DAG) that represents the data relations and the data flow between the tasks. It imposes an execution order on tasks so that a task can be executed only if all the incident tasks (i.e., tasks on the other end of incoming edges) have completed their execution and the data needed by the task is available. For the purpose of our discussion, we assume that a thread-based runtime executes the tasks in a TDG, and each instance of a task is executed at any given time by only one thread. Furthermore, we limit the number of threads to the number of cores in the system. It does not affect our discussion since threads just execute the available tasks. To achieve better compute saturation, one needs to increase the number of available tasks rather than the number of threads.

We can characterize any TDG using a set of metrics and rules (Cormen et al., 2009; Blelloch, 1996):

• T_*p*(n): the execution time of a computation with p threads with input size n.
• T₁(n): the work of the computation (i.e., the total execution time on one core) for input size n.
• T_∞(n): the depth of the computation (i.e., the total sum of all task times on the critical path) for input size n.
• Sp(n)=T1(n)Tp(n): the speedup of the computation.
• Work rule: the execution cannot be faster than when we divide the whole work T₁(n) equally between cores, that is Tp(n)≥T1(n)p or S_*p*(n) ≤ p (we ignore super-linear speedups for the sake of simplicity).
• Depth rule: the critical path is a chain of dependencies, so the tasks on this path must be executed one after the other, giving us a lower bound on the execution T_*p*(n)≥T_∞(n).

6.2.2 Efficiency and isoefficiency

The efficiency function is defined as E(p,n)=Sp(n)p, whereas the isoefficiency (Grama et al., 1993, 2002), which binds together p and n for a specific, constant efficiency, is simply a contour line on the surface of E(p, n). To clarify this point, we first define the total overhead time: T_*o*(p, n) = pT_*p*(n)−T₁(n). This is the total amount of time that all of the threads spend without contributing to the solution of the problem, including resource contention, idle time, and scheduling overhead. Rearranging this equation and fixing the efficiency yields the isoefficiency relation: T1(n)=E1-ETo(p,n).

Isoefficiency is a helpful concept for the analysis of parallel algorithms. It allows users to consider various alternatives of the algorithm and steer their choice toward the one in which the problem size grows more slowly in relation to the core count. This means they are able to maintain high efficiency at scale without needing to have extremely large inputs. In reality, resource contention overhead might render an algorithm unscalable. The technique discussed in this section tackles this problem by modeling efficiency functions that help pinpoint the overhead origin.

We can identify three different efficiency functions for a task-based application: (i) E_*ac*(p, n): the actual efficiency function that reflects realistic performance as the application experiences resource contention and scheduling overhead; (ii) E_*cf*(p, n): The contention-free efficiency function, modeled according to the results of replaying empty (e.g., no code and no memory or resources access) task skeletons in the TDG; (iii) E_*ub*(p, n): the upper bound on the efficiency of the application. This function describes an ideal situation of maximum speedup that is hardly achievable in practice. By further analyzing the differences between the efficiency functions, we can gain a number of important insights. First, the contention discrepancy between actual and contention-free efficiencies Δ_*con* = E_*cf*(p, n)−E_*ac*(p, n) characterizes how severe the resource contention overhead is. It can point the user to focus on reducing the resource contention. Second, the structural discrepancy between upper bound and contention-free efficiencies Δ_*str* = E_*ub*(p, n)−E_*cf*(p, n) characterizes the optimization potential of the application at the level of the task graph (Shudler et al., 2017).

6.2.3 Isoefficiency modeling

Figure 20 shows the workflow to model the efficiency functions, which in turn, gives us models of isoefficiency. First step is instrumenting the task-based code and then constructing the TDG by running the code for p = 1 and increasing n. The second step is benchmarking both the code and the replay for increasing n and p. We select a range for these values beforehand. The benchmark then runs for each combination of p and n from these ranges. The final step is producing the empirical models.

6.2.4 Task replay engine

Instrumented task-based code allows us to construct TDGs for increasing input sizes. To obtain task runtimes free of shared resource contention overhead, we perform the TDG construction runs using only a single thread.

When replaying the TDG, we simulate each task by implementing it as a function that receives the task runtime as an argument and then “busy-waits” in a loop for that duration. The overhead of this function is minimal. It only includes querying the time, repeating a loop counter, and accessing one local variable to store the accumulated time. Note that during the replay run, we ensure that the combination of the scheduler and tied/untied tasks is the same as it is during the normal runs.

6.3 Compositional performance modeling

Having a whole program at hand raises questions that are difficult to handle, e.g., What influence does changing an algorithm have regarding the overall performance? and What kind of improvement can I hope for when removing a certain bottleneck? To address this problem, we will adapt a data-flow–centric view, tracking both the latency and the throughput of algorithms. This shifts perspective and allows us to inspect the effects that parts of a program have on the overall goal. After all, exchanging a part of the program with a better scaling alternative, e.g., sorting IDs with mergesort instead of bubblesort, might have unexpected side effects. Staying in the example of mergesort vs. bubblesort, the obvious change is in the time complexity of processing a list with n elements completely—mergesort takes O(nlog(n)) time while bubblesort takes O(n²) time. The time complexity affects throughput of the algorithm: For sizable lists, mergesort will take less time per list to sort it completely and, thus, can sort more lists in a given time than bubblesort. This simple observation can be misleading as it neglects the latency, i.e., the time the algorithm takes to process a data element. On the one hand, the whole latency for a complete list will be smaller for mergesort, so much is expected. On the other hand, looking at the time complexity for processing the list might have been the wrong approach. It can very well be the case that the sorted list will be processed element-by-element, and not as a whole. When looking at the latency of the sorted elements in the case of mergesort, we see that n elements are sorted after O(nlog(n)) time—all at once. Bubblesort, in contrast, produces the n elements each after O(n) time, i.e., it takes longer in general, but the first elements arrive earlier. Note here that we picked latency and throughput as the easiest to analyze, but this approach works naturally with other performance criteria as well, e.g., with hardware resources that need to be shared, like network bandwidth, memory capacity, or cache.

This reasoning underlies compositional performance modeling. We conceptually split a program along natural boundaries into modules, analyze the modules on their own, and then compose the modules again—effectively, we apply the always-present idea of divide-and-conquer. In the general workflow, this can happen at various stages:

• Knowing the whole structure of a program, splitting it allows one to isolate a certain region we deem necessary for further investigation. If, after investigation and possibly changes in code, we see that the latency and throughput (and possibly other performance-related metrics) stayed constant or improved, we can be sure that the overall performance does not degrade.
• Having performance models for every node in the call path of the algorithm, we can substitute different models “on paper” and check the overall influence—it might very well be that an improvement in the algorithm does not translate into an overall improvement.
• Having performance models for every leaf node in the call path, we can calculate the performance models for all inner nodes. This approach is the most complex one but offers the greatest benefits in return.

The whole approach hinges on composition operators, i.e., higher-order functions that take as input performance models, hyperparameters, and—if necessary—more, and return a performance model again. These functions encapsulate an aspect of structure in program code that sometimes gets overlooked: No matter what functions I combine during programming, the overhead/characteristics of the combination are constant. We will repeat the three most basic operators here, but note that further operators for more difficult structural combinations are easily imaginable. We will use the latency function l(·), which returns for a given program the time it takes to process one element, and the throughput function t(·), which returns the average waiting time between two elements.

Sequential concatenation: Given two programs P₁, P₂, we define the sequential concatenation P = (P₁⊕P₂) as when the same execution unit (thread, process, rank,...) executes both sequentially. We define l(P) = l(P₁)+l(P₂) as the time it takes a data element to traverse P₁ and P₂ is just the sum of the individual times. t(P) = t(P₁)+t(P₂) with the same idea.

Parallel concatenation (functional parallelism): Given two programs P₁, P₂, we define the parallel concatenation P = (P₁||P₂) as the result when two distinct execution units, one computing P₁ and the other P₂, with a means (e.g., a queue) to synchronize the data elements in between. l(P) = l(P₁)+l(P₂)+ϵ₁+ϵ₂, as functional parallelism does not speed up the execution time of the functions on their own. Note here that ϵ_{1, 2} represents the additional time introduced by this pattern, e.g., pushing and popping from a thread-safe queue, although this could be emulated with P = [(P₁⊕push)||(pop⊕P₂)]—it is merely a question of coarser or finer modeling. t(P) = max[t(P₁)+ϵ₁, t(P₂)+ϵ₂], as the throughput is dominated by the time of the slower stage.

Parallel execution (task parallelism): Given a program P₁ and a number of execution units k, we define the parallel execution P = r_*k*(P₁) as the result when k distinct execution units each perform P₁. l(P) = l(P₁)+ϵ, as task parallelism does not speed up the execution time of the function on its own, and ϵ represents the synchronization overhead like before. t(P) = 1/k×(t(P₁)+ϵ), as the waiting times averages out.

Given these, low-hanging fruits like rep_*k*—the composition operator for for-loops—are easily definable, but also more complex ones such as MapReduce. The granularity of the operators and their details depend on the problem at hand: Are we modeling the execution of a time-critical function (inside the operating system's kernel or within an audio driver), the workflow of a scientific simulation code, or the design of a distributed algorithm for edge-device computing. Some might require modeling hardware ports of the CPU, some the waiting times of synchronization, some the communication time over the network, and some might not care for such small noise-inducing parts. Neglecting synchronization overhead, it is an easy calculation to see that functional parallelism is not better than task parallelism (see Figure 21) when the execution units are equally distributed.

7 Related work

The long history of performance modeling includes a wide range of manual methods where models are derived by human experts (Kerbyson et al., 2001; Hoefler et al., 2011; Bauer et al., 2012). Designing comprehensive methods that work automatically and can be applied to whole applications with little user guidance has long been the goal of research in performance modeling. Examples include recording memory access patterns to predict performance across different architectures (Marin and Mellor-Crummey, 2004), categorizing loops to create performance models of sequential algorithms (Zaparanuks and Hauswirth, 2012), using linear regressions to generate models from empirical measurements (Goldsmith et al., 2007), and employing roofline models to find an optimal parameter configuration for a given system (Lo et al., 2014). The PMaC tool suite develops scaling models through a two-stage approach: first creating a model for single-node performance based on memory architecture, and then combining with a model of network communication overhead (Meswani et al., 2013). Extrapolation techniques have also been employed to predict performance using limited hardware resources. Zhai et al. (2010) predict performance from single-node runs, while Wu and Müller (2012) synthesize program traces from small-scale executions to analyze communication patterns.

7.1 Multi-parameter modeling

While these methods demonstrate considerable capability, none fully support multi-parameter models. When multiple parameters are considered, existing approaches typically examine them individually or assume fixed relationships between them. PALM creates multi-parameter models but requires users to annotate source code with micro-models applicable only to specific code sections (Tallent and Hoisie, 2014). After conducting detailed per-function measurements, the framework automatically combines these micro-models into structural macro-models. This approach shares similarities with Aspen (Spafford and Vetter, 2012), a specialized language for defining such micro-models. Vuduc et al. (2004) select optimal algorithm implementations by automatically generating numerous candidates and choosing the one with superior performance based on empirically derived models, which may include multiple parameters. However, only the linear regression for determining model coefficients is automated; the model hypothesis must still be selected manually, limiting the approach's automation in performance modeling. Extra-P achieves greater automation without requiring prior source-code annotation.

Creating multi-parameter performance models from measurements presents several challenges, including identifying performance-relevant parameters and balancing experimental costs against model quality. Siegmund et al. (2015) propose a black-box process to analyze the interactions between configuration options in an application, which requires identifying all parameters beforehand and makes no assumptions about parameter interactions. Jayakumar et al. (2015) present another multiparameter approach where execution signatures are extracted from target applications to represent different execution phases. These signatures are then matched with reference kernels from a database, using static analysis to derive performance models for unknown kernels. Machine learning methods have also been successfully applied to performance modeling tasks (Ipek et al., 2005; Lee et al., 2007).

7.2 Static code analysis

Several research efforts have focused on enabling performance modeling through static source code analysis. Kuhnemann et al. (2004) propose a source code analyzer that generates predefined runtime functions for MPI routines and estimates runtime functions for computational kernels with loop instrumentation. Meng and Norris (2017) implement static analysis with polyhedral loop modeling within the ROSE compiler framework, producing not an analytical model but a Python script that models program behavior and dynamically evaluates selected metrics. Lee et al. (2015) provide COMPASS, a static modeling tool built on the OpenARC compiler. Their analysis generates annotations containing models in Aspen, which are then combined to construct a structured model of the entire application. Hammer et al. (2015) derive roofline models through a combination of source code analysis and cache simulations. Gschwandtner et al. (2018) propose static analysis for the Insieme compiler that creates parametric models for both time and energy consumption, using a single instrument run and static dataflow analysis. Bhattacharyya et al. (2015) present a hybrid performance modeling tool designed for online performance modeling. Their approach applies static analysis of loop complexity to cluster loops with a structural similarity, allowing estimation of a single model for each cluster.

8 Outlook

Extra-P has focused on finding scalability bugs in programs running on CPUs. However, as computational systems evolve toward increasingly complex and heterogeneous architectures, the execution on CPUs is only one part of the story. With the advent of GPGPU computing, the computational focus has been shifting toward GPUs, so that support for the performance modeling of GPU execution is crucial. Extra-Ps hardware-agnostic modeling already allows modeling GPU performance today. Nevertheless, future work should strongly emphasize developing modeling strategies that explicitly account for heterogeneity, incorporating features specific to accelerators' execution paradigms, memory hierarchies, data parallelism capabilities, scheduling overheads, and communication bottlenecks between different device types. Understanding how code behaves differently on CPUs versus GPUs or other accelerators is crucial for accurate predictions in mixed-device systems.

Moreover, the pervasive issue of measurement noise inherent to empirical performance modeling remains a challenge. While the above techniques can be used to reduce their impact, further research is needed to improve the coverage of the approaches. Furthermore, while empirical modeling with Extra-P has been successful in finding scalability bugs, identifying the root cause of specific performance anomalies or scalability bottlenecks is still an underexplored area. Another key area for future improvements is a more extensive use of machine learning. While Extra-P already employs several machine learning methodologies, the advent of more and more powerful techniques may enable new performance modeling approaches that overcome current limitations.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Justin M. Wozniak, Argonne National Laboratory (DOE), United States

Reviewed by: Xingfu Wu, Argonne National Laboratory (DOE), United States

Marta Garcia-Gasulla, Barcelona Supercomputing Center, Spain

Wayne Strasser, Liberty University, United States

Footnotes

https://github.com/extra-p/extrap

This subsection has been adapted from Calotoiu et al. (2013).

This subsection has been adapted from Calotoiu et al. (2016).

This subsection has been adapted from Ilyas et al. (2017).

This subsection has been adapted from Copik et al. (2021).

The subsection has been adapted from Copik et al. (2021).