OpenMP-annotated code dataset for large language model fine-tuning on parallel programming tasks

1 Introduction

High-performance computing (HPC) plays a critical role in scientific discovery, engineering simulation, and data-intensive applications. Open Multi-Processing (OpenMP) is one of the most widely adopted shared-memory parallel programming interfaces, enabling developers to write multi-threaded applications in C, C++, and Fortran. However, correctly implementing OpenMP directives requires significant expertise, as developers must understand parallel programming concepts, data dependencies, and performance optimization strategies.

Recent advances in large language models (LLMs) have demonstrated remarkable capabilities in code generation and completion tasks (Jiang et al., 2024; Huynh and Lin, 2025; Zan et al., 2023). These models, trained on vast corpora of source code, can assist developers by generating code snippets, completing partial implementations, and even translating natural language descriptions into executable programs. However, general-purpose code LLMs often struggle with domain-specific parallel programming constructs such as OpenMP pragmas. The scarcity of high-quality, task-specific training data for HPC code represents a significant barrier to developing effective AI assistants for parallel programming (Nichols et al., 2023). While existing code generation benchmarks focus primarily on sequential programming tasks in languages like Python and Java, there is limited availability of curated datasets specifically targeting parallel programming paradigms.

This data report presents a curated dataset specifically designed for fine-tuning LLMs on OpenMP pragma completion tasks. The dataset contains 77,890 source files comprising over 15 million lines of code extracted from 387 GitHub repositories. Each training sample is structured to teach models the relationship between code context, loop structures, and appropriate OpenMP directives. This dataset addresses a critical gap in available resources for training AI models on parallel programming tasks and provides a foundation for developing intelligent code completion tools for HPC developers.

The primary contributions of this dataset are: (1) a systematically collected corpus of real-world OpenMP code from active HPC projects, (2) structured annotations that isolate OpenMP pragmas and their associated loop contexts, and (3) comprehensive preprocessing and quality filtering to ensure dataset integrity. This resource enables researchers to develop and evaluate LLMs specifically tailored for parallel programming assistance.

2 Methods

2.1 Data collection period and source selection

Data collection was conducted between June and July 2024 using the GitHub API. Source repositories were identified through a systematic query process targeting HPC-relevant codebases. The selection criteria were designed to ensure code quality and relevance to OpenMP development practices:

• Primary languages: C and C++ (specified as repository primary language).
• Repository topics: HPC, OpenMP, parallel-computing, scientific-computing, high-performance-computing, computational-science, proxy-application, and mini-app.
• Minimum stars: ≥3 [indicating community validation and active use (Borges and Valente, 2018)].
• Repository scope: publicly accessible repositories with permissive licenses.

The GitHub API query utilized a custom Python script that performed systematic searches combining repository topics and language filters (Gousios and Spinellis, 2012). Authentication was handled through GitHub personal access tokens to enable comprehensive repository access. The search process iterated through pagination (up to 34 pages per query) to exhaustively collect matching repositories, with rate-limiting safeguards to comply with API restrictions.

This filtering strategy yielded 387 unique repositories, balancing dataset size with code quality (Cosentino et al., 2016; Munaiah et al., 2017). The minimum star threshold ensured that repositories had achieved some level of community review (Borges and Valente, 2018), while the topic filters specifically targeted HPC applications where OpenMP usage reflects realistic parallel programming patterns.

2.2 Repository cloning and file extraction

Identified repositories were cloned locally using GitPython library to create local copies organized by repository full name (owner/repository structure). This organization preserves provenance information for each source file.

Source file extraction focused exclusively on C/C++ implementation and header files, identified by extensions: .c, .cc, .cpp, .cxx, .C, .h, .hh, .hpp, .H, .hxx, .Hxx, and .HXX (Allamanis and Sutton, 2013). Recursive directory traversal proved approximately 2–3 × faster than glob-based approaches for large directory structures. Files containing invalid path characters (brackets) were excluded to prevent filesystem conflicts.

Initial extraction yielded 105,861 source files totaling 22,653,593 lines of code (0.71 GB).

2.3 Preprocessing and quality filtering

To ensure dataset quality and prevent training bias, a multi-stage preprocessing pipeline was implemented:

2.3.1 Encoding validation

Files containing non-UTF-8 characters were removed by attempting to read each file with UTF-8 encoding. Files that could not be decoded were excluded to prevent tokenization issues during model training.

2.3.2 Size-based filtering

Two size constraints were applied:

• Minimum token count: files with fewer than 15 tokens (whitespace-delimited) were excluded as they typically contained only boilerplate or comments.
• File size limits: files exceeding 1 MB were removed, as these typically represented embedded libraries, generated code, or raw data rather than human-authored source code.

2.3.3 Deduplication

Duplicate files are prevalent across GitHub repositories due to forking, vendored dependencies, and copied implementations (Markovtsev and Long, 2018). SHA-256 hashes of file contents were computed using a memory-efficient streaming approach. Files with identical content hashes were deduplicated, retaining only the first occurrence.

After preprocessing, the dataset comprised 77,890 unique source files with 15,367,210 lines of code (0.49 GB), representing an 18% reduction from the initial collection primarily due to deduplication.

2.4 OpenMP pragma extraction and annotation

The core dataset transformation extracts individual OpenMP parallel for constructs and formats them for pragma completion tasks. An automated extraction pipeline implements this transformation:

2.4.1 Pattern matching

A compiled regular expression (#pragma omp parallel for.*) identified all OpenMP parallel for directives in each source file. The pattern uses multiline matching to handle pragmas spanning multiple lines.

2.4.2 Loop boundary detection

For each identified pragma, a bracket-matching algorithm extracted the complete associated loop structure:

• Locate the opening brace { following the pragma.
• Traverse the code incrementing a bracket stack counter for each { and decrementing for each }.
• Extract the complete loop body when the stack returns to zero.
• Exclude pragmas where bracket matching failed (typically single-statement loops without braces).

2.4.3 Comment stripping

C/C++ style comments (// and /* */) were removed from pragma lines using pattern matching to isolate the actual directive syntax. This prevents the model from learning spurious comment‘patterns.

2.4.4 Context window extraction

Each sample includes a configurable amount of preceding code context, enabling the model to learn contextual patterns such as variable declarations, array definitions, and computational patterns that inform appropriate pragma selection.

2.4.5 Structural annotation

Extracted samples are annotated with special tokens to clearly delineate components:

• < LOOP-START> and < LOOP-END>: mark loop boundaries.
• < OMP-START> and < OMP-END>: mark pragma boundaries.

This tokenization strategy enables the model to distinguish between code to be generated (pragma) and conditioning context (loop body).

2.4.6 Directive scope

This dataset focuses on extracting #pragma omp parallel for directives and their variants [e.g., parallel for simd, parallel for reduction(…)] as prediction targets. The extraction captures various clauses including scheduling policies [schedule(static), schedule(dynamic), and schedule(guided)], data-sharing (private, shared, reduction, firstprivate, and lastprivate), synchronization (nowait), loop transformations (collapse), thread control (num_threads), and ordering (ordered). Other OpenMP directive types such as target (for accelerator offloading), task/taskloop (for task-based parallelism), and the OpenMP 5.0+ loop directive are not systematically extracted as prediction targets, though they may appear in surrounding code context.

2.5 Dataset structure and format

The final dataset is stored in JSON Lines format (.jsonl), with each line representing a single training sample. Each sample contains:

• Source file path: full path to the original source file (enables provenance tracking).
• Pragma directive: the extracted OpenMP pragma directive.
• Context length: number of context characters included before the loop.
• Annotated sample: the complete annotated sample (context + annotated loop + annotated pragma).

This format enables efficient streaming during training and preserves metadata for subsequent analysis or dataset versioning.

2.6 Intended use and model training protocol

This dataset is specifically designed for fine-tuning causal language models on OpenMP pragma completion tasks. The recommended training protocol involves:

• Tokenization: process the annotated sample field using the target model's tokenizer with padding and truncation (recommended maximum length: 512 tokens).
• Label preparation: use standard causal language modeling where input sequences serve as both inputs and labels.
• Training objective: cross-entropy loss on next-token prediction.
• Evaluation task: given context and loop structure, generate the appropriate OpenMP pragma.

The dataset enables models to learn:

• Relationships between computational patterns and parallelization strategies.
• Appropriate scheduling policies (static, dynamic, and guided).
• Data sharing clauses (private, shared, and reduction).
• Loop dependency analysis informing pragma selection.

3 Dataset analysis and characteristics

3.1 Corpus statistics

The final OpenMP dataset exhibits the following characteristics (Table 1).

Table 1. OpenMP dataset statistics.

The preprocessing pipeline reduced the initial collection by approximately 18%, primarily through deduplication (17.0% reduction) and size filtering (additional 11.3% reduction of the deduplicated set) as shown in Table 2.

Table 2. Dataset properties after each preprocessing step.

3.2 File type distribution

The dataset spans multiple C/C++ file types, with the distribution of lines of code revealing the predominance of implementation files:

• Implementation files (.c, .cpp, .cc, .cxx, .C): majority of LOC, containing actual OpenMP-annotated compute kernels.
• Header files (.h, .hpp, .hh, .H): lower LOC count, typically containing template implementations and inline functions.

This distribution reflects typical C/C++ project structure, where OpenMP pragmas appear predominantly in implementation files containing computational loops.

3.3 OpenMP pragma patterns

The dataset captures diverse OpenMP usage patterns from real-world HPC applications:

• Scheduling policies: examples include static, dynamic, and guided scheduling with various chunk sizes.
• Data sharing clauses: private variables, shared arrays, and reduction operations [e.g., reduction(+:s)].
• Nested parallelism: some samples include nested parallel regions (though single-level pragmas dominate).
• Loop variants: both increment and decrement loops, varying iteration patterns.

The pragma #pragma omp parallel for schedule(static) appears frequently, representing a common parallelization pattern for regular, independent loop iterations. More complex pragmas with reduction clauses reflect numerical computation patterns (e.g., dot products, norm calculations).

3.4 Application domains

The source repositories span diverse HPC application domains:

• Scientific computing: numerical methods, linear algebra operations (BLAS-level implementations).
• Geospatial analysis: GIS applications with parallel raster processing.
• Proxy applications: miniapps designed to represent computational kernels from large-scale simulations.
• Computational libraries: reusable parallel algorithm implementations.

This domain diversity ensures the dataset captures varied computational patterns and parallelization strategies, rather than overfitting to specific application characteristics.

3.5 Data quality considerations

Several quality features distinguish this dataset:

• Real-world code: extracted from actively maintained repositories rather than synthetic examples.
• Community validation: minimum star threshold indicates code review and usage.
• Deduplication: SHA-256 hashing prevents training on redundant patterns.
• Size filtering: eliminates both trivial files and unwieldy library dumps.
• Encoding validation: UTF-8 compliance ensures tokenization compatibility.

3.6 Limitations and biases

Users should consider the following characteristics when interpreting results:

• Language scope: limited to C/C++; Fortran OpenMP code is not represented.
• Directive scope: this dataset is designed for loop-level shared-memory parallelization with parallel for pragmas. The following are not included as prediction targets:

• - target directives for GPU/accelerator offloading.
• - task/taskloop constructs for task-based parallelism.
• - Standalone simd directives for explicit vectorization.
• - sections for functional parallelism.
• - OpenMP 5.0+ features such as the loop directive.

• While these constructs may appear in surrounding code context, they are not prediction targets. Models trained on this dataset may not generalize to other parallelization patterns without additional training data.

• GitHub bias: repository availability and topic tagging may not represent all HPC development practices.
• Temporal snapshot: collected in June-July 2024; newer OpenMP features may be underrepresented.
• Single-statement loops excluded: bracket-matching algorithm currently skips loops without explicit braces.

4 Data reuse and interpretation guidelines

4.1 Recommended use cases

This dataset is optimized for:

• Fine-tuning code generation models on OpenMP pragma completion and translation.
• Developing intelligent code assistants for parallel programming.
• Analyzing OpenMP usage patterns across HPC codebases.
• Benchmarking model performance on domain-specific code tasks.
• Training retrieval systems for parallel programming documentation.

4.2 Data access and format

The dataset is distributed in JSON Lines format, enabling:

• Streaming processing for memory-constrained environments.
• Parallel loading via line-based sharding.
• Easy filtering based on metadata fields (source file path, pragma type, and context length).

4.3 Preprocessing recommendations

Researchers reusing this data should consider:

• Tokenizer compatibility: ensure the tokenizer handles special annotation tokens ( < LOOP-START>, etc.) appropriately.
• Context window sizing: the context length metadata enables experimentation with varying context window sizes.
• Train/validation splitting: implement file-based or repository-based splitting to prevent data leakage.
• Pragma diversity: consider stratified sampling if balancing representation of different pragma types.

4.4 Dataset updates and versioning

As OpenMP standards evolve and new HPC repositories emerge, this dataset may be extended to address current scope limitations. Planned extensions include expanding directive coverage to systematically extract target directives for GPU/accelerator offloading, task/taskloop constructs for task-based parallelism, standalone simd directives for explicit vectorization, and OpenMP 5.0+ features such as the loop directive as prediction targets. Future work may also explore multi-task learning approaches that enable directive-type selection in addition to clause generation.

Any dataset updates should be:

• Deposited as independent versions in the repository.
• Documented with collection date ranges and repository counts.
• Published as Addendum articles linking to this initial Data Report.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Richard Gerber, Berkeley Lab (DOE), United States

Reviewed by: Akash Dutta, Advanced Micro Devices, United States