MedPerf enhances User Experience with improved Data Preparation Pipelines in Federated Clinical Studies

What It Is

The MLCommons medical working group continues driving real-world benchmarking and evaluation of AI/ML models in healthcare through MedPerf, an open-source federated research orchestrator.Leveraging expertise from large-scale global clinical studies, MedPerf now integrates Apache Airflow to greatly simplify the distribution, orchestration, and monitoring of data preparation pipelines in federated research. By combining MedPerf’s federated experiment orchestration with Airflow’s mature scheduling and monitoring capabilities, the community can more easily conduct reproducible clinical AI research.

Why It Is Needed

The need for Airflow integration arose from challenges encountered in previous clinical studies. Previously, MedPerf benchmark committees had to create a single container for data preparation and distribute it to data providers for local execution. This monolithic containerization approach often led to unexpected errors, such as data provider input types not matching expectations or confusion over execution instructions. Troubleshooting was difficult, requiring remote assistance from pipeline developers. Additionally, a single container did not allow data providers to easily recover after errors, and individual pipeline steps could not be reused across other pipelines. In the Federated Learning for Postoperative Segmentation of Treated glioblastoma (FL-PoST) study, a single container that could be interrupted and resumed was developed (see prior technical report), which addressed most issues but lacked scalability, as each new study required rebuilding the container.

Solution

The Airflow integration in MedPerf aims to improve the usability and robustness of data preparation in a scalable manner. With Airflow, multiple containers can be chained into a single data preparation workflow, bringing two key benefits to MedPerf users:

• Each container becomes a modular step in the data preparation pipeline, reusable across different workflows.
• Airflow provides a WebUI for pipeline execution and monitoring, making it easier to diagnose errors and allowing users to restart the pipeline from any step.

To accommodate non-technical users (e.g., study coordinators), they only need to build a pipeline using a single YAML file (example here). The MedPerf client automatically converts it into executable Airflow Directed Acyclic Graphs (DAGs) for Airflow to execute.

Technical Details

Each time MedPerf instantiates Airflow, it runs locally with a randomly generated password and automatically shuts down after data preparation completes or upon user interruption, ensuring the security of data providers’ sensitive data. The integration also includes a small tool that periodically reports data preparation status to the MedPerf server via Airflow’s REST API. This helps benchmark committees review the progress of data preparation across the entire federation (multiple sites) and proactively assist participating sites.

Example: Data Preparation for the FL-PoST Study

The implementation of the new data preparation pipeline for the Federated Learning for Postoperative Segmentation of Treated glioblastoma (FL-PoST) study is detailed in the README.md file of the MedPerf code repository. Key highlights are described below.

Data preparation status can be easily viewed in the Airflow WebUI. Figure 1: DAG view in Airflow, showing each data preparation step and its status.

Figure 2 shows a partial graphical view of the first few steps of the workflow. Figure 2: Partial graphical view of generated tasks in Airflow.

The data preparation pipeline for this study also includes a manual review step, where a human operator from the data provider manually reviews the generated tumor segmentation maps and approves or corrects annotations. Meanwhile, Airflow places the task in the "Up for Reschedule" state, waiting for files to be reviewed and copied into place according to the data preparation guidelines (see Figure 3). Once all review files are in place, Airflow automatically resumes execution. Figure 3: Airflow waiting for manual review of two tasks while running other tasks.

The "Approval Operator" feature in Airflow 3.1.0 allows human operators to manually approve pipeline runs. In the FL-PoST study, such approval is requested after all brain scans have been processed and reviewed (see Figure 4). This is the final step, after which the pipeline automatically generates training and test splits for the experiment. Figure 4: Required action notification in the Airflow WebUI.

The required YAML file follows a convention similar to the container-config.yaml used when submitting containers in MedPerf, but allows chaining multiple containers in a single step. The basic structure is shown below, illustrating a single step. The mounts field has the exact same structure as the container-config.yaml used for submitting model and metrics containers in MedPerf.

steps:
  - id: unique_id_for_step
    type: container
    image: container_image_name:tag
    command: command to run in container
    mounts:
      input_volumes:
        data_path:
          mount_path: /path/to/input/data/directory
          type: directory
        labels_path:
          mount_path: /path/to/input/labels/directory
          type: directory
      parameters_file:
        mount_path: /path/to/parameters.yaml
        type: file
      output_volumes:
        output_path:
          mount_path: /path/to/output/data/directory
          type: directory
        output_labels_path:
          mount_path: /path/to/output/labels/directory
          type: directory
    next: id_for_next_step_or_null_if_last_step

Future Outlook

The Apache Airflow integration is now available in MedPerf and has been deployed in production for the FL-PoST study. We invite the broader healthcare AI community to explore the implementation, test it in their own federated workflows, and contribute back to the project.

• Explore the full implementation in the MedPerf GitHub repository.
• See YAML examples for building custom pipelines here.
• Interested in conducting federated clinical studies with MedPerf? Join our community.

The medical working group will continue developing MedPerf as a community resource for reproducible and trustworthy AI evaluation in healthcare. Follow MLCommons to track future progress.

About MedPerf
MedPerf is an open-source platform developed by the MLCommons medical working group for benchmarking and evaluating healthcare AI/ML models. Designed for federated clinical research, it enables secure, reproducible AI evaluation across multiple sites without requiring patient data to leave local systems.

About MLCommons
MLCommons is a non-profit engineering consortium dedicated to accelerating machine learning innovation for human benefit through open collaboration across industry, academia, and research institutions. It drives progress by developing benchmarks, datasets, and best practices.