iDPP @ CLEF 2023 Guidelines - Intelligent Disease Progression Prediction

Tasks

iDPP CLEF 2023 offers two evaluation tasks focused on predicting the progression of Multiple Sclerosis (MS) and one position papers task on impact of exposition to pollutants on predicting the progression of Amyotrophic Lateral Sclerosis (ALS).

For Tasks 1 and 2 on MS, participants are given a dataset containing 2.5 years of visits. This dataset comes from two clinical institutions, one in Pavia, Italy, and the other in Turin, Italy, and it contains data about real patients, fully anonymized.
For Task 3 on ALS, participants are given a dataset containing 6 months of visits. This dataset comes from two clinical institutions, one in Lisbon, Portugal, and the other in Turin, Italy, and it contains data about real patients, fully anonymized.

All the datasets are highly curated and they are produced from the BRAINTEASER Ontology (BTO), developed by the BRAINTEASER project, which ensures the consistency of the data represented. Moreover, several checks have been performed to ensure that all the instances are clean, contain proper values in the expected ranges, and do not have contradictions.

Task 1 - Predicting Risk of Disease Worsening (Multiple Sclerosis)

Task 1 focuses on ranking subjects based on the risk of worsening, setting the problem as a survival analysis task. More specifically the risk of worsening predicted by the algorithm should reflect how early a patient experiences the "worsening" event, and should range between 0 and 1.

Worsening is defined on the basis of the Expanded Disability Status Scale (EDSS), according to clinical standards.

In particular, we consider two different definitions of worsening corresponding to two different sub-tasks:

Task1a: the patient crosses the threshold EDSS ≥ 3 at least twice within a one-year interval;
Task1b: the second definition of worsening depends on the first recorded value, according to current clinical protocols:
- if the baseline is EDSS < 1, then the worsening event occurs when an increase of EDSS by 1.5 points is first observed;
- if the baseline is 1 ≤ EDSS < 5.5, then the worsening event occurs when an increase of EDSS by 1 point is first observed;
- if the baseline is EDSS ≥ 5.5, then worsening event occurs when an increase of EDSS by 0.5 points is first observed.

For each sub-task, participants are given a dataset containing 2.5 years of visits, with the occurrence of the worsening event and the time of occurrence pre-computed by the challenge organizers.

The training data are available upon registration for the challenge and the test data will be available one week before the run submission deadline. Please see the Datasets and Important Dates sections for more information.

Task 2 - Predicting Cumulative Probability of Worsening (Multiple Sclerosis)

Task 2 refines Task 1 by asking participants to explicitly assign the cumulative probability of worsening at different time windows, i.e., between years 0 and 2, 0 and 4, 0 and 6, 0 and 8, 0 and 10.

In particular, as in Task 1, we consider two different definitions of worsening corresponding to two different sub-tasks:

Task2a: the patient crosses the threshold EDSS ≥ 3 at least twice within a one-year interval;
Task2b: the second definition of worsening depends on the first recorded value, according to current clinical protocols:
- if the baseline is EDSS < 1, then the worsening event occurs when an increase of EDSS by 1.5 points is first observed;
- if the baseline is 1 ≤ EDSS < 5.5, then the worsening event occurs when an increase of EDSS by 1 point is first observed;
- if the baseline is EDSS ≥ 5.5, then worsening event occurs when an increase of EDSS by 0.5 points is first observed.

For each sub-task, participants are given a dataset containing 2.5 years of visits, with the occurrence of the worsening event and the time of occurrence pre-computed by the challenge organizers.

The training data are available upon registration for the challenge and the test data will be available one week before the run submission deadline. Please see the Datasets and Important Dates sections for more information.

Task 3 - Position Papers on Impact of Exposition to Pollutants (Amyotrophic Lateral Sclerosis)

We will evaluate proposals of different approaches to assess if exposure to different pollutants is a useful variable to predict time to Percutaneous Endoscopic Gastrostomy (PEG), Non-Invasive Ventilation (NIV), and death in ALS patients.

This task will be based on the same data and the same design as Task 1 in iDPP CLEF 2022. Therefore, both training and test data are available immediately and you can build on the experience gained last year. The difference with respect to last year's task is that we complement those data with environmental data to investigate the impact of exposition to pollutants on prediction of disease progression.

Since both training and test data are immediately available, we consider these submissions as position papers. However, we expect participants to use training and test data in the usual way, as it if were a regular challenge and the test data were released at the "last minute" without ground-truth, i.e., without the possibility of overfitting.

Participants are asked to rank subjects based on the risk of early occurrence of:

Task3a: NIV (Non-Invasive Ventilation) or (competing event) Death, whichever occurs first;
Task3b: PEG (Percutaneous Endoscopic Gastrostomy) or (competing event) Death, whichever occurs first;
Task3c: Death.

For each of these tasks, participants are given a dataset containing 6 months of visits and are asked to rank patients on the risk of occurrence of one of the above events after month 6. Participants are also given a series of environmental data, such as PM10 (particulate matter with a diameter of 10 microns or less) or CO (Carbon monoxide) or NO₂ (Nitrogen dioxide). Ranges for environmental data series vary depending on the pollutant but, for most patients and on average, they cover from (up to) 90 months before Time 0 up to 6 months after Time 0, where Time 0 is the time of the first ALSFRS-R questionnaire. Please, see the Datasets and the Important Dates sections for more information.

In particular, for each subtask, we ask three types of submissions from participants:

baseline submissions without using any environmental data. Ideally, any submission using environmental data (see below) should be accompanied by a baseline submission, in order to have the possibility of measuring the performance gap due to the use of environmental data;
submissions using only environmental data 6 months before and after Time 0;
submissions using whatever time window of environmental data, as preferred by participants.

Participating

To participate in iDPP CLEF 2023, groups need to register at the following link:

Register

Important Dates

Registration closes: April 28, 2023
Test data release: May 3, 2023
Runs submission deadline: May 10, 2023
Evaluation results out: May 26, 2023
Participant and position paper submission deadline: June 5, 2023
Notification of acceptance for participant and position papers: June 23, 2023
Camera-ready participant papers submission: July 7, 2023
iDPP CLEF Workshop: September 18-21, 2023 during the CLEF Conference in Thessaloniki, Greece

Datasets

Tasks 1 & 2 Datasets (MS)

Tasks 1 and 2 share the same datasets.
The following data are available for the training and test sets of both sub-tasks (a and b):

Static data;
Dynamic data (2.5 years), containing information on: relapses, EDSS scores, evoked potentials, MRIs, and MS course.

For a reference on the EDSS score, please, visit here.

The following data are available as ground-truth:

The worsening-occurrence, as defined in Tasks description, expressed as a Boolean variable with 0 meaning "not occurred" and 1 meaning "occurred".
The time-of-occurrence, expressed as relative delta with respect to Time 0 in years (also fractions).

Training and test datasets follow a (roughly) 80-20% proportion.

Training Dataset

There is a separate dataset for each subtask (a or b).

Sub-task a: available here
441 patients, 481 relapses, 2,661 EDSS scores, 1,211 evoked potentials, 960 MRIs, and 310 MS courses.
Sub-task b: available here
511 patients, 553 relapses, 3,069 EDSS scores, 1,522 evoked potentials, 966 MRIs, and 325 MS courses.

Test Dataset

There is a separate dataset for each subtask (a or b).

Sub-task a: available here, outcome here.
111 patients, 95 relapses, 675 EDSS scores, 278 evoked potentials, 236 MRIs, and 68 MS courses.
Sub-task b: available here, outcome here.
129 patients, 125 relapses, 813 EDSS scores, 299 evoked potentials, 266 MRIs, and 75 MS courses.

Details on the Datasets

Each of the datasets consists of the following files:

static_vars.csv: contains the static variables about a patient.
In the following document, you can find a description of each column in the CSV file.
relapses.csv: contains the (relative) start date of relapses.
In the following document, you can find a description of each column found in the CSV file.
edss.csv: contains the (relative) date when EDSS scores were measured, together with the EDSS score evaluated by clinicians.
In the following document, you can find a description of each column found in the CSV file.
evoked-potentials.csv: contains the tests on evoked potentials.
In the following document, you can find a description of each column found in the CSV file.
mri.csv: contains the data involving MRIs, e.g. the area on which MRIs have been performed and the observed lesions.
In the following document, you can find a description of each column found in the CSV file.
ms-type.csv: contains information about the MS course.
In the following document, you can find a description of each column found in the CSV file.
outcomes.csv: contains the patients' worsening occurrence, together with the time of occurrence.

More in detail, the outcomes.csv file adopts the following format:


100619256189067386770484450960632124211 1 9.4
101600333961427115125266345521826407539 0 6.4
102874795308599532461878597137083911508 0 0.8
123988288044597922158182615705447150224 1 3.3
100381996772220382021070974955176218231 0 15
...

where:

Columns are separated by a white space;
The first column is the patient ID, an hashed version of the original patient ID (should be considered just as a string);
The second column is the worsening-occurrence. It is a Boolean variable with 0 meaning "not occurred" and 1 meaning "occurred";
The third column is the time-of-occurrence. It is a floating point number in the range [0, 15].

Task 3 Dataset (ALS)

The following data are available for both the training and the test set:

The first available ALSFRS-R questionnaire at Time 0 (both single question scores and total score). Thus, for example, time-of-onset and time-of-diagnosis will be expressed as relative delta with respect to Time 0 in months (also fractions);
The slope of the ALSFRS-R score between time-of-onset and Time 0 as:

For a reference on the ALSFRS-R scale, please, visit here.
All the other static data;
Visits (6 months), containing other ALSFRS-R questionnaires or Spirometry (FVC);
Environmental data, such as PM10 (particulate matter with a diameter of 10 microns or less) or CO (Carbon monoxide) or NO₂ (Nitrogen dioxide). Ranges for environmental data series vary depending on the pollutant but, for most patients and on average, they cover from 20 months before Time 0 up to 6 months after Time 0.

The following data are available as ground-truth:

Time of event (NIV, PEG, or Death); or
Censoring time, i.e. time of the last available visit if none of the previous events occurs.

Training and test datasets follow a (roughly) 80%-20% proportion.

Training Dataset

There is a separate dataset for each of the possibile subtasks (a, b, or c).

Sub-task 3a (NIV or Death): available here
1,432 Patients, 3,636 ALSFRS-R questionnaires, 1,185 spirometry tests.
Sub-task 3b (PEG or Death): available here
1,679 Patients, 4,214 ALSFRS-R questionnaires, 1,490 spirometry tests.
Sub-task 3c (Death): available here
1,716 Patients, 4,306 ALSFRS-R questionnaires, 1,518 spirometry tests.

Test Dataset

There is a separate dataset for each of the possibile subtasks (a, b, or c).

Sub-task 3a (NIV or Death): available here, outcome here
346 Patients, 868 ALSFRS-R questionnaires, 273 spirometry tests.
Sub-task 3b (PEG or Death): available here, outcome here
422 Patients, 1,039 ALSFRS-R questionnaires, 358 spirometry tests.
Sub-task 3c (Death): available here, outcome here
486 Patients, 1,210 ALSFRS-R questionnaires, 411 spirometry tests.

Details on the Datasets

Each of the datasets consists of the following files:

static_vars.csv: contains all the static variables about a patient, e.g. onset date, diagnosis data, sex and so on.
In the following document, you can find a description of each column found in the CSV file.
visits.csv: contains all the events about visits, ALFRS-R questionnaires, therapies and so on.
In the following document, you can find a description of each column found in the CSV file.
environmental.csv: contains all the environmental data.
In the following document, you can find a description of each column found in the CSV file.:
outcome.csv: it contains the occurred event (NIV, PEG, DEATH, or NONE) and the time of the event in months, or the censoring time, i.e. the time of the last visit, when the event is NONE.

More in the detail, the outcome.csv file adopts the following format:


0x4bed50627d141453da7499a7f6ae84ab 1 PEG 20.5
0x4d0e8370abe97d0fdedbded6787ebcfc 1 PEG 18.3
0x5bbf2927feefd8617b58b5005f75fc0d 1 DEATH 17.6
0x814ec836b32264453c04bb989f7825d4 0 NONE 37.4
0x71dabb094f55fab5fc719e348dffc85 1 PEG 8.2
...

where:

Columns are separated by a white space;
The first column is the patient ID, a 128 bit hex number (should be considered just as a string);
The second column indicates whether the one of the above events occurred (1) or not (0);
The third column is the occurred event. It comes from a controlled vocabulary and it can be either NIV, PEG, DEATH, or NONE;
The fourth column is the time of the event, or the censoring time, from Time 0 in months.

Participant Repository

Participants are provided with a single repository for all the tasks they take part in. The repository contains the runs, resources, and possibly the code produced by each participant in order to promote reproducibility and open science.

The repository is organised as follows:

submission: this folder contains the runs submitted for the different tasks.
score: this folder contains the performance scores of the submitted runs.
code: this folder contains the source code of the developed system.
resource: this folder contains any additional resources created during the participation.
report: this folder contains the participant report.

The submission and score folders are organized into sub-folders for each task as follows:

submission/task1: for the runs submitted to the first task. Similar structure for the other tasks.
score/task1: for the performance scores of the runs submitted to the first task. Similar structure for the other tasks.

Participants which do not take part in a given task can simply delete the corresponding sub-folders.

Here, you can find a sample participant repository to get a better idea of its layout.

The goal of iDPP CLEF 2023 is to speed up the creation of systems and resources for MS and ALS progression prediction as well as openly share these systems and resources as much as possible. Therefore, participants are more than encouraged to share their code and any additional resources they have used or created.

All the contents of these repositories are released under the Creative Commons Attribution-ShareAlike 4.0 International License.

Run Submission Guidelines

Participating teams should satisfy the following guidelines:

The runs should be submitted in the textual format described below;
Each group can submit a maximum of 10 runs for each subtask, thus amounting to maximum 20 runs for each of Task 1 and Task 2 and 30 runs for Task 3.

Task 1 Run Format

Runs should be submitted as a text file (.txt) with the following format:


100619256189067386770484450960632124211 0.897 upd_T1a_survRF
101600333961427115125266345521826407539 0.773 upd_T1a_survRF
102874795308599532461878597137083911508 0.773 upd_T1a_survRF
123988288044597922158182615705447150224 0.615 upd_T1a_survRF
100381996772220382021070974955176218231 0.317 upd_T1a_survRF
...

where:

Columns are separated by a white space;
The first column is the patient ID, an hashed version of the original patient ID (should be considered just as a string);
The second column is the risk score. It is expected to be a floating point number in the range [0, 1];
The third column is the run identifier, according to the format described below. It must uniquely identify the participating team and the submitted run.

It is important to include all the columns and have a white space delimiter between the columns. No specific ordering is expected among patients (rows) in the submission file.

Task 2 Run Format

Runs should be submitted as a text file (.txt) with the following format:


100619256189067386770484450960632124211 0.221 0.437 0.515 0.817 0.916 upd_T2b_survRF
101600333961427115125266345521826407539 0.213 0.617 0.713 0.799 0.822 upd_T2b_survRF
102874795308599532461878597137083911508 0.205 0.312 0.418 0.781 0.856 upd_T2b_survRF
123988288044597922158182615705447150224 0.197 0.517 0.617 0.921 0.978 upd_T2b_survRF
100381996772220382021070974955176218231 0.184 0.197 0.315 0.763 0.901 upd_T2b_survRF
...

where:

Columns are separated by a white space;
The first column is the patient ID, an hashed version of the original patient ID (should be considered just as a string);
The second column is the cumulative probability of worsening between year 0 and 2. It is expected to be a floating point number in the range [0, 1].
The third column is the cumulative probability of worsening between year 0 and 4. It is expected to be a floating point number in the range [0, 1].
The fourth column is the cumulative probability of worsening between year 0 and 6. It is expected to be a floating point number in the range [0, 1].
The fifth column is the cumulative probability of worsening between year 0 and 8. It is expected to be a floating point number in the range [0, 1].
The sixth column is the cumulative probability of worsening between year 0 and 10. It is expected to be a floating point number in the range [0, 1].
The seventh column is the run identifier, according to the format described below. It must uniquely identify the participating team and the submitted run.

It is important to include all the columns and have a white space delimiter between the columns. No specific ordering is expected among patients (rows) in the submission file.

Task 3 Run Format

Runs should be submitted as a text file (.txt) with the following format:


0x4bed50627d141453da7499a7f6ae84ab 0.897 upd_T3a_EW6_survRF
0x4d0e8370abe97d0fdedbded6787ebcfc 0.773 upd_T3a_EW6_survRF
0x5bbf2927feefd8617b58b5005f75fc0d 0.773 upd_T3a_EW6_survRF
0x814ec836b32264453c04bb989f7825d4 0.615 upd_T3a_EW6_survRF
0x71dabb094f55fab5fc719e348dffc85x 0.317 upd_T3a_EW6_survRF
...

where:

Columns are separated by a white space;
The first column is the patient ID, a 128 bit hex number (should be considered just as a string);
The second column is the risk score. It is expected to be a floating point number in the range [0, 1];
The third column is the run identifier, according to the format described below. It must uniquely identify the participating team and the submitted run.

It is important to include all the columns and have a white space delimiter between the columns. No specific ordering is expected among patients (rows) in the submission file.

Since different time windows may be considered, participants are allowed to submit predictions for a variable number of patients. We encourage participants to submit predictions for as many patients as possible. To avoid favoring runs that consider only a few patients, submitted runs will be evaluated based on their correctness as well as the number of patients included. The number of patients included is also reported in the output of the provided evaluation scripts.

Submission Upload

Runs should be uploaded in the repository provided by the organizers. Following the repository structure discussed above, for example, a run submitted for the first task should be included in submission/task1.

Runs should be uploaded using the following name convention for their identifiers:

<teamname>_T<1|2|3><a|b|c>_[type_]<freefield>

where:

teamname is the name of the participating team;
T<1|2><a|b|c> is the identifier of the task the run is submitted to, e.g. T1b for Task 1, subtask b;
type describes the type of run only in the case of Task 3 (it can be omitted for Task 1 and 2). It should be one among:
- base for a baseline run;
- EW6 when using environmental data in a time window of 6 months before and after Time 0;
- EWP when using environmental data in a time windows chosen by the participant; in this case it is suggested to use freefield to provide information about the adopted time window;
freefield is a free field that participants can use as they prefer to further distinguish among their runs. Please, keep it short and informative.

For example, a complete run identifier may look like

upd_T3a_EW6_survRF

where

upd is the University of Padua team;
T3a means that the run is submitted for Task 3, subtask a;
EW6 means that environmental data in a time window of 6 months before and after Time 0 have been used;
survRF suggests that participants have used survival random forests as a prediction method.

The name of the text file containing the run must be the identifier of the run followed by the txt extension. In the above example

upd_T3a_EW6_survRF.txt

Run Scores

Performance scores for the submitted runs will be returned by the organizers in the score folder, which follows the same structure as the submission folder.

For each submitted run, participants will find a file named

<teamname>_T<1|2|3><a|b|c>_[type_]<freefield>.score.txt

where <teamname>_T<1|2|3><a|b|c>_[type_]<freefield> matches the corresponding run. The file will contain performance scores for each of the evaluation measures described below. In the above example

upd_T3a_EW6_survRF.score.txt

Evaluation Measures

Task 1 - Predicting Risk of Disease Worsening (Multiple Sclerosis)

The effectiveness of the submitted runs will be evaluated using the Harrell's Concordance Index (C-index).

Task 2 - Predicting Cumulative Probability of Worsening (Multiple Sclerosis)

The effectiveness of the submitted runs will be evaluated with the following measures:

ROC curve: the area under the receiver operating characteristic curve at each of the time intervals;
O/E Ratio: the ratio of observed to expected events at each of the time intervals.

To compute the AUROC and O/E Ratio, we applied censoring to the ground truth values using the following schema. Let A, B, C, and D be four subjects, where:

A experienced the outcome at t_A;
B was censored at t_A;
C experienced the outcome at t₃;
D was censored at t₃.

We report the outcome occurrence label and outcome time for each possible scenario of censoring time, which we refer to as t₁, t₂, and t₃ in the table presented below.

* Being the outcome of the subject at censoring time t_i unknown, the subject can not be considered for evaluation at censoring time t_i

Task 3 - Position Papers on Impact of Exposition to Pollutants (Amyotrophic Lateral Sclerosis)

The effectiveness of the submitted runs will be evaluated with the following measures:

ROC curve: the area under the receiver operating characteristic curve at each of the time intervals (6, 12, 18, 24, 30, 36 months);
Concordance Index (C-index).

Computing the Evaluation Measures

Scripts for computing the above evaluation measures are available in the following repository.

Participant Papers

Participants are expected to write a report describing their participation in iDPP CLEF 2023, their proposed solution, what features have been used for prediction, the analysis of the experimental results and insights derived from them.

Participant reports will be peer-reviewed and accepted reports will be published in the CLEF 2023 Working Notes at CEUR-WS, indexed in DBLP and Scopus.

A template for the report is contained in the report folder of each participant repository. The template does not only define the layout of the report but it also provides a suggested structure for it and illustrates the kind of expected contents.
Participants are strongly encouraged to use LaTeX, whose template provides all the details about the report. However, it is possible to also use Word and a basic template is provided as well.

Participant papers are expected to be 10-20 pages in length, excluding references.

The schedule for submission, revision, and camera ready of the participant report is detailed above in the Important Dates section.

Participant papers can be submitted via Easychair at the following link https://easychair.org/conferences/?conf=clef2023 by choosing the Intelligent Disease Progression Prediction (iDPP) track.

Get in Contact

If you need any additional information, please get in contact with us writing to Nicola Ferro, ferrodei.unipd.it.

Participation Guidelines

iDPP CLEF 2023

Task 1 - Predicting Risk of Disease Worsening (Multiple Sclerosis)

Task 2 - Predicting Cumulative Probability of Worsening (Multiple Sclerosis)

Task 3 - Position Papers on Impact of Exposition to Pollutants (Amyotrophic Lateral Sclerosis)

Participating

Tasks 1 & 2 Datasets (MS)

Training Dataset

Test Dataset

Details on the Datasets

Task 3 Dataset (ALS)

Training Dataset

Test Dataset

Details on the Datasets

Task 1 Run Format

Task 2 Run Format

Task 3 Run Format

Submission Upload

Run Scores

Task 1 - Predicting Risk of Disease Worsening (Multiple Sclerosis)

Task 2 - Predicting Cumulative Probability of Worsening (Multiple Sclerosis)

Task 3 - Position Papers on Impact of Exposition to Pollutants (Amyotrophic Lateral Sclerosis)

Computing the Evaluation Measures