iDPP @ CLEF 2022 Guidelines - Intelligent Disease Progression Prediction

Tasks

iDPP CLEF 2022 offers two evaluation tasks focused on predicting the progression of Amyotrophic Lateral Sclerosis (ALS) and one position papers task on explainability of these AI algorithms.

Task 1 - Ranking Risk of Impairment for ALS

Participants are asked to rank subjects based on the risk of early occurrence of

Task1a: NIV (Non-Invasive Ventilation) or (competing event) Death, whichever occurs first;
Task1b: PEG (Percutaneous Endoscopic Gastrostomy) or (competing event) Death, whichever occurs first;
Task1c: Death.

For each of these tasks, participants are given a dataset containing 6 months of visits and are asked to rank patients on the risk of occurrence of one of the above events after month 6.

In particular, for each subtask, we ask two type of submissions from participants:

submissions using only data available until Time 0, i.e. the time of the first ALSFRS-R questionnaire;
submissions using data available until Month 6.

Indeed, from the clinicians point of view, it is of interest to understand what they can say the first time they see the patient (Time 0) and what they can say if they collect additional data for the following 6 months.

Task 2 - Predicting Time of Impairment for ALS

Participants are asked to predict the time of the event. Where the event is

Task2a: NIV (Non-Invasive Ventilation) or (competing event) Death, whichever occurs first;
Task2b: PEG (Percutaneous Endoscopic Gastrostomy) or (competing event) Death, whichever occurs first;
Task2c: Death.

For each of these tasks, participants are given a dataset containing 6 months of visits and are asked to predict a time window of occurrence of one of the above events after month 6.

In particular, for each subtask, we ask two type of submissions from participants:

submissions using only data available until Time 0, i.e. the time of the first ALSFRS-R questionnaire;
submissions using data available until Month 6.

Task 3 - Position Papers on Explainability of AI Algorithms for ALS

This task is not an evaluation challenge but rather discussion papers and proposals on how to make these prediction algorithms explainable, also in a visual way.
However, authors of the papers are welcome to use the datasets below, if they wish to give examples of their algorithms and solutions.

Position papers follow the same schedule for submission, peer revision, and camera ready as the papers of the participants in the challenge. See the section on Important Dates below for details.
Accepted position papers will be published in the CLEF 2022 Working Notes at CEUR-WS, indexed in DBLP and Scopus.

Position papers use the same template as participants papers, available here.
Authors are strongly encouraged to use LaTeX. However, it is possible to also use Word and a basic template is provided as well.

Position papers are expected to be 5-10 pages in length, excluding references.

Position papers can be submitted via Easychair at the following link https://easychair.org/conferences/?conf=clef2022 by choosing the Intelligent Disease Progression Prediction (iDPP) track.

Participating

To participate in iDPP CLEF 2022, groups need to register at the following link:

Registration Closed

Important Dates

Registration closes: April 22, 2022
Runs submission deadline: May 6, 2022
Evaluation results out: May 20, 2022
Participant and position paper submission deadline: May 27, 2022
Notification of acceptance for participant and position papers: June 13, 2022
Camera-ready participant papers submission: July 1, 2022
iDPP CLEF Workshop: September 5-8, 2022 during the CLEF Conference

Datasets

The following data are available for both the training and the test set:

The first available ALSFRS-R questionnaire at Time 0 (both single question scores and total score). Thus, for example, time-of-onset and time-of-diagnosis will be expressed as relative delta with respect to Time 0 in months (also fractions);
The slope of the ALSFRS-R score between time-of-onset and Time 0 as:

For a reference on the ALSFRS-R scale, please, visit here.
All the other static data;
Visits (6 months), containing other ALSFRS-R questionnaires or Spirometry (FVC).

The following data are available only for the training set:

Time of event (NIV, PEG, or Death); or
Censoring time, i.e. time of the last available visit if none of the previous events occurs.

Training and test datasets follow a (roughly) 80%-20% proportion.

Training Dataset

There is a separate dataset for each of the possibile subtasks (a, b, or c) and the same dataset is used for both "Task 1 - Ranking Risk of Impairment for ALS" and "Task 2 - Predicting Time of Impairment for ALS".

Sub-task a (NIV or Death): available here
1,454 Patients, 3,668 ALSFRS-R questionnaires, 1,189 spirometry tests.
Sub-task b (PEG or Death): available here
1,715 Patients, 4,264 ALSFRS-R questionnaires, 1,506 spirometry tests.
Sub-task c (Death): available here
1,756 Patients, 4,366 ALSFRS-R questionnaires, 1,536 spirometry tests.

Test Dataset

There is a separate dataset for each of the possibile subtasks (a, b, or c) and the same dataset is used for both "Task 1 - Ranking Risk of Impairment for ALS" and "Task 2 - Predicting Time of Impairment for ALS".

Sub-task a (NIV or Death): available here
350 Patients, 872 ALSFRS-R questionnaires, 273 spirometry tests.
Sub-task b (PEG or Death): available here
430 Patients, 1,049 ALSFRS-R questionnaires, 361 spirometry tests.
Sub-task c (Death): available here
494 Patients, 1,220 ALSFRS-R questionnaires, 414 spirometry tests.

Details on the Datasets

Each of the datasets consists of the following files:

static_vars.csv: contains all the static variables about a patient, e.g. onset date, diagnosis data, sex and so on.
In the following document, you can find a description of each column found in the CSV file.
visits.csv: contains all the events about visits, ALFRS-R questionnaires, therapies and so on.
In the following document, you can find a description of each column found in the CSV file.
outcome.csv (only for training): it contains the occurred event (NIV, PEG, DEATH, or NONE) and the time of the event in months, or the censoring time, i.e. the time of the last visit, when the event is NONE.

More in the detail, the outcome.csv file adopts the following format:


0x4bed50627d141453da7499a7f6ae84ab 1 PEG 20.5
0x4d0e8370abe97d0fdedbded6787ebcfc 1 PEG 18.3
0x5bbf2927feefd8617b58b5005f75fc0d 1 DEATH 17.6
0x814ec836b32264453c04bb989f7825d4 0 NONE 37.4
0x71dabb094f55fab5fc719e348dffc85 1 PEG 8.2
...

where:

Columns are separated by a white space;
The first column is the patient ID, a 128 bit hex number (should be considered just as a string);
The second column indicates whether the one of the above events occurred (1) or not (0);
The third column is the occurred event. It comes from a controlled vocabulary and it can be either NIV, PEG, DEATH, or NONE;
The fourth column is the time of the event, or the censoring time, from Time 0 in months.

Participant Repository

Participants are provided with a single repository for all the tasks they take part in. The repository contains the runs, resources, and possibly the code produced by each participant in order to promote reproducibility and open science.

The repository is organised as follows:

submission: this folder contains the runs submitted for the different tasks.
score: this folder contains the performance scores of the submitted runs.
code: this folder contains the source code of the developed system.
resource: this folder contains any additional resources created during the participation.
report: this folder contains the participant report.

The submission and score folders are organized into sub-folders for each task as follows:

submission/task1/round1: for the runs submitted to the first task. Similar structure for the other tasks.
score/task1/round1: for the performance scores of the runs submitted to the first task. Similar structure for the other tasks and rounds.

Participants which do not take part in a given task can simply delete the corresponding sub-folders.

Here, you can find a sample participant repository to get a better idea of its layout.

The goal of iDPP CLEF 2022 is to speed up the creation of systems and resources for ALS progression prediction as well as openly share these systems and resources as much as possible. Therefore, participants are more than encouraged to share their code and any additional resources they have used or created.

All the contents of these repositories are released under the Creative Commons Attribution-ShareAlike 4.0 International License.

Run Submission Guidelines

Participating teams should satisfy the following guidelines:

The runs should be submitted in the textual format described below;
Each group can submit a maximum of 5 runs for each subtask, thus amounting to maximum 15 runs for each of Task 1 and Task 2;
For each task, participants are asked to submit two types of runs: either using only the information available at Time 0 or using all the information available in the first 6 months.

Task 1 Run Format

Runs should be submitted as a text file with the following format:


0x4bed50627d141453da7499a7f6ae84ab 0.897 0 PEG upd_T1b_M6_survRF
0x4d0e8370abe97d0fdedbded6787ebcfc 0.773 1 PEG upd_T1b_M6_survRF
0x5bbf2927feefd8617b58b5005f75fc0d 0.773 2 DEATH upd_T1b_M6_survRF
0x814ec836b32264453c04bb989f7825d4 0.615 3 NONE upd_T1b_M6_survRF
0x71dabb094f55fab5fc719e348dffc85 0.317 4 PEG upd_T1b_M6_survRF
...

where:

Columns are separated by a white space;
The first column is the patient ID, a 128 bit hex number (should be considered just as a string);
The second column shows the prediction score that generated the ranking. It is expected to be a floating point number in the range [0, 1]. This score must be in descending (non-increasing) order;
The third column is the rank of the patient by her/his risk of impairment, starting from 0. This is expected to be a strictly increasing integer number. It is important to include the rank so that we can handle tied scores (for a given run) in a uniform fashion;
The fourth column is the predicted event. It comes from a controlled vocabulary and it can be either NIV, PEG, DEATH, or NONE. Note that, since each sub-task is focused on the prediction of a specific event (NIV, PEG, or DEATH), this column will contain that event or the competing event DEATH or NONE;
The fifth column is the run identifier, according to the format described below. It must uniquely identify the participating team and the submitted run.

It is important to include all the columns and have a white space delimiter between the columns.

Task 2 Run Format

Runs should be submitted as a text file with the following format:


0x4bed50627d141453da7499a7f6ae84ab 6-12 PEG upd_T2b_M6_survRF
0x4d0e8370abe97d0fdedbded6787ebcfc 18-24 PEG upd_T2b_M6_survRF
0x5bbf2927feefd8617b58b5005f75fc0d 24-30 DEATH upd_T2b_M6_survRF
0x814ec836b32264453c04bb989f7825d4 >36 NONE upd_T2b_M6_survRF
0x71dabb094f55fab5fc719e348dffc85 >36 PEG upd_T2b_M6_survRF
...

where:

Columns are separated by a white space;
The first column is the patient ID, a 128 bit hex number (should be considered just as a string);
The second column shows the prediction window in months. Possible values are taken from a controlled vocabulary as follows:
- 6-12: the event will happen in the range of months (6, 12];
- 12-18: the event will happen in the range of months (12, 18];
- 18-24: the event will happen in the range of months (18, 24];
- 24-30: the event will happen in the range of months (24, 30];
- 30-36: the event will happen in the range of months (30, 36];
- >36: the event will happen in the range of months (36, +∞).
- >36: the event will happen in the range of months (36, +∞).
The third column is the rank of the patient by her/his risk of impairment, starting from 0. It is important to include the rank so that we can handle tied scores (for a given run) in a uniform fashion;
The fourth column is the predicted event. It comes from a controlled vocabulary and it can be either NIV, PEG, DEATH, or NONE. Note that, since each sub-task is focused on the prediction of a specific event (NIV, PEG, or DEATH), this column will contain that event or the competing event DEATH or NONE;
The fifth column is the run identifier, according to the format described below. It must uniquely identify the participating team and the submitted run.

It is important to include all the columns and have a white space delimiter between the columns.

Submission Upload

Runs should be uploaded in the repository provided by the organizers. Following the repository structure discussed above, for example, a run submitted for the first task should be included in submission/task1.

Runs should be uploaded using the following name convention for their identifiers:

<teamname>_T<1|2><a|b|c>_<train>_<freefield>

where:

teamname is the name of the participating team;
T<1|2><a|b|c> is the identifier of the task the run is submitted to, e.g. T1b for Task 1, subtask b;
train is data window used to train the algorithm. It can be either M0, if only the data available at Time 0 have been used, or M6 if all the data available in the first 6 months have been used;
freefield is a free field that participants can use as they prefer. Please, keep it short and informative.

For example, a complete run identifier may look like

upd_T2b_M6_survRF

where

upd is the University of Padua team;
T2b means that the run is submitted for Task 2, subtask b;
M6 means that the algorithm has been trained using all the data available in the first 6 months;
survRF suggests that participants have used survival random forests as a prediction method.

The name of the text file containing the run must be the identifier of the run followed by the txt extension. In the above example

upd_T2b_M6_survRF.txt

Run Scores

Performance scores for the submitted runs will be returned by the organizers in the score folder, which follows the same structure as the submission folder.

For each submitted run, participants will find a file named

<teamname>_T<1|2><a|b|c>_<train>_<freefield>.score.txt

where <teamname>_T<1|2><a|b|c>_<train>_<freefield> matches the corresponding run. The file will contain performance scores for each of the evaluation measures described below. In the above example

upd_T2b_M6_survRF.score.txt

Evaluation Measures

Task 1 - Ranking Risk of Impairment for ALS

The effectiveness of the submitted runs will be evaluated with the following measures:

ROC curve and/or the precision-recall curve (and area under the curve) to show the tradeoff between clinical sensitivity and specificity for every possible cut-off of the risk scores;
Concordance Index (C-index) to summarize how well a predicted risk score describes an observed sequence of events;
E/O ratio and Brier Score to assess whether or not the observed event rates match expected event rates in subgroups of the model population;
Combination of the previous.

Task 2 - Predicting Time of Impairment for ALS

The effectiveness of the submitted runs will be evaluated with the following measures:

Classification performance (e.g. precision, recall, accuracy) across different intervals;
Weighted average distance from true time window;
Combination of the previous.

Participant Papers

Participants are expected to write a report describing their participation in iDPP CLEF 2022, their proposed solution, what features have been used for prediction, the analysis of the experimental results and insights derived from them.

Participant reports will be peer-reviewed and accepted reports will be published in the CLEF 2022 Working Notes at CEUR-WS, indexed in DBLP and Scopus.

A template for the report is contained in the report folder of each participant repository. The template does not only define the layout of the report but it also provides a suggested structure for it and illustrates the kind of expected contents.
Participants are strongly encouraged to use LaTeX, whose template provides all the details about the report. However, it is possible to also use Word and a basic template is provided as well.

Participant papers are expected to be 10-20 pages in length, excluding references.

The schedule for submission, revision, and camera ready of the participant report is detailed above in the Important Dates section.

Participant papers can be submitted via Easychair at the following link https://easychair.org/conferences/?conf=clef2022 by choosing the Intelligent Disease Progression Prediction (iDPP) track.

Get in Contact

If you need any additional information, please get in contact with us writing to Nicola Ferro, ferrodei.unipd.it.

Participation Guidelines

iDPP CLEF 2022

Task 1 - Ranking Risk of Impairment for ALS

Task 2 - Predicting Time of Impairment for ALS

Task 3 - Position Papers on Explainability of AI Algorithms for ALS

Participating

Training Dataset

Test Dataset

Details on the Datasets

Task 1 Run Format

Task 2 Run Format

Submission Upload

Run Scores

Task 1 - Ranking Risk of Impairment for ALS

Task 2 - Predicting Time of Impairment for ALS