Participation Guidelines

iDPP CLEF 2022

Tasks

iDPP CLEF 2022 offers two evaluation tasks focused on predicting the progression of Amyotrophic Lateral Sclerosis (ALS) and one position papers task on explainability of these AI algorithms.

Task 1 - Ranking Risk of Impairment for ALS

Participants are asked to rank subjects based on the risk of early occurrence of

  • Task1a: NIV (Non-Invasive Ventilation) or (competing event) Death, whichever occurs first;
  • Task1b: PEG (Percutaneous Endoscopic Gastrostomy) or (competing event) Death, whichever occurs first;
  • Task1c: Death.

For each of these tasks, participants are given a dataset containing 6 months of visits and are asked to rank patients on the risk of occurrence of one of the above events after month 6.

In particular, for each subtask, we ask two type of submissions from participants:

  • submissions using only data available until Time 0, i.e. the time of the first ALSFRS-R questionnaire;
  • submissions using data available until Month 6.

Indeed, from the clinicians point of view, it is of interest to understand what they can say the first time they see the patient (Time 0) and what they can say if they collect additional data for the following 6 months.

Task 2 - Predicting Time of Impairment for ALS

Participants are asked to predict the time of the event. Where the event is

  • Task2a: NIV (Non-Invasive Ventilation) or (competing event) Death, whichever occurs first;
  • Task2b: PEG (Percutaneous Endoscopic Gastrostomy) or (competing event) Death, whichever occurs first;
  • Task2c: Death.

For each of these tasks, participants are given a dataset containing 6 months of visits and are asked to predict a time window of occurrence of one of the above events after month 6.

In particular, for each subtask, we ask two type of submissions from participants:

  • submissions using only data available until Time 0, i.e. the time of the first ALSFRS-R questionnaire;
  • submissions using data available until Month 6.

Task 3 - Position Papers on Explainability of AI Algorithms for ALS

This task is not an evaluation challenge but rather discussion papers and proposals on how to make these prediction algorithms explainable, also in a visual way.
However, authors of the papers are welcome to use the datasets below, if they wish to give examples of their algorithms and solutions.

Position papers follow the same schedule for submission, peer revision, and camera ready as the papers of the participants in the challenge. See the section on Important Dates below for details.
Accepted position papers will be published in the CLEF 2022 Working Notes at CEUR-WS, indexed in DBLP and Scopus.

Position papers use the same template as participants papers, available here.
Authors are strongly encouraged to use LaTeX. However, it is possible to also use Word and a basic template is provided as well.

Position papers are expected to be 5-10 pages in length, excluding references.

Position papers can be submitted via Easychair at the following link https://easychair.org/conferences/?conf=clef2022 by choosing the Intelligent Disease Progression Prediction (iDPP) track.

Participating

To participate in iDPP CLEF 2022, groups need to register at the following link:

Registration Closed

Important Dates

Datasets

The following data are available for both the training and the test set:

The following data are available only for the training set:

Training and test datasets follow a (roughly) 80%-20% proportion.

Training Dataset

There is a separate dataset for each of the possibile subtasks (a, b, or c) and the same dataset is used for both "Task 1 - Ranking Risk of Impairment for ALS" and "Task 2 - Predicting Time of Impairment for ALS".

Test Dataset

There is a separate dataset for each of the possibile subtasks (a, b, or c) and the same dataset is used for both "Task 1 - Ranking Risk of Impairment for ALS" and "Task 2 - Predicting Time of Impairment for ALS".

Details on the Datasets

Each of the datasets consists of the following files:

More in the detail, the outcome.csv file adopts the following format:


0x4bed50627d141453da7499a7f6ae84ab 1 PEG 20.5
0x4d0e8370abe97d0fdedbded6787ebcfc 1 PEG 18.3
0x5bbf2927feefd8617b58b5005f75fc0d 1 DEATH 17.6
0x814ec836b32264453c04bb989f7825d4 0 NONE 37.4
0x71dabb094f55fab5fc719e348dffc85 1 PEG 8.2
...
					

where:

Participant Repository

Participants are provided with a single repository for all the tasks they take part in. The repository contains the runs, resources, and possibly the code produced by each participant in order to promote reproducibility and open science.

The repository is organised as follows:

The submission and score folders are organized into sub-folders for each task as follows:

Participants which do not take part in a given task can simply delete the corresponding sub-folders.

Here, you can find a sample participant repository to get a better idea of its layout.

The goal of iDPP CLEF 2022 is to speed up the creation of systems and resources for ALS progression prediction as well as openly share these systems and resources as much as possible. Therefore, participants are more than encouraged to share their code and any additional resources they have used or created.

All the contents of these repositories are released under the Creative Commons Attribution-ShareAlike 4.0 International License.

Run Submission Guidelines

Participating teams should satisfy the following guidelines:

Task 1 Run Format

Runs should be submitted as a text file with the following format:


0x4bed50627d141453da7499a7f6ae84ab 0.897 0 PEG upd_T1b_M6_survRF
0x4d0e8370abe97d0fdedbded6787ebcfc 0.773 1 PEG upd_T1b_M6_survRF
0x5bbf2927feefd8617b58b5005f75fc0d 0.773 2 DEATH upd_T1b_M6_survRF
0x814ec836b32264453c04bb989f7825d4 0.615 3 NONE upd_T1b_M6_survRF
0x71dabb094f55fab5fc719e348dffc85 0.317 4 PEG upd_T1b_M6_survRF
...
					

where:

It is important to include all the columns and have a white space delimiter between the columns.

Task 2 Run Format

Runs should be submitted as a text file with the following format:


0x4bed50627d141453da7499a7f6ae84ab 6-12 PEG upd_T2b_M6_survRF
0x4d0e8370abe97d0fdedbded6787ebcfc 18-24 PEG upd_T2b_M6_survRF
0x5bbf2927feefd8617b58b5005f75fc0d 24-30 DEATH upd_T2b_M6_survRF
0x814ec836b32264453c04bb989f7825d4 >36 NONE upd_T2b_M6_survRF
0x71dabb094f55fab5fc719e348dffc85 >36 PEG upd_T2b_M6_survRF
...
					

where:

It is important to include all the columns and have a white space delimiter between the columns.

Submission Upload

Runs should be uploaded in the repository provided by the organizers. Following the repository structure discussed above, for example, a run submitted for the first task should be included in submission/task1.

Runs should be uploaded using the following name convention for their identifiers:

<teamname>_T<1|2><a|b|c>_<train>_<freefield>

where:

For example, a complete run identifier may look like

upd_T2b_M6_survRF

where

The name of the text file containing the run must be the identifier of the run followed by the txt extension. In the above example

upd_T2b_M6_survRF.txt

Run Scores

Performance scores for the submitted runs will be returned by the organizers in the score folder, which follows the same structure as the submission folder.

For each submitted run, participants will find a file named

<teamname>_T<1|2><a|b|c>_<train>_<freefield>.score.txt

where <teamname>_T<1|2><a|b|c>_<train>_<freefield> matches the corresponding run. The file will contain performance scores for each of the evaluation measures described below. In the above example

upd_T2b_M6_survRF.score.txt

Evaluation Measures

Task 1 - Ranking Risk of Impairment for ALS

The effectiveness of the submitted runs will be evaluated with the following measures:

Task 2 - Predicting Time of Impairment for ALS

The effectiveness of the submitted runs will be evaluated with the following measures:

Participant Papers

Participants are expected to write a report describing their participation in iDPP CLEF 2022, their proposed solution, what features have been used for prediction, the analysis of the experimental results and insights derived from them.

Participant reports will be peer-reviewed and accepted reports will be published in the CLEF 2022 Working Notes at CEUR-WS, indexed in DBLP and Scopus.

A template for the report is contained in the report folder of each participant repository. The template does not only define the layout of the report but it also provides a suggested structure for it and illustrates the kind of expected contents.
Participants are strongly encouraged to use LaTeX, whose template provides all the details about the report. However, it is possible to also use Word and a basic template is provided as well.

Participant papers are expected to be 10-20 pages in length, excluding references.

The schedule for submission, revision, and camera ready of the participant report is detailed above in the Important Dates section.

Participant papers can be submitted via Easychair at the following link https://easychair.org/conferences/?conf=clef2022 by choosing the Intelligent Disease Progression Prediction (iDPP) track.

Get in Contact

If you need any additional information, please get in contact with us writing to Nicola Ferro, ferrodei.unipd.it.

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the grant agreement No GA101017598. The European Commission’s support for this project and the production of this website does not constitute an endorsement of the contents, which reflect the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein