Motivation

IMPORTANT: all the details, datasets and resources related to the SYMPTEMIST are provided directly at the: SYMPTEMIST web

There has been a substantial effort in developing text mining and NLP resources to automatically process clinically relevant information for entities such as diseases or medications, as well as on automatic anonymization or de-identification of clinical records.

Despite the crucial relevance of clinical symptoms, representing a key concept for diagnosis, prognosis as well as understanding of diseases and individual patient characteristics, far less effort was made to automatically detect and normalize symptoms from clinical texts. This can be partially explained by the considerable difficulty in properly annotating and characterizing symptoms within clinical content, being particularly complex and heterogeneous due to the complexity of medical language and the diversity of existing conditions and clinical specialties.

In the light of recent developments and exploitation of language models and transformer technologies, there is a pressing need to enable more systematic extraction and normalization of symptoms from clinical documents using named entity recognition approaches. Proper annotation criteria and annotation guidelines, together with high quality manually annotated Gold Standard corpora, with clear multilingual adaptation potential are needed to foster the development and evaluation of automatic symptom extraction tools.

Track goals

In an effort to improve and evaluate systems for automatic detection and normalization of symptoms from clinical texts, the SYMPTEMIST track invites researchers, biomedical industry professionals, NLP, and ontology experts to develop systems capable of detecting automatically mentions of clinical symptoms and normalizing or mapping them to a widely used multilingual clinical vocabulary, namely SNOMED CT.

Task Definition

The SYMPTEMIST will be structured into two independent sub-tasks, each taking into account a particularly important use case scenario:

SYMPTEMIST-entities subtrack: requires automatically finding symptom mentions in published clinical cases.

SYMPTEMIST-linking subtrack: requires automatically finding disease mentions in published clinical cases and assigning, to each mention, a Snomed-CT term.

Training data

The SYMPTEMIST corpus, is a manually annotated corpus consisting of 1000 clinical case reports written in Spanish, where several clinical experts have exhaustively labeled mentions of clinical symptom mentions. All mentions were also manually mapped to their corresponding SNOMED CT concept identifiers

Evaluation Metrics

Evaluation will be done by comparing the automatically generated results to the results generated by manual annotation of experts. Evaluation metrics The primary evaluation metric for the sub-track 1 (SYMPTEMIST – entities) and sub-track 2 (SYMPTEMIST – linking) will consist of micro-averaged precision, recall and F1-scores.

Task Organizers:

• Martin Krallinger, Barcelona Supercomputing Center
• Salvador Lima, Barcelona Supercomputing Center
• Eulalia Farre, Barcelona Supercomputing Center
• Luis Gasco, Barcelona Supercomputing Center