Tasks

INT: Interactor normalization task

(Given an article extract a set of biological entities, namely mentioned interactor proteins)
Extraction of a ranked list of interactor proteins in terms of their primary UniProt accession number limited to a total maximum of 50 proteins per article. Participating teams are asked to return a ranked list of proteins that are detected as being used to describe an interaction in the article (short: "interactor proteins") in terms of their primary UniProt accession number together with a normalized confidence score (again, range: ]0..1]) or an unique integer rank ([1..50]) for each interactor, as well as an optional list of corresponding protein mentions/passages used for extracting the normalization. This optional list provided by participants will serve in the evaluation phase to analyze problematic normalizations. The evaluation will be based on comparison between the automatically generated results and the manual protein normalization done by the authors and by the database curators. The macro-averaged results will be used to score the performance of the submissions, computing the macro-averaged f-score (based on calculating the f-score per-document and then averaging it across documents).

IPT: Interaction pair task

(Extraction of biological relations, namely pairs of interacting proteins)
Extraction of protein interaction pairs in terms of their primary UniProt accession number limited to a total maximum of 50 interactions per article. Participants are asked to return for each article a list of non-redundant, binary interaction pairs together with a normalized confidence score (range: ]0..1]) or an unique integer rank ([1..50]). Directionality of the interaction will not be considered, thus also redundancy resulting from alternative interaction directionality has to me removed from the training and test set evaluation files. The evaluation will be based on comparison between the automatically generated results and the manual protein interaction pair normalization done by the authors and database curators. The macro-averaged results will be used to score the performance of the submissions, computing the macro-averaged f-score (based on calculating the f-score per-document and then averaging it across documents).

ACT: Article categorization sub-task

(Binary classification of articles as relevant for extracting interaction annotations)
Binary classification of full text articles as relevant for extracting/containing protein-protein interaction annotations together with a confidence score for the classification in the ]0..1] range (excluding zero). This is the document classification task. Evaluation will be based on AUC (area under the ROC curve). Here, full text articles will be provided to the participants, as opposed to the IAS of BC2, where only abstracts were used.

Evaluation

Evaluation will be based on direct comparison of automatically generated predictions by participating systems against a Gold Standard of manually generated annotations, so-called Structured Digital Abstracts (SDAs) generated by the original article authors themselves as well we by expert biological database annotators. In order to assure a more robust evaluation setting and to be able to directly compare and visualize different online predictions the BioCreative MetaServer (BCMS) platform will be used.

Technical Features

Currently, there are 12 research groups world-wide providing results from their Annotation Servers to the public via the BioCreative MetaServer platform. Using this simple-to-implement infrastructure will allow you to directly compare your tool's performance to other Annotation Servers - making evaluation of your new algorithms and pipelines simpler - and will give you a head-start for future BioCreative challenges, which will use the platform for evaluation, thus increasing the visibility of your systems. The organizers will provide the participants with sample scripts, technical resources and all information to implement their Annotation Servers.

The issues to solve are:

• XML-RPC communication with the BioCreative MetaServer as defined by the platform;
• Full-text capability (the BCMS will send your AS the text using the XML-RPC service);
• Real-time annotations (not more than a few minutes per article);

An Annotation Server is simply a “remote procedure call” (RPC) web service that the MetaServer - or any other client, if you wish to make your Annotation Server accessible to the general public - can connect to. That is, an RPC function is no different than any other function/method in regular programming languages, but with the added benefit that the function caller can be a program running on a different computer in another location with Internet access. The clients (MetaServer) request annotations for MEDLINE IDs or, as in the case of this challenge, for full-text from the Annotation Server. The Annotation Server then is expected to respond to the request by returning these annotation results as structured data. Setting up such a RPC web-service is fairly simple;  it consists of a server (which can be your oldest PC in the lab) connected to the Internet running a simple script (we will provide examples in various programming languages to the participants) capable of calling your text-mining pipeline with the text it received as parameter. Finally, the script parses/receives the results of your pipeline and sends them back to the client/MetaServer.

To help participants focus on a particular task, the BCMS will allow downloading results from more basic tasks generated by other ASs, which you can use as input for your own system. E.g., if you want to find interaction pairs (IPT task) only, you will be able to first fetch the article classification (ACT task) and all protein normalizations (INT task).

The BioCreative MetaServer and more information about it can be found here: http://bcms.bioinfo.cnio.es/.

Technical background on implementing Annotation Servers can be found here: http://docs.google.com/Doc?id=dwv3dr3_3cn38rsf6

The exact specifications for the full-text Annotation Servers will be published on the particpating team’s pages (see registration below).

Registration and Mailing List

To join the BioCreative mailing list: http://lists.sourceforge.net/mailman/listinfo/biocreative-participant

Additionally to obtain data and get access to all parts of this page (BioCreative copora, workshop proceedings, team registration, etc.): http://www.biocreative.org/accounts/register/

If you are have a user account, you can register a team for BC II.5 at: http://www.biocreative.org/team/create/

Optionally, you can add any additional members to your team with their email and ask those member to create an account on this website with the same email. Everybody in your team then has access to a team page, where we will publish information, downloads, and results for each participating team at: http://www.biocreative.org/team/

References

1. Ceol et al., Linking entries in protein interaction database to structured text: The FEBS Letters experiment., FEBS Letters (2008) vol. 582 (8) pp. 1171-7
2. Gerstein et al., Structured digital abstract makes text mining easy., Nature (2007) May 10;447(7141):142
3. Hahn et al., Text mining: powering the database revolution., Nature (2007) May 10;447(7141):142
4. Hirschman et al., Overview of BioCreAtIvE: critical assessment of information extraction for biology., BMC Bioinformatics (2005) vol. 6 Suppl 1 (1471-2105 (Electronic)) pp. S1
5. Krallinger et al., Evaluation of text-mining systems for biology: overview of the Second BioCreative community challenge., Genome biology (2008) vol. 9 Suppl 2 pp. S1
6. Leitner et al., A text-mining perspective on the requirements for electronically annotated abstracts., FEBS Letters (2008) vol. 582 (8) pp. 1178-81
7. Leitner et al., Introducing meta-services for biomedical information extraction., Genome biology (2008) vol. 9 Suppl 2 pp. S6