Table of Contents
- Online PPI Challenge Overview
- Article Classification Task
- Interaction Method Task
- The BioCreative Meta-Server
Online PPI Challenge Overview
Introduction
Here we present the details for online participation in the PPI task via the BioCreative Meta-Server (BCMS), and, as announced it is just as and very similar to BioCreative II.5, by providing one or more Annotation Servers. This gives you two advantages:
- You will be able to submit an extra five runs, in addition (and, therefore, independently) to the offline runs you want to submit.
- The test phase will be the week after next, while the next week should be used to ensure your Annotation Server to which the BCMS connects is working properly - meaning you will have one extra week to train/tune your systems.
Dates
- BCMS beta website online: latest on 9th of August, 2010, at approx. 12:00 UTC/GMT
- Training/Setup phase: 9th-13th of August, 2010
- Test phase: 16th-20th of August, 2010
Technical Overview
For participants so far unfamiliar with the functioning of the BioCreative Meta-Server (BCMS), here is a quick outline of the online challenge: The BCMS will send an XML-RPC web-service request to an Annotation Server you provide; this is a simple XML-RPC web server that accepts one String input parameter - the (plain-) text to analyze. The web server needs to expose one or two methods, one for the full-text (Interaction Method Task) and the other for MEDLINE abstracts (Article Classification Task).
After receiving an annotation request/remote method call (i.e., a text string) from the BCMS, the Annotation Server should initiate your text-mining pipeline with the received text and later the result(s) of your text-mining system should be responded back to the BCMS by the Annotation Server as the XML-RPC method's return value. Note that the BCMS has an UPPER TIME CONSTRAINT OF MAXIMALLY 10 MINUTES per article (i.e., if the BCMS does not receive a response within 10 minutes, the annotation request is treated as a failure). The results your Annotation Server sends back to the BCMS must follow a defined format, for both the ACTask and the IMTask. Although the BCMS accepts various annotation result types, such as a gene/protein mentions, protein interaction pairs, etc., in the context of BC III and the PPI task, only two annotation types are relevant/need to be set up:
- "interaction" with a boolean value and a confidence score for the ACTask, and
- "normalizations" with one or more PSI:MI term IDs (together with the confidence score and a unique rank for each term) for the IMTask.
Differences to BioCreative II.5
For participants of BioCreative II.5, here is a short list of the relevant, minor changes in the online setup you need to take care of in your Annotation Servers:
- As there are two tasks on two different corpora (ACT MEDLINE abstracts and IMT fulltext files), each task will require a different XML-RPC method that the BCMS will call on your Annotation Server: for the ACTask the RPC method name your AS should implement/provide needs to be called "Medline.getAnnotation" (new), for IMT it expects "Fulltext.getAnnotation" (as BC II.5).
- Since there are no known sections we can provide from converting PDF to plaintext, there are no more sections in the text string parameter of the RPC methods (sorry for that, as publishers were mostly unwilling to provide XML - Elsevier being the big exception here); This means, the input parameter of your RPC methods will only receive a simple UTF-8 encoded string, the whole article/abstract (instead of a List of Structs with section classifications and section content as in BC II.5). The IMT articles are plaintext conversions (via pdftotext) of the PDF - just as the plaintext article files in the offline setting, but containing the PMID of the article as the first line ("
pubmed:\d+"). Possibly, we can provide the text contained in the HTML files from the test set for the IMT task (sans markup) - which is obviously not possible for the training set, although (Martin bravely and manually collecting the HTML of 300 articles for the test set was a great piece of work already!); if we can provide the stripped HTML text online too, we will inform you during the online "training/setup phase". Equally, the ACT articles are the same as the plaintext MEDLINE abstracts in the offline setting. - The IMS task in essence is a normalization task, i.e., it works just as the protein normalization task of BC II.5 (i.e., annotation type/struct key "normalizations"), the only difference is that you need to report PSI:MI term IDs (e.g., "MI:0006" - without the quotes, naturally) instead of UniProt accessions, i.e., technically the same thing: a string. The database name of the normalization now is different ("OBO-MI", in BC II.5 it was "UniProt") - however, you do not need to provide the "database" key in the annotation, the default has been changed to "OBO-MI" for the challenge (and, most likely, you did not provide this value for BC II.5 either, as it defaulted to "UniProt" then). The ACT task annotation type and result format is exactly the same as for BC II.5 ("interaction").
Article Classification Task
ACT - Remote Procedure Call
The ACTask focuses on binary classification of MEDLINE abstracts together with a confidence score (floating point value in the range (0,1]) and an optional evidence passage (a desirable addition, as it greatly increases the value/usability of a text mining system to humans). Due to the nature of this setup (your AS only sees one article at a time), no explicit ranking of articles is possible and each article is treated as an independent instance. The Annotation Server (AS) needs to expose the following method for the ACT:
Struct Medline.getAnnotation(String text)
I.e., the AS will receive a UTF-8 encoded MEDLINE abstract as input string (the same as the plain-text files distributed in the training, development, and test set for the offline task); I.e., the texts contain additional relevant information apart from title and abstract, such as MeSH terms, Journal name, PMID, etc.). The AS is expected to return a Struct (analogous to Hashes or Key-Value-Dictionaries in most programming languages such as Java, Python, or Perl) to the BCMS. This returned Struct must follow the pre-defined format of the BCMS; The result value you need to return should be formatted as follows:
{ "interaction": [ { "has_interaction": boolean, "confidence": float }, ], }
A more verbose description of this value: the Struct should contain the (hash/dictionary) key "interaction", which in itself should contain a List with exactly one element: another Struct (in case you are wondering: this List is put into here to streamline the result structure with the other annotation types the BCMS accepts). This inner Struct then should contain two keys, "has_interaction" and "confidence", which should hold a Boolean value and a Float value, respectively, which represent the article classification result (true/false and the confidence score) of your text mining system for that article in the sense of the ACT (see PPI task description for more information). The evidence passage for the annotation (which only makes sense for True classifications) can be provided as an optional "evidence" key with the passage string extracted from the abstract's text in the inner Structure, e.g.: { "has_interaction": true, "confidence": 0.983, "evidence": "We show, via yeast two-hybrid, bla..." }.
ACT - Evaluation
For evaluation purposes, Matthew's Correlation Coefficient and regular Accuracy measures will be used to measure pure [boolean, unranked] classification performance. To measure the strength of your confidence values, and to evaluate the results against the primary evaluation function (the AUC iP/R-curve; see BC II.5 evaluation for more details), your full results will then be sorted by their confidence values and PMID (to ensure constant sorting for results with equal confidence values), producing an intrinsic ranking from the confidence scores you reported. I.e., evaluation will be the same as for the offline version, the only difference being that you cannot report an explicit rank for each article due to the nature of the online setting.
ACT - Notes
You will receive the plaintext string of the MEDLINE abstract. If you for some reason do not want to use that string, you can parse the PMID from that text string by using the following regular expression: /^pubmed:(\d+)$/ and then pass that PMID to your text-mining system. I.e., you are not obliged to use the text and are free to use any other/additional information you see fit.
Interaction Method Task
IMT - Remote Procedure Call
The IMTask focuses on annotations of OBO PSI:MI interaction detection method terms on fulltext articles together with a confidence score (float in range (0,1]) and a rank (integer value in range [1,n] where n is the number of MI term annotations you provide for the article) that must be unique with respect to the article (i.e., no PSI:MI term ID may be used twice on the same article and no rank may be used twice on the same article). Ideally, also the evidence passage of each annotation should be provided. The Annotation Server (AS) needs to expose the following method:
Struct Fulltext.getAnnotation(String text)
(Note that the only real difference in this method call to the ACTask method call is the use of "Fulltext" instead of "Medline")
I.e., the AS will receive a UTF-8 encoded string of the text as extracted from the PDF file - same as the plaintext supplied in the training and development set - as input string. As no article ID (i.e., the PMID, which is known in the offline setting as it is the plaintext file's name) would be supplied here, we will add a line containing the PMID as the first line to the String sent to the AS, predending "pubmed:PMID\n\n" to it (where PMID is the variable PMID [a positive integer], naturally).
The AS is then expected to return a (XML-RPC) Struct value (analogous to Hashes or Key-Value-Dictionaries in most programming languages such as Java, Python, or Perl). This returned Struct must follow the pre-defined format of the BCMS; The result your need to return should be formatted as follows:
{ "normalizations": [ { "accession": string, "confidence": float, "rank": integer }, ... ], }
A more verbose description of this value: the Struct should contain the (struct/hash/dictionary) key "normalizations", which in itself should contain a List of inner Struct elements, one for each PSI:MI interaction detection method term you wish to annotate. This inner Struct then should contain three keys:
- "accession": a String - the PSI:MI term ID string, just as used in the offline evaluation, e.g.,
"MI:0006", - "confidence": a Float (the confidence score), and
- "rank": an Integer (rank) value.
- "evidence": Finally, an optional evidence passage for the annotation should be provided as a String value.
"database", with the value "OBO-MI", to define the database you are normalizing to, but the BCMS is set to default to this database value for the time of the BC III challenge. This inner Struct represents a single method annotation of your text mining system for the article string you received from the BCMS in the sense of the IMTask (see PPI task description for more information).
IMT - Evaluation
Will follow the same guidelines as the offline task; meaning the main evaluation function being the AUC-iP/R score, but we will also look into the overall performance using F-scores to encourage participants to not simply return all 115 valid IDs for each article.
IMT - Notes
You can return up to 250 normalizations for a single article (the BCMS default), although there are only 115 valid PSI:MI interaction detection method terms anyways, and if you want to improve the precision result of your system, you should only return a subset of those, too. To make the online task fair wrt. the offline setting, we will add the PubMed ID to the article text by prepending it to each article text String the BCMS will send to your AS (as "pubmed:[the PMID integer]" followed by two newlines).
The BioCreative Meta-Server
Location/URL: http://bcms_beta.bioinfo.cnio.es/
Online Challenge Overview
A special beta-version of the BCMS (as used during BC II.5) will be provided for the duration of two weeks to run the online evaluation. If you wish to participate online, you will need to create an account on the BCMS beta web-site and contact Florian Leitner with the account name and you official BioCreative III team ID to participate. Then we will set up a team and server management environment for you, where you can configure the locations of your Annotation Servers and start/stop/monitor requests coming from the BCMS.
Runs and Time Limits
For each run you wish to submit in the online challenge (and up to 5), you need to set up a distinct Annotation Server (i.e., at a different URL and/or internet port). You have one week to set up your servers and should try to run the complete training set or at least a significant portion thereof through your server in that time. The following week the test phase is held. You can use up to three days (until Wednesday, 12:00 UCT/GMT, whence all teams have to have started their test phase) of this second test phase week for more training/setup, but the test phase will close on Friday, no matter how far you have progressed or when you started. See section Dates for the exact dates. For each article the BCMS sends to your Annotation Server, you will have up to a maximum of 600 seconds time (10 min; although you can configure this value to some lower setting if you think your servers can do better than that) before the BCMS produces a time-out error for that article and re-requests it from the server after a waiting period.
BCMS - Usage
Account Creation
First, you will have to create a new user account on the BCMS beta website (link to the website will be announced latest on Monday, 9th of September, 2010). Then you will need to contact Florian Leitner with the account name and you official BioCreative III team ID; We will then setup the configuration environment for you on the website. If you later want to add other team members so they too can have access to this configuration site, your team members can create their own accounts and you can add those users from the team settings page (naturally, you are also free to simply share the login credentials with your team members; note that it is not recommended that two persons try to configure the same server at the same time, as this might get you confused - although it is not possible to "wreck any [major, at least...] havoc").
Server Setup
Once Florian confirmed you that your environment is ready, you can log in/return/reload the site again and you will see a panel where you can add team information (although this serves no special purposes and you are not obliged to add/fill any data). On all pages, additional help information is displayed in a gray column on the left. After confirming/adding your team info (or by simply "returning to the team-page"), you will arrive at the "Team Page", where you can configure/setup your Annotation Servers. The important information to have available when setting up a new server are:
- The complete (and unique!) URL of each new server; This value could be, for example: "
http://www.example.com:8001/rpc". Note that each URL must be accessible from the Internet (institution firewalls!!!) - An ["admin"] e-mail address to which the BCMS should send reports about any errors or other logs; this can/has to be configured for each server separately.
Team Page Overview
After setting up a server, back on the "Team Page", you will see a few lines added to the central table, telling you about the configuration you chose (server URL, admin e-mail), the state of the server (more on that in short), the number of articles in the server's queue (initially, zero), and the number of failures and the number of log messages (with a link to that log). To the left there is an extensive help on the Team Page explaining what each field means. The initial server state is "disabled"; before you start your server, you should try to "ping" your server's Medline or Fulltext XML-RPC method: this sends a "dummy text" string to the server's corresponding RPC method and just makes sure that some response (e.g., an empty Struct, any other ACT classification, or any IMT results are returned - the actual result content is ignored, as long as it is not a technical failure or timeout) is received. If this "ping" does not work (i.e., the orange message box tells you something else than a successful ping), follow the link to that server's logs to see what went wrong; if you cannot fix/figure out the problem yourself, please contact/e-mail Florian for further assistance. Note that it does not make sense to move on using the server if this "ping" fails, this is the minimum functionality requirement test.
Server Management
A server can, in essence, only be online or offline. If it is online, you cannot change the configuration of the server (mainly, server URL or email address) or reset the article queue. To access the configuration in the online state, you will have to tell the BCMS to "halt" the request cycle of that server, and then change the configuration once the last running request has completed and the server is offline. Once the server is offline (or disabled, which is the initial state), you can change the configuration, but also reset the training set (test set articles will be set by the organizers, obviously) articles by pressing on the buttons to reset a server's article queue. You can do this separately for the ACT and IMT; However, your server is expected to expose the relevant method for each task (see the ACT and IMT remote procedure call sections). This means, if you only are participating in one task, only reset/load the queue with the respective articles. If you want to participate in both tasks, press both buttons. Finally, there is a button to clear a server's queue, which you can use if you want to restart the training, after but also before the articles in the queue were all processed, or if you only participate in one task (ie., the server only provides one of the two XML-RPC methods) and accidentally added the article set from the other task to your queue. Finally, "disabled" (as opposed to "offline") means that you turned your server off and do not want the system to activate the server; For example, if the BCMS is restarted for any other reason, first all AS are set to offline; then, when the BCMS restarts again, servers in "disabled" state will not be started by the system, but "offline" servers will be restarted (set to "online"). However, you can still "ping" a server in disabled state to make sure it is working - e.g., before going back online again.
Training to test phase
If you manage to run through the whole queue of training set articles (for one or both tasks), you can ask Florian to send you the evaluation results for that run to check your performance is as expected (include your server and team ID!). Once your are satisfied with all results of your servers, or the test phase deadline is reached, you can/have to enter the test phase. To do this, contact Florian with your team ID and, if you set up more than five servers or want to only use a subset of the servers you configured in the BCMS interface, the Server IDs or URLs [as shown in your Team Page] you want to participate with. We will then load the test set articles into the queues of the respective servers and you will no longer be able to reset the queues of your servers. However, you will be able to chose when to start (and also pause) the test phase, i.e., when the BCMS should send requests to your Annotation Servers.
Once your queues are loaded, we will inform you, and you can at any time from there on start your servers/test phase runs, either all at once or one after another - as you wish. However, every server will only receive each test set article once (unless an error occurs) and therefore will only be allowed/able to to return one result set per article. If there are any errors or problems, (connection problems, timeouts, etc.) the process will not move on (and resend the same article) until the BCMS has received a valid annotation result (even an empty one is valid!) for that article. If you cannot resolve the error, you will have to skip the article and produce an empty result: In case you, for some reason, want to skip such a "problematic" article, the Team Page will provide you with the option to (via a button) to skip that article; However, this is treated as an "empty" annotation (false negatives!) Ideally, by Friday on end of the test phase week you should have completed all articles (6000 ACT MEDLINE abstracts and/or 300 IMT fulltext articles). If not, the missing articles and annotations will be treated just as skipped articles (i.e., empty annotations, increasing false negatives).
top
