Project Discovery - Day 1 results

2016-03-16   |   0 Comments
CCP Citizen Science EVE Online Human Protein Atlas MMOS Project Discovery Proteome

Fig. 1. Confusion matrix of gamer predictions versus HPA predictions. Here columns represent votes of gamers. Confusions occurring with a frequency below 1% are set to black for clarity.
Fig. 2. Frequency of classification per class for gamers (red) compared to HPA annotations (blue). Some classes appear highly over-represented (cytoplasm), while others appear highly under-represented (centrosomes).

Thanks Science Fans!

The response to Project Discovery has been amazing and overwhelming. You have taken to the game in a way none of us expected. Keep it up!

First, in an effort to stream line our interactions with you we have created a number of official social media outlets:

EVE forums: We have a number of official accounts we will respond to questions with.
Twitter: HPA_Discovery - dedicated to posting updates, announcements and all things Project Discovery. Twitter is severely character limited so it's difficult to answer questions there.
Reddits: /r/EVE and /r/projectdiscovery - We will be here for support as well.
Youtube: Human Proteome - We will be posting frequently including a "Discovery with the HPA" video series where we play the game for 5 minutes and you can see what samples we encounter and how we reason though them. We hope to make this a daily upload.

If there is a topic on there we may have missed that you want feedback to, ping a user or tweet it at our HPA_Discovery account.

Over the past week there has been a lot of curiosity about how you are doing so far, so without further ado, here is what we know.

Some of you followed this post from MMOS last Thursday reviewing the participation of the first day of project discovery.
Here is the quick summary from day 1 in case you missed it:

  • >400,000 submissions
  • >23,000 players
  • 14,500 tasks reached consensus

    The analysis in this post includes data for day 1 only, I hope to update you again soon as more data is analyzed.

    Annotation Frequency

    Players seem to annotate with approximately the same frequency as we do here at the HPA (1.4 vs 1.5 classifications per image).

    Annotation Precision

    Overall, gamers achieved a 71% precision (percentage of time something the gamers labeled was predicted by the HPA), with 49% recall (percentage of predicted HPA labels also labeled by the gamers). Gamers exactly matching HPA suggestions 47% of the time during the first day.

    Although the recall and exact matching number may seem low, it is important to note that there are several new categories the HPA was not annotating before, so many mismatches can be attributed to the refinements in the annotations being provided by the gamers. The previously absent Nucleoli-rim pattern is a good example of this where usually HPA predicted Nucleoli, but the gamers have refined our annotations. In the case where a single label was predicted by the HPA, gamers agreed 65% of the time, indicating, not surprisingly, a better exact-match percentage in these cases.

    Further, HPA suggested annotations are based on a number of images for each antibody, combining ~4 fields per cell type for 3 or more cell types into a "best fit" annotation. As a result, gamer agreement with HPA predictions in terms of precision was never expected to be above 70-80%.

    Most of the disagreement between players and HPA predictions involve understandable confusion, such as centrosome vs microtubule organizing center (MTOC) or nucleus vs nucleoplasm. Other classes appear to be go-to annotations for gamers as seen in the confusion matrix, Fig 1. Notice some categories have no data, as the HPA has no annotations for these so a confusion could not be calculated. Off-diagonals in these categories correspond to HPA predictions that were not matched, meaning the HPA felt there was some additional feature not annotated by the gamers.
    Note that since this is a many-to-one confusion matrix, gamers are actually over-penalized occasionally such that if there is more than one unmatched HPA prediction each one gets a count when there is an unmatched gamer prediction.

    Even for images where HPA annotations remain unmatched, gamers seem to be on the right track. Ignoring cytoplasm, and the reasonable confusion between nucleus vs nucleoplasm, the extra HPA prediction(s) not matched were near the decision threshold or the next most popular answer nearly 30% of the time.

    Classification Frequency

    Ultimately these are incredibly encouraging results, though there is some room for improvement. Looking at the classification frequency, we see that some classes are very under-trained (centrosomes vs Negative and MTOC) while others appear to be a go-to for gamers when they are unsure (Cytoplasm) as seen in Fig 2. In fact, cytoplasm accounts for over 47% of the over annotation by gamers. No single class is under annotated more than 14% of the time (over annotation = gamer predicts a feature with no-remaining HPA feature to match. under annotation = HPA predicts feature with no remaining gamer feature to match).

    That's it for this update, but before I go, let me just say congrats to those of you that have gotten the combat suite, (top in game reward)! Amazing! We are planning some IRL rewards for top contributors and some special Project Discovery themed things for Fanfest. In addition to the IRL swag, we will be adding top contributors to the official protein atlas next release and adding them to the acknowledgments of our future publication. Here's hoping for reasonable usernames. Keep up the great work and remember to look out for the details! ...o7