The fundamental design flaw Project Discovery 2020

Communications Center General Discussion
#1

The fundamental design flaw is:

People who are taking it seriously have a disadvantage.
People who just draw a triangle and submit it will be better at the “game”

It feels like your personal submission is not compared to the community submitted stuff. This new one just says “completed” no matter what you draw. And you get rewards.

At this point I am asking myself:
Am I stupid for taking Project Discovery seriously and taking like 3 hours on some days to mark samples, when I could just be finished in… idk… a hour drawing triangles?

#2

btw… seems like a task that could easily be automated. Does CCP do anything against abusing of PD?

And how is that acceptable?

1224×614 292 KB

#3

If they’re smart they’ll have some kind of system where the same tasks are given to multiple people and only answers that agree within a certain error range are considered valid. So if you’re a shameless asshole you can submit junk data to get the rewards as fast as possible and not get punished for it, but your submissions will get thrown out.

(Now, whether or not you trust CCP enough to believe that they’ve implemented something like that is a question I leave for you to answer.)

1 Like
#4

You are wrong here.

I asked the researcher behind the project (Ryan Brinkman) about this very issue and it seems to not be a problem. This is his reply:

"This was confusing to CCP folks too at the start, and they coded up a gold test gate based on that that doesn’t actually reflect the important information we are trying to capture.

The key here is empty space doesn’t matter in the scoring. What matters its what cells are captured within the boundaries of the gate. The science we are doing is seeing how many cells are inside and how much they chance between different patients. So a big date around the right 50 cells is exactly as current as a tight gate close around them as long as the big gate isn’t contaminated by cells that should be associated with a different gate."

#5

I just stopped doing it since it seemed most of the answers mine were compared to were quite horrible.

#6

I’m not talking about marking cells with less vertices, I’m talking about just drawing a random square or triangle somewhere in the space.
Because right now, there is no system that makes you loose your 99% or whatever and most of the samples are not even rated. You can literally draw a triangle, submit, draw a triangle, submit, repeat.

I just feel bad for all the people that really are giving their best and are not rewarded for it by the game. Instead the people that don’t take it seriously are rewarded.

#7

That’s a very odd result indeed. Samples that receive an accuracy score belong to the so-called gold standard (references) group.

After watching ccp stream on Project Discovery, where the chief scientist involved (Ryan Brinkman) explains a few things in more detail, I would definitely “gate” four clusters (3 in the lower half, 1 in the top half). All those clusters would be populated significantly (have centers colored differently than just the low population blue code). That is exactly what the scientist on the stream said is the next step for better quality analysis. Moreover, both axes (horizontal and vertical) in these 2D plots are logarithmic and cover three orders of magnitude each…

I guess this is an old “gold standard” from the very beginning of Project Discovery. It should be replaced or deleted, because it actually forces you to make a wrong analysis and the wrong conclusions.

P.S. Dr. Brinkman did mention that consistently bad gating by contributors leads to their analyses being taken out of the study (as I understood it, part of this project involves machine learning)

#8

Yeah, and that’s already addressed. It doesn’t matter as long as you’re hitting the cells.

I feel bad for people wasting their lives doing things the wrong way when it doesn’t matter how pristine and well thought out they think their work is when ultimately it doesn’t matter AND EVEN THE LEAD RESEARCHER HAS STATED THAT THE EMPTY SPACES DO NOT MATTER.