ELO scoring has been suggested (which I generally think is good) - but the current voting system would not allow you to try testing any ‘bad’ hypothesis
So I’m suggesting a new voting system
Everyone can vote for a design if they want (but limited to 8 votes)
Everyone starts with ‘no opinion’ for each design
Everyone can choose to ‘thumbs up’ or ‘thumbs down’ a design
Those designs that get the most votes in total (up + down) are the ones selected
if you thumbed up a design that got selected and it works - gain points
if you thumbed down a design that got selected and it works - lose points
if you thumbed up a design that got selected and it doesn’t work - lose points
if you thumbed down a design that got selected and it doesn’t work - gain points
I’d suggest you don’t show how many votes are currently in for any particular design (just show your votes) - that way people can’t be biased by other people’s opinions (and actually have to decide on their own)
or add
5) no votes may be cast until 3 days after the lab has been opened (meaning people get to consider a range of designs before voting starts).
Suggest you (eventually?) weight the votes - giving more value to those people who have previously ‘voted correctly’ - being more likely to vote correctly again in future. I think these weightings should be applied to the ‘total number of votes’ calculation - when determining which designs to try and synthesise
Possible (if you mostly want successes) that you might want to add that weighting if (and only if) that vote is cast in favour of the design being successful.
Oh and give points to the creator of a design, if it is picked (by total number of votes) not based on whether it is success/failure (allowing people to design for failure to test something).
These ideas are great! (As a side note, it’s really cool that the community is still doing incredibly well despite obvious weaknesses in the candidate selection process.) We’re deferring major changes to the voting system as we prepare our initial results for publication. Our next priority will be concentrating on version 2 of the voting system built around much higher throughput – which may in fact obviate the need for “voting.”
You’ve been deferring changes to the voting process for 6 months (just re-read some of the old posts.) How long is your paper going to take? And how much time can some of the voting changes take to implement? As I read old posts, it seems that people have left EteRNA because of voting inequities and no response on your part other than “Great idea … sometime in the future … working on paper … blah blah.”
Here are 2 suggestions for trial that should be really quick and easy to implement:
Hide the vote count until after voting closes.
Hide the designer’s name and design name (use only system-generated ID number) until after voting closes.
These don’t have to be permenant and they are not a full solution, but worthwhile testing, and no harm in doing so.
Slydog. Thanks for the encouragement. We’re really glad you’re playing.
To address your very valid concerns: writing the paper does take months (that’s the price of careful science, unfortunately!). For EteRNA especially, a prime contributor to the wait is that the biochemical experiments take a long time. Plus, our choice of experiments has evolved as the community’s performance has improved, slowing things down further.
At the same time, we’ve promised ourselves not to change the “structure” of the game (even as we acknowledge it’s limitations) during these many long months because doing so would complicate the experiment. Even if we could achieve better results, say, by anonymizing the voting, other scientists could look at our results and claim that we’re change the rules as we go along, running an uncontrolled experiment and that our results are therefore not valid.
That may seem like a crazy criticism to you (and us) because we all understand EteRNA so well. But when we send out our results to real reviewers who cannot take the time to fully understand what we’re doing, it’s importnat that we present them with the simplest possible narrative of how we conducted these experiments.
The good news is that we hope to have finished running experiments in a few weeks (really!) and that, at that point, we will be much freer to make changes to the game itself without worrying about “tainting” the results. And when that happens, we really need the creative input of you and the rest of the community to make EteRNA as fun, and scientifically effective as possible.
I hope this all seems reasonable, and we welcome further thoughts and comments.
Actually, that makes perfect sense to me. I totally understand the need to maintain consistency for research purposes, I just didn’t understand the nature of the paper you are working on. But this hints at it and I can understand the need to continue as is for now. Thanks for the great explanation!
