Voting Solely on Design Merit - How to Implement?

Since finally entering the LAB part of EteRNA, I have developed some issues and concerns with the voting system, and I know I’m not alone, the boards are filled with others voicing similar concerns.

Before jumping in, however I just want to insert a disclaimer: “None of the ideas below are things which I am even 100% certain I am in favor of. They are just a sampling of issues I’ve noted, and wanted to bring into the public conversation.”

I think there is a case to be made for a many alterations to the voting system, but I am going to bring up only two for the sake of this conversation: Namely, 1) Blind Tally" voting, and 2) “Blind Author” voting.

I might add right up front that I believe neither of these ideas will popular or well-received, however, I believe both could have an ultimately positive effect on the voting process itself, as well as on the results of that process - since both remove information that really has no value to add to the SIMPLE BASIC MERIT of the design submission itself.

  1. “Blind Tally” voting (not showing the amount of votes any design had received until AFTER the voting was closed) would minimize, if not entirely eliminate, the (quite natural) “pile-on” effect; voting with the crowd. Currently it is too easy to make up your mind to cast a vote using this criteria alone, and it makes votes cast this way not have the same true value as a vote cast based on one’s own individual best-effort analysis.

  2. “Blind Author” (not revealing the author of each design until AFTER the voting was closed) would be minimize or eliminate “guru-following;” voting for a design solely because it was composed by a top player. Newer players, or non-top scoring players, would then stand a much greater chance of having their design voted for or selected - assuming the design was good - since they would not be dismissed right away because they are new and unknown.

(Note: This might also necessitate a non-personal submission-naming scheme as well)

Both of these changes aim to necessitate honest personal evaluation of the design alone as criteria to cast a vote, by removing too-easy substitutes for more difficult and time-consuming analysis.

I want to stress that, even though I am advancing these ideas, even “I” myself, would miss this information being included; I LIKE seeing who did what design, and how many other have voted on it; it definitely enhances personal involvement and interest. But I keep catching myself beginning to make voting decisions BASED on these factors, before - (or even totally WITHOUT) - having taken the time to analyse each and every one, and then make my decision based on that analysis - (especially when time is running out), and I feel this pollutes the voting effort, and is bad for EteRNA and its goals.

Granted, these ideas would make the voting analysis effort MUCH more demanding of the players; it takes a lot of time and effort and thought to go through dozens of submissions, I know, and we naturally look for ways to narrow down the candidate base, so that we can concentrate our analysis efforts more profitably and economically. But shouldn’t this winnowing process be based on something like past high nature-score? …or from one’s own growing sense and intuition about chances for success derived from free energy and melting point value ranges? … or simply from how similar a design is to other successful designs? … (instead of how many others chose it, or what amount of esteem the author may command?)

Perhaps the search should be defined and promoted in earnest (among players and staff) - to come up with more appropriate and relevant, and valuable criteria for the design winnowing process, while simultaneously seeking for ways to keep these interesting and valuable - but peril-fraught - criteria like vote tally and author from affecting the voting results in unintended and perhaps even counter-productive ways.

Alternately, perhaps initial winnowing-down should even be conducted by the EteRNA staff rather than the players, as this could also reduce conflict-of-interest issues among players, as well as tap the most knowledgeable resource to influence the most likely successful starting pool of designs.

In closing I just want to say that I am not necessarily lobbying for any of these changes; rather I am just wanting to share my perceptions and concerns, and looking to advance this conversation in all the players interests, and in the interests of the goals of EteRNA itself.

I think this is a fantastic discussion to have. And something that the people working at eteRNA talk a lot about. Thanks so much for starting this discussion! I hope that other players post on what they think is the best RNA selection setup.

Personally I have always been a fan of “Guru following” as you put it. Mainly because I do not have the time to analyze every RNA. So I depend on other people to narrow down the search space for me. Though I agree that a newer player with a good design might have a harder time getting synthesized.

Anyhow, thanks for posting this discussion! What do other people think?

dimension9, thanks for sharing your insights and sorry my response has been delayed.

As you may know one grand goal of EteRNA is to figure out an efficient model to solve the unsolved RNA synthesis problem, and voting is at the center of it. We don’t claim the voting policy we have is final - in fact, it’s only the beginning and we believe it’ll go over MANY changes based on our experiences with users.

At this point, we only had experiences with very few number of people and thus we think it’s too early to make any decisions on the policy change yet. We think actual changes will start to apply when we gain more people and get more feedback after public opening.

The blind tally is definitely in the list of ideas to try. If the “pile-on” effect seems to dominate the voting decision this is probably the first option we’ll try (or increasing the reward on “right” vote to motivate people to think more.)

I’m actually with Alex on the “Blind author” matter. If you have been very successful at lab then I think you deserve to be given more credit. In fact, this will also be a motivation for good players to keep trying their best, since they don’t want to lose their fame. New players, will be motivated to design good RNAs to become a guru themselves.

Of course, this can lead the voting to become a popularity contest, where all people vote for their friends. However, I don’t think blind author will necessarily prevent this problem (They can still say “hey vote for design 11111, that’s mine.”) Rather, I think this problem should be approached by motivating players to vote for the right design.

I think that trying one or more of these options would be good to try. However, you said:

>>Alternately, perhaps initial winnowing-down should even be conducted by the EteRNA staff rather than the players, as this could also reduce conflict-of-interest issues among players, as well as tap the most knowledgeable resource to influence the most likely successful starting pool of designs.

I don’t think that would be fair to anyone, because who is to say that the staff are so much better then the players and judging structures?

I’d like to start a conversation on people’s voting strategies for The Finger. Why did you vote the way you did?

This reply was created from a merged topic originally titled
Why I Voted the Way I Did for the Finger.

So there’s been a bit of discussion about how to make voting in the lab both more fair, and less of a needle in a haystack endeavor.

I think I have a intuitive, democratic idea that could address both problems: instead of just one lab, with top 8 going to synthesis, have a three tier system. Gold lab gets the most designs synthesized (say, 5), silver the next most (2), and bronze the least (just 1).

Everyone starts in bronze at 10,000 pts, as they do currently, but to move up you either have to have a successful design, or vote for successful designs (or maybe win a GU competition, etc). You can only vote for designs from your own lab, and the thresholds for moving up could calibrated to keep the gold and silver labs at a manageable number of participants.

The tierd system would make finding good designs easier (at least for people in gold/silver), and voting for bad Christmas tree designs would have a built in, democratic penalty: if you vote for things that dont fold, you never move up.

-Ribozyme

This reply was created from a merged topic originally titled
Tiered labs for better, easier voting.

I oppose blind author, because implementations of blind author either

  1. allow people to know the ID number of their submission, which encourages chat-spamming of your submission even more than the current system, or
  2. do not allow people to know the ID number of their submission, which completely eliminates any collaboration on designs.

I’m also opposed to the blind tally system, but mostly because the “current number of votes” is the only form of collaboration with other voters I currently have during voting. If I could see anything else regarding other peoples’ opinions of the designs (“good job!” “interesting plan” “seems to be put together randomly” “all GC again”) or even thumbs up and thumbs downs (with thumbs downs not counting against votes) then I’d be more willing to give up the knowledge of how many votes each design has.