In last few rounds after the public launch, we have witnessed a few shortcomings of our current voting system for selecting synthesis candidates. Because now we have hundreds of lab submissions now, it is extremely hard to go through all designs to pick out the best one. In most cases, people are intimitdated by the number of designs they have to look through, and often decide to vote on designs that already got lots of votes (“snowballing”) or only look at single metric (such as free energy) to vote. Devs had a meeting about this today and came up with a conclusion that the voting system is not suitable for the system like us, where we have massive number of candidates.
Instead, we are now thinking of applying Elo Rating System for synthesis candidate selection. If you saw the movie “The social network”, you’ll recognize this system right away as this was used for the FaceMash system. In this system, users are continuously asked to pick the better of 2 candidates. Each user decision create a partial ordering of candidates, and the system tries to come up with a total ordering out of all partial orderings trying to minimize inconsistency. In the end, we will be synthesizing top 8 designs in the total ordering.
Instead of “voting”, there will be a “review” button, where you’ll be asked to pick the better of 2 randomly picked candidates. In the review interface, you’ll be able to do full comparison of 2 candiates - you’ll be able to see their statistics and interactively play with both deigns. You can do as many “reviews” as you want, and you’ll be rewarded by how many “correct reviews” did.
The system has many great advantages. First, every design will be reviewed by someone. We can setup a system such that designs that didn’t get reviewed are more likely to be chosen for random review candidates and it’lll make sure very design gets reviewed. Second, one-to-one comparison will allow players to make more in-depth decision than having to go through hundreds of designs. Third - the quiz like quality of the review will stimulate people to learn & ask more before they make decisions.
The system does have few issues. The biggest issue is that it would reduce social aspect of candidate selection. For example, it’ll be now hard to tell people “vote for my design ABCDE, I think it’s cool” in chat to promote your design. We plan to solve this issue by still having the “design browser” so you can still browse through every submitted designs and review a specific design, and even start doing “reviews” relevant to that specific design (i.e, fix one candidate of the review to be that specific design). Also, we could allow people to leave comments on each design when they review, so people who come later can see them.
The details still needs to be worked out. For example, how are we going to reward people based on their reviews? - do we say a review is correct or wrong only if 2 candidates in the review are synthesized and can be compared? If not, how can we rate reviews that involve non synthesized designs? We are still working on these questions and it may take some time for us to come up with a final system, but we wanted to throw this idea out to EteRNA players and see what players think.