Comparison system issues

  1. How are comparison pairs chosen? Is it uniformly random, or do you give preference to A) designs with fewer comparisons so far or B) pairs that have similar Elo ratings? (A) would make sure every design gets as close to the same number of comparisons as possible. (B) might help refine the ordering a bit.

  2. What do you think of this idea?

  3. Right now a player can only either predict that one design will beat another (which affects their ratings) or skip making a prediction (which doesn’t affect ratings at all). What do you think about adding the option to predict a tie? Ties are not that unlikely in rounds where a lot of good designs get synthesized–by my count, there have been eight ties in the last six rounds, or about 5% of all pairs of round winners–and right now there is no way for players to get points from them or for the rating system to account for them. Ties are still relatively rare, though, so maybe correctly predicting one should get you extra points. Elo is already designed to handle ties (they count as half a win), so that shouldn’t be an issue.

  1. Right now it’s completely random, but we are going toward something similar to (A) which will also take in players’ past lab elo performances as well

  2. Yes - Maybe 2-3 days before synthesis candidates are chosen, Elo should bias its selection to top designs

  3. Yes - we are in process of adding tie.