@wisdave - congrats on your upcoming retirement! I realize it was not the most practical suggestion, nor am I implying that people should have to take courses to design, I just wanted to convey that you have an unusual resource in your backyard, as the UW biochemistry department is really a standout department, and I can say that never having attended there but having interacted and followed the work of many of the profs there and their students who are all doing amazing work.
I’ve kept track of my own predictions for a few previous rounds out of curiosity. Now that Jeehyung has released results from the pairwise comparison beta, I’ve gone and plotted my predictions vs. actual scores and gotten a Kendall tau rank correlation coefficient of 0.511. That seems pretty good to me, especially since I’m only one person and am nowhere near the savviest lab participant. In addition, my predictions have gotten more accurate in successive lab rounds, which suggests that I’m learning how to predict better.
The Kendall tau I got probably can’t be directly compared to the one Jeehyung calculated for the beta results for a number of reasons, but it does seem to bode well for a prediction-based selection system. I’ve made predictions for Lab 201 Round 1 as well so I’ll have an update when the synthesis results come out and compare them to the beta results for that round (which should be a fairer fight since the beta will only have to deal with one round worth of designs instead of four). The best thing would still be to implement predictions as just a way to earn points first so we can get data on it from more than just one player.
Here are my predictions (left column) and the actual synthesis scores (right column), along with the mean absolute error (MAE) and Kendall’s tau for each round:
@jee regarding the Elo Ranking results, I’m wondering if its possible to just have a little bit more data.
How many times each design was involved in a pairing comparison?
How many contradictions there are? Like if I compared designs A, B and C. Where I voted A is better than B, then voted B is better than C, And then I voted C is better than A. Sort of like an error factor?
Interesting Elo facts
Average Player went through 22 Games and completed 19 Comparisons.
And about 16% of the Games were skipped. Most probably due to indecision or not willing to compare.
I wonder what the correlation will be for the first round of a shape though. Are you keeping track of Round One of the Bulged Star? I have to admit that my predictions for this one vary depending on my mood, sometimes I think we’ll nail it and sometimes I think we’ll average around 70
@alan.robot: According to slide 19 of the powerpoint, apparently what we call a “bulge” is just a small internal loop, and an actual bulge is something else (what I would have called a “kink” if I hadn’t read this powerpoint).
Sharp eye aldo, I completely missed that! And being precise with the terminology also helps you match up with the right table to look at in the energy terms: http://rna.urmc.rochester.edu/NNDB/tu…
@Berex, I only made predictions for the four rounds above. I stopped making them when it looked pretty certain we were going to go with the pairwise comparison method, then I got curious again lately.
