If there is a 1-1 loop (only one un-bonded nt opposite only one un-bonded nt), then:
If both nts are Gs, then add 5 points.
If one nt is a G and the other nt is an A, then add 3 points.
If both nts are A, then add 1 point.
Thank you,
Merryskies
Dear merryskies,
Your strategy has been added to our implementation queue with task id 8. You can check the schedule of the implementation here.
ETA of the implementation is Monday 6/6/2011
Thanks for sharing your idea!
EteRNA team
Dear Merryskies,
We are glad to report that your strategy has been implemented and tested.
We could not get a very good testing out of your strategy. It’s NOT because your strategy is wrong, but because your strategy could not make a distinction between designs when they don’t have 1-1 loops. (Lab puzzles The Finger, The Star, The Cross, and The Branches don’t have 1-1 loop at all).
We would love to hear more comprehensive version of this strategy that can cover broader range of shapes!
Length : Your strategy was implmented with 20 line of code.
Ordering : We ran your strategy on all synthesized designs and ordered them based on predicted scores. The correlation of your strategy’s ordering with the ordering based on the actual scores was -0.473. (1.0 is the best score, -1.0 is the worst score. A completely random prediction would have 0 correlation)
Please note that the numbers specified above will change in future as we’ll rerun your algorithm whenever new synthesis data is available.
More detailed result has been posted on the strategy market page. Thank you for sharing your idea, and we look forward to other brilliant strategies from you!
So if you took the opposite of this strategy (i.e. subtracted the points from 100 instead of adding them to 0) you’d get a correlation of 0.473, which would be the best so far? Seems hard to believe given that the only information used is from 1x1 internal loops, which aren’t even present in every lab. In fact, I get a tau of -0.032 when I run R’s Kendall function on the data.
@ aldo, a
As you pointed out most designs don’t even have 1-1 loops, and merry’s algorithm scores all of them 0. Note that we consider the decision to be wrong when the algorithm says two designs tie, but they actually don’t. That’s why merry’s data got negative correlation.
If we were to subtract scores from 100, most designs will get 100 and the correlation would be still very low. In fact, I tried this and it gives out -0.426.
But that wouldn’t be a low correlation, that would be a fairly decent negative correlation. If you flip data with a negative correlation shouldn’t you get data with a positive correlation?
Also, for what it’s worth, I get taus of 0.123 and 0.674 respectively for individual labs 103 and 201 (the two completed labs that have 1x1 internal loops), again using R’s Kendall function.
aldo, the KT correlation only looks at how good the strategy is in “pair-wise ordering things”.
if the actual score of 5 designs were 100, 80, 60, 40, 20 and a strategy scores them 0, 0, 0, 0, 0 - it’s KT correlation value will be -1, since it didn’t get any pair-wise ordering right. The exactly same thing happens even if the strategy scores everything 100, 100, 100, 100, 100. It still gets every pair-wise ordering wrong, and it’s KT correlation value will be -1.
Yes you are right that in general, when you have negative correlation and you flip it, you’ll get a high correlation. In this case however, the problem is that the strategy says everything is a “tie” and the flipping actually does not affect the outcome that much.
if the actual score of 5 designs were 100, 80, 60, 40, 20 and a strategy scores them 0, 0, 0, 0, 0 - it’s KT correlation value will be -1, since it didn’t get any pair-wise ordering right.
But that would be saying that there’s a negative correlation between the predictions and actual scores when in fact there is no correlation. A negative correlation (e.g. 100, 80, 60, 40, 20 vs. 20, 40, 60, 80, 100) would actually be useful because it could be flipped to produce a positive correlation, whereas a zero correlation provides no useful information. From what I can tell, merryskies has close to zero rank correlation over all the data but positive rank correlation for labs 103 and 201 individually.
actually aldo, that’s a really good point.
We have been considering = when < or > to be discordant, but we should count that as neither concordant and discordant.
All strategy scores have been updated with the new standard.
Dear merryskies,
We made a change in the way we score the ordering. (Detail : Before if your algorithm said score of A and B is the same when they are not, it decreased your ordering score. In the new scoring, such decision does neither increase nor decreases your ordering score).
Your ordering score is now -0.0069