The following arXiv preprint claims to have developed an AI program that successfully solves all 100 structures of the original Eterna100 benchmark:
Is this for real?
1 Like
Interesting! Thanks for sharing this. I’ll note that I don’t see any code availability so I can’t verify there. Also it looks like the assumption is that it would run with 200 cores over 24 hours, whereas the benchmark conventionally only uses one core, so it’s not really a fair comparison assuming I’m understanding that correctly.
1 Like
maybe it should be tested before conforming, but if it works it could be very useful
I also wonder how much of the execution time is spent in calls to the secondary structure prediction algorithm, vs. the reinforcement learning they implemented.
In other words, the general flow of all these algorithms (to my knowledge) is a loop of design sequence –> predict SS –> determine “how similar” the predicted SS is to the target SS –> design new sequence. The “smarter” the design of the next sequence, the fewer the number of expected SS predictions that need to be done, but this might require more computation in terms of the design. The benefit of testing fewer designs is more the more expensive the SS predictor is (so Ribonanza would suffer more from a less “smart” algorithm than Vienna, for example). I suspect that for a greedy-type algorithm, which probably does very little computation in the design phase, the repeated calls to Vienna could well amount to >90% of the total execution time. I don’t know if this is less true for their algorithm.
I also don’t know precisely how the “how similar” part is defined–if it’s purely the number of base pairs in common, or if the algorithms use the difference in predicted free energy between the MFE and the target as well.
I will note on a later read I may have been incorrect about the parallelization, it may be a proper comparison. But I’m not certain
Perhaps it would be more correct for them to say that the Eterna 100 collection has been solved for the Vienna energy model using sufficient compute power and by making multiple attempts for each puzzle.
1 Like
@jandersonlee Was there any discussion about or specification in the original Eterna100 paper defining how many attempts are allowed? I always assumed multiple attempts were allowed and part of the expected process that led to the assertion that no algorithm can solve the most difficult puzzles. But I never read the paper carefully and am not familiar with standard computer science benchmarks.
I assumed the same thing. This isn’t AI but with AI you can solve the puzzle by learning from your failures as much as your successes.. If multiple passes is how they solved it , bullly for them, good work.
Multiple attempts are certainly allowed. Most algorithms do some variation of hill-climbing and if you start off in the wrong region of course you may need to start over. My beef is that the Eterna100 contains many hard puzzles that were designed to be hard for players to solve in the Vienna (1.x?) energy model; however better energy models now exist so designing to solve those hard puzzles is now moot in many respects. It would be be better (in my estimation) to set up a new set of target shapes more along an inverse CASP approach, where a shape that is known from crystallography in advance and programs must try to construct a sequence to fold into that shape, or else like Eterna where an artificial target shape is given and lab tests are done to make sure that the constructed sequences fold that way.
1 Like
I was one of the primary designers of the Eterna 100 but I didn’t have the foresight to understand that the Vienna energy model was an approximation to nature that was over the years due to be replaced by better energy models and eventually by non-energy model AI-driven folding algorithms. It started out as a collection of about 40 puzzles that Eli, mat747 and I used to test an Eterna Script based inverse folding algorithm to help and try and improve it. It got expanded to 100 puzzles by picking other player-designed puzzles that were progressively harder to solve (based on the number of players who had solved them). However many of those harder puzzles suffered from the fact that they relied on quirks in the Vienna energy model to make them “tricky” for players to solve - it helped to understand some particular quirk of boosting or blocking in that specific energy model. I suppose that developing strategies for solving those last 10%-30% of puzzles could potentially be applied to alternative energy models as well, but it doesn’t directly help with designing sequences that fold as Nature does. (Actually I should not be quite so harsh. I don’t think the number of puzzles only solvable in Vienna 1.85 (or whatever the particular energy model was) is anywhere near 30%, but the Eterna 100 leaves out riboswitches and pseudoknots and a lot of non-canonical bonding, and it does lean heavily to synthetically derived (versus naturally occurring) shapes, so I still think it could be improved.)
Another new inverse design method that uses the Eterna100 benchmark: https://academic.oup.com/nar/article/53/2/gkae1306/7962006
Apparently the Eterna100 set is being used routinely to evaluate design algorithms.
Excellent. Can’t get better marketing than free through a published article.