Hey all! You may remember that when we introduced RibonanzaNet-SS, we also added metrics to the specbox called eF1 and eF1,cross-pair which were presented in the paper as a way to determine how accurate we predict the model’s prediction is. However, there were a number of limitations in how that metric was formulated, and over the past few weeks we’ve been working on some alternative metrics that we hope will be more accurate and reduce “weird” behavior. We’ve also implemented them as constraints to make it easier to refer to
A demo of how this will work is available on our development site using a variant of OpenKnot Round 4 Week 1 (SARS-CoV-2 frameshift 6XRZ). The graphics are temporary, thrown together by me while Sharif (our designer) comes up with something better 
Looking for a couple bits of feedback:
- Does this score appear usable/“solvable”?
- Any input on how we are displaying these as constraints?
We hope to make this available in the OpenKnot Round 6 labs early this week as folks work on their final submissions to the lab before it closes next weekend. Depending on how things play out, we may make this available more broadly in the specbox - or make further adjustments.
For those interested, some implementation details:
- These metrics work similarly to the existing “TEA” (Target Expected Accuracy) metric in the specbox. However, instead of using the MCC (Matthews Correlation Coefficient) formula, we use the F1 formula. The idea is that instead of a classic F1 where you characterize a prediction as “positive” or “negative” (eg, whether a pair was predicted or not) and “true” or “false” (ie, whether the prediction is accurate or not - which we don’t know yet!), we use weighted values for these categories based on the base pairing probabilities.
- We have variants of these constraints both with respect to the target structure (ie, how confident are we that the sequence will fold into the target) as well as with respect to the natural structure (ie, how confident are we that the sequence will fold into whatever the model predicted).
- For the pseudoknot variants, what we do is “zero out” the contributions of any bases that are not predicted to be “cross pairs”. This means unlike the “global” version, we don’t necessarily include information about whether there are missing cross pairs - instead we are focusing on whether the cross pairs that are predicted are correct.
