Discussion of the switch scores

1

There has been some bewilderment about the switch scores lately after the results of the Cloud Switch lab data got published. A lot of the scores did get revised, but some of them lookes just like earlier. I asked Rhiju and got an explanation. He ask us to think about how we want the switch score. Hereby I pass on the conversation between us:

Hi Rhiju!

News post said to mention if we found anything odd with the scored. I got really happy when I saw Hotcreek score a clean winner. But sad when I saw the scores on the two separate states. I hope it is just me reading it wrong. I know it is hard getting all bases scoring correct in the winners. This one looks exactly like the former problem with the scores.

Link to the design ingame

Eli

Eli
I just took a look – I think the numerical scores are doing what they are supposed to, but, maybe that’s not great.

For hotcreek, the nucleotides that change pairing status do go in the right direction. so it gets a perfect switch score!

Obviously, as you point out the separate state scores aren’t so great. In this case, we were more interested in getting evidence of the right conformational change than making sure that every hairpin is super-solid. But you’re right that it is important to reward and discriminate designs that have the ‘whole package’ rather than somehow winning on a technicality.

Perhaps we can devise a hybrid score – something like

(1/2) * switch_score + (1/4) * shape_score + (1/4) * shape_score

and that can be the ‘final’ score though we can also present all three.

What do you and the other players think? Or is there a better scoring scheme?

This discussion is super-valuable as we will be doing more switches – some as player projects, some as expert projects – in coming months. We want the default score system to be reasonable… and even though we worked on it hard last year, this massive increase in throughput is making us revisit everything. Super cool.

Best,
Rhiju

2

Hi Rhiju!

Thx for your explanation. I like your hybrid score idea. All I want is to be able to see if a design has the whole package. The problem with the current score, is that them having similar score could lure us into think they are equally good.

Preferable I would like be able to sort after “true score” in the RNA lab results columns. It will help us much in our hunt for determining what is a good design and what is causing it to switch. It will be better for us if we know that the designs we are comparing is in about the same range.

Eli

3

I think there is a good reason the top three scoring designs all had an “unstable” hairpin. In fact, if I had to choose, i would pick those designs as the winners, simply because they, out of the many designs in the lab top notch, definitely did not have the ligand binding site form in the first state. I cannot say the same for many other designs in that lab.

Essentially, we could technically see that the 2nd state was not the preferred state of the RNA, simply because that 6-5 loop could not be stable with that highly unstable structural motif combined with an A-U closing base pair. Plus, as far as I can tell, many designs in the lab top notch did not score 94+ in any state, making it a bit of a waste to switch to a hybrid score system.

And, to restate the problem with that design once again, there was a highly unstable triloop/internal loop structure.

In addition, I feel it would be a better idea to score the RNA based on the shape it *did not* fold into. For example, if I am looking at state 1 of an RNA molecule, what is the score of state 2 with that same chemical footprint? If both state 1 and 2 score 94+ at the same time, it would seem rather counter productive to give it the full points, since there is a clear indication that the ON state may be present just as often as the OFF state.

Plus, most player created RNA switches are absolutely notorious for including highly unstable structural motifs, like A-U rich helices, triloops, lonely base pairs. Each of these will prove to be detrimental to any score for the first or second state of the RNA.

If I had to choose a different scoring system, I would go with the first state score being irrelevant, and the 2nd state score being 25%+ of the final score. This would stick with the fact that there can be a multitude of OFF states, but the ON state is the important part.

If we were to include a score for the first state, I believe that players should be able to define the first state and how they believe it would form, and compare the score against their personal prediction.

4

I added a section about the scoring system in the bottom part of this doc.

5

I agree that we should probably add some shape scoring to the switch labs. I’m torn between 50% switching + 25% shape1 + 25% shape2 (+target) and 50% switching + 50% shape2 (+target). As Brourd points out, the “off” shape is in some sense less critical. However if that is the case then perhaps the “switching” score should be isolated to particular bonds related to the “active” region of the switched molecule rather than relative to a particular “off” shape.

Better yet - how soon before the yeast-based “glow” testing is ready? Then the scoring can be based on relative change in brightness?

7

Love the idea of getting the switch score back, based on the intensity of green light. :slight_smile: Also agree with JL, that it is making sense if the bases in the region involved in the active switch area, are receiving higher value in the switch score, than far regions not directly involved.

8

I’m not entirely certain, but I believe the “glow” testing was for the theophylline riboswitch. On the other hand, I believe I remember Rhiju mentioning something similar about FMN, but I could not find the relevant post.

As for scoring based on the structures, that is still a very difficult thing to do. The great thing about the switch score, is its status as a constant. As long as the reactivity signal for a base shifts 0.1 in the correct direction, you get a point. Even then, it can be almost impossible to score 94+, simply because your design *has* to be perfect.

For instance, in the case of the lab Top Notch, only 10 bases are actually scored, and in order to get a 94 or higher, 9.4 or more bases must switch correctly. Not much room for error, eh?

I would personally prefer a system that lowers the score for a “winning” design before going and adding the “shape” based score in.

I also have some reservations about the shape based scoring anyway.

The multiple states of the RNA can all be detected at the same time, with SHAPE signals indicating this is the case. How do we differentiate which structure was the most populous? Since unpaired bases are given quite a bit of leeway (I assume at least .25 below whatever the threshold between paired and unpaired is for a design), both structures could theoretically score 100 as long as all bases are in that “sweet signal spot.”

Would this clearly indicate which state the RNA was folding into?

In my mind, the perfect scoring system for an RNA switch would be defined like this:

  1. A switch score, where certain bases are given a higher weight in the scoring. Nucleotides in the “open loop”, multi branch loop strands, and hairpin loops, are far more likely to have *consistent* high reactivity signals, that clearly indicate if they are paired or unpaired. As such, the switch scores should be more focused on their identity, and whether or not they do or do not switch. The other bases should definitely be taken into account, but if they do not switch correctly or quite as much, their contribution to the switch score should be minimal.

  2. Lower the score for a winning switch design. While 94% of all bases correctly switching is a great goal for us to set, it is high in some instances, where the number of bases that switch are minimal and internal loop bases are likely to get in the way of an accurate reading. How about 90%? This would allow a little bit of room for error, but, still require a majority of the bases to switch.

  3. A basic target for the structure based scoring. In many instances, players will be unable to reach 94 with many player created switches. It is a simple observation that many of these structures are unstable on their own, and how SHAPE signals for the second state can exist at the same time as the first. A useful alternative is to lower this to a simple target where all designs/states have to score above a specific percentage, one example being 80%.

We can also combine all of this with jandersonlee’s idea, scoring the active parts of the molecule. Normally, the bases that “switch” are quite prevalent in a design, and determining their identity in both +/-FMN conditions should indicate how well the RNA is folding. If you combine this score with those nucleotides that the switching bases interact with, this would prove to be an excellent way to predict the structure the RNA folds into. However, this score, much like the switch score, will be highly focused, meaning any errors that occur can and will affect how well the RNA scores overall.

Great news is, eventually players will be able to score the bases in their own projects, so I assume we can all test out our scoring ideas and see how accurate our different ideas will be.

9

Thinking out loud—what’s important in a switch and why? The final shape, or the ligand binding that brings it about? I think this varies with the switch and there is no general rule. The original shape seems less important as we are trying to shift away from it for some reason or other but this might not be true. Is it a very specific shape to shape switch we are really looking for, or a final shape arrived at from some more nebulous beginning?

If there is no general rule on the shape-ligand question then you need two separate scoring systems for the two choices.