How can labs be improved to better support player research?

Feedback Feature Suggestions
#1

Currently, the rewards for players submitting lab designs can only be based on how well a design’s secondary structure (to the extent it is predicted by SHAPE values) matches a target structure chosen by the lab designer. But player scientists have been asking questions about RNA folding that are difficult to fit into that paradigm. An obvious example would be Nando’s labs that challenge players to create a pseudoknot. And yet, any design that scored 100% would be guaranteed not to be a pseudoknot, regardless of what he chose as a target structure. This is an extreme case, but there have been many labs where players need to choose whether to design (and vote) to maximize their score or maximize the contribution to the stated research goal.

So my question for all lab participants is "What changes could the devs make – in the way player/designers create projects, how other players create their individual designs, or how designs are scored – to better align the scoring with the potential for making a contribution to science?

To get the discussion started, here are a few of possibilities that Rhiju and I came up with:

* Allow the lab designer to designate key base positions that are particularly relevant to the research objective. When a player is designing their submission, those positions might glow, and hovering over that position could pop up some text about how the choice of base assignment at that position could address the research objective.

* Allow the lab designer more options for how designs for their project will be selected for synthesis. For example, one of these options might be to allow the lab designer to vote for an unlimited number of designs on their own lab. This way, other users would still have the biggest say in which designs got chosen for synthesis, but the designer could give a boost to any designs s/he felt actually addressed the research objective.

* Allow the designer to specify, as part of their project proposal, what percentage of the design’s score will be determined by the normal scoring, and what percentage will be decided by the designer, based on his/her judgement about how well the design addressed the research objective.

But what are your ideas? This is something Rhiju wants to address, but so far it hasn’t been a high priority. If we, as players, can come to some consensus that is workable, that would bump the priority for the devs.

#2

From what I understand, some of these features are already in development in some form. For example, I believe lab admins shall have the ability to pick specific designs, which will then be synthesized.

In addition to that, lab admins will also be able to choose how their labs are scored, by scripting the conditions for scoring, although, this will probably be specifically based on SHAPE reactivities instead of lab admin judgement. We’ll see to what extent we can script the scoring conditions when they release that feature in the near future.

Now, the designation of key base positions could be interesting. They already have a “Display Auxillary Information about RNAs” optional setting. I could potentially see an option to have a text popup be attributed to that, if the person proposing a project wanted it.

#3

Rhiju asked me to start this discussion, because this was currently a low priority for the devs.

Do you like the features you described in the first two paragraphs? Do you think others might be more important?

And addressed to the devs: Are there some changes actually close to release?

#4

On the scoring side, something should be added, yes. This said, I wonder if going all the way to resorting to scripting, is really necessary. I would have imagined that giving lab creators the possibility to describe simply (3 different characters presented as a chain is all it takes) where they expect bases to be paired or unpaired, and where it doesn’t matter, would have been enough for most purposes.

In terms of selection, I also see a few additional possibilities:

  • pure admin selection, with no voting participation of any kind from other players (but how to guarantee the lab admin is gonna do his job on time?)
  • users vote + admin veto (so that the lab creator can block candidates s/he judges uninteresting, or rule-breaking, while still having the crowd deciding which candidates are the best). Btw, this seems to me a better option than the proposed unlimited votes for the lab admin.
#5

On the scoring issue, I also question how well scripted scoring would work in practice. For one, there are many players who would like to submit research labs, but only a few players who have 'fessed up to being able to script. And my gut feel says that accurately reducing a research objective to an algorithmic score would, in general, not be a simple task. Even deciding whether two bases have paired can be a huge judgement call unless the molecule as a whole is tightly constrained. Take any mid-to-low scoring lab submission and see what the Eterna algorithm estimates as the actual folding. I certainly don’t put much weight on its results.

But even for a project where you could accurately describe the objective by specifying the target SHAPE values at a few positions (say create an “isolated yellow” at a certain position), would one really want to give 100 points to someone who succeeded and 0 points to someone who didn’t? If there were two designs that differed only in one base position, and one created the isolated yellow and the other didn’t, I would say the latter was almost as valuable as the former, because together they highlight a key variation in the formation or non-formation of the yellow. Certainly both are more valuable than a completely dissimilar sequence that has the same SHAPE scores in the region of interest.

#7

_In terms of selection, I also see a few additional possibilities:

  • pure admin selection, with no voting participation of any kind from other players (but how to guarantee the lab admin is gonna do his job on time?)_

I would say that if the admin doesn’t do his job on time, his lab doesn’t get synthesized. People who put work into the designs would probably be disappointed, and less likely to vote for any more of that admin’s proposals. Seems like it should be a self-correcting problem.

- users vote + admin veto (so that the lab creator can block candidates s/he judges uninteresting, or rule-breaking, while still having the crowd deciding which candidates are the best). Btw, this seems to me a better option than the proposed unlimited votes for the lab admin.

In case it wasn’t clear, the intent of the proposal was to allow voting (once) for an unlimited number of designs, not to allow an unlimited number of votes. This gives the admin less power than a veto. It also has the social dynamics advantage of being a positive action, rather than a rejection.

#9

Very good points, and thanks for the clarification.

This said, as a lab admin, I’d like to be able to forcibly exclude these beginner mistakes I still see too often. Xmas trees and cub scouts just don’t interest me, and if anyone wants to keep trying them, I’d suggest they create their own labs. And as I mentioned, there could be cases where certain rules cannot be enforced (like in my recent lab proposal, where I’d like designers to place a chain of at least 4 keto-bases in a specific 7 bases long segment) by the server or the client. I know I can count on most players to do the right thing, but some may not care, and I’d rather be unpopular than see synth slots used for a different goal than the one I set for my lab.

#11

Perhaps we could get the effect you want, without the connotation of rejection, by having a “voter advised” selection category as an alternative to the current “voter selected” one. This category, which would be prominently displayed in the Proposed Labs description, would keep the current voting (and rewards for voting), but entrust the final decision of which sequences to be synthesized to the project administrator’s judgement.

If we did this, we should make the admin’s decisions transparent. Currently, once a sequence has been selected for synthesis, the number of votes it received isn’t accessible from the normal UI. I think it would be important for “voter advised” labs that players can see how the admin used their discretion, so they can take that into account when deciding to vote for, or participate in, future labs by that admin.

Also, in order to propose a “voter advised” lab, a player might have to first earn a “lab administrator” badge by having had one or two of their labs go through the synthesis cycle.

#13

What is the “Display Auxiliary Information about RNAs” option setting? I looked at the current lab submission form, and don’t see it there.

#15

In the flash interface, under game options, there is an option to “Display Auxillary Information about RNAs.” The only use for this so far has been the theophylline hammerhead ribozyme lab, where it indicated the position at which backbone of the RNA would be cleaved. My suggestion would be to somehow incorporate this into the project proposals.

#17

I’d forgotten all about that. So the devs do have some experience implementing lab-specific annotations.

Could you take one of your research labs and mock up an example of how you would make use of some kind of annotation capability to help convey the research objective?

#18

This may not work for all situations, but I’m trying out an idea for aligning lab scoring with the research goals of a project, without having to make any changes in how labs are chosen or scored. See Impossible Folding I.

If nothing else, it will be interesting to see how players respond.

#19

* Allow the designer to specify, as part of their project proposal, what percentage of the design’s score will be determined by the normal scoring, and what percentage will be decided by the designer, based on his/her judgement about how well the design addressed the research objective.

Here’s a variation that I like better.

Allow the lab designer to specify, as part of their project proposal, the maximum number of bonus points they can add to the standard score (as determined from the target structure and the SHAPE scores). Lab designers would have to explain, as clearly as possible, how they will award bonus points.

Now that I write it down, I see two sub-variations.

  1. Designers award bonus points after synthesis candidates have been chosen, but without waiting for the synthesis results. In this case, they would be awarding the bonus points on the basis of how well the design addresses the research objective. The designer would have to register the awards within, say, a two week period (i.e. an interval shorter than the earliest plausible time for the lab results to be produced.)

  2. Designers award bonus points after synthesis data is available. In this case, the bonus points could be awarded on the basis of how well the design met a challenge posed by the designer (e.g. “Make position 17 yellow while positions 16,18, 27,28 and 29 are all blue.” This would address a similar need as a customized scripted scoring algorithm, but be accessible to non-scripters. It would have the disadvantage, though, of delaying the final results for a synthesis round.