Feedback thread for "Cloud Lab"

Feedback Feature Suggestions
1

Dear all,

As you all already know, we have launched the cloud lab early this week and we would love to gather feedbacks to improve it! I have listed several chat notes / messages we got so far - please add your own too!

  • Show number of votes : at least to the puzzle owner
  • Need a way to preview the puzzle in the game (when the puzzle is in the “Pending” state)
  • Separate “submission” and “voting” phase. For example, let people submit puzzles for 3 weeks and then have 1 week voting period.
  • Need a way to delete proposed (pending) puzzles
  • Need a way to edit puzzles
2

Scoring your own lab projects

If I was running a project, is there some way I can change the scoring? For example, instead of the current threshold of 0.5 for paired bases to score a full point, let’s say I instead want to make 0.5 = 0.5 points, 0.6 = 0.75 points, and anything 0.7 or greater = 1 full point. Or, perhaps for loop bases, I would like to leave hairpin, internal loops, and multiloop bases as is, but, change the scoring for loop bases that are a part of the dangling open loops to less than or equal to 0.5 = a full point. Essentially, I guess I am asking for a dynamic scoring system that allows players to change how the RNA is scored based on their own rules and observations. The “developer” score could be kept separate, for use with the strategy market and EteRNAbot. In addition, the requirements for how a base would score a full point should be posted on the lab’s main page.

1 Like
3

Editing Puzzles

In regards to the cloud lab, I think editing puzzles could be allowed, however, when the project structure or constraints have been edited, all active votes should be removed. This could work, if players who vote on a project automatically “follow” and receive updates about the project. When the structure is edited, all players who voted receive an automatic notification, and are asked if they would like to renew their vote for the project. This would be the easiest way to do so, instead of players voting for a project and later finding out the project has been changed from what they originally envisioned it to be. However, the structure can not be changed after the fact, unless the creator has a specified goal, and other players approve of the project changes.

1 Like
4

Dynamic and Unconstrained Lab Structures

This is something I could see more in the “future” of the player projects.

As of right now, players are limited as to what their project can be. More specifically, they are limited to a specific target structure, and for those players who are thinking “big” this will definitely limit their hopes and dreams. So, let’s go over a few proposals that could help this idea out.

Dynamic Lab structures

This is definitely something that could be interesting for players to get a hold of (as it is being developed for the Theophylline Ribozyme switch puzzles). Essentially, players could define a specific “structure” and players could fill in the rest of the 63 bases required for the design however they wish. This could allow a project that says “Insert this 2-2 Loop with an isolated base pair on a multiloop.” Instead of 10 separate projects, one with a 2-2 loop of a multiloop, one with a 2-2 loop off of the open loop, one with a 2-2 loop off of a hairpin loop, and so on and so forth. If this could prove too difficult to implement, then, the next idea may be better.

Multiple Defined Target Structures in a Single Project

So, let’s say a player like myself comes up with the project “A-U closing base pairs.” Ultimately, my goal with this project would be to define the upper limit for the chemical detection of A-U pairs when used as closing base pairs of a loop. However, there are hundreds of different circumstances I could ultimately test with this, but the number of different lab projects that would be needed to test these all would be enormous, taking up valuable space if they were all running at the same time. So, instead, when I propose a project, I will have the 30 different target structures already a part of the project itself, and the lab admins (or players who participate in the lab?) will ultimately choose which targets get synthesis slots that round, when a target is closed, etc. If this is too difficult to do, perhaps this next idea could work.

Lab Project Grouping and Categorization

So, one thing I noticed while going through the “past labs” is that there are now quite a large number. Maybe we can “group” all these individual labs into packets. For example, all the lab targets for the Player Projects could be categorized into a single project named “EteRNA Cloud: Player Projects - R69”. Once we click on the project, a new page opens that contains all of the lab targets, or,it could be broken down once again, with each player who submitted a project getting their own “folder” that contains all the labs they proposed. Ultimately, this could make it easier for players if all projects were categorized in a form like this, instead of going through everything trying to find a specific lab. This also gives players the chance to propose new labs that can be voted on for activity, which can be placed into the same project folder, making it easier to find specific data.

Unconstrained Lab targets

If I proposed a project that asked players to go to a database, submit and cite a sequence based on natural RNA sequences, they would most likely break any structural constraints that are there. However, if players were able to submit a sequence and structure, similar to how the puzzlemaker works, this would allow them to do so. This is somewhat like the dynamic lab structures idea, however, with this, none of the target structure would be predefined, it would just be free designing of RNA based on whatever the lab admin/proposer asks of the players for their project.

5

The ability for the lab admin to set the number of submissions/votes a player gets to use in a lab.

If I as a lab admin create a project that very few people participate in, but really wish to see my idea get synthesized, I would like to see the ability to change the number of submissions a player can make in the project. In addition, I would like the ability to change the total number of votes a player can ultimately use. We could set a minimum and maximum for this, to prevent abuse and players from getting the ability to vote for every possible design, we could set the vote limit to 1/4 of all slots when greater than 32, and the minimum could be set to equal the number of potential slots a player can get for a lab. In addition, the lab admin should be able to set rules in regards to how synthesis slots are allocated when the amount is low. For example, the admin could set it to where each active lab participant gets one slot based on votes, and if there are any left over, then goes down the list of participants until all slots are allocated.

6

Admin ability to run bots on the design

Administrators should be given the chance to allow NUPACK, EteRNAbot, Vienna, and any other algorithm the developers wish to include into the game, against their lab design. This could be activated by checking a box on the lab proposal page, and the slots the algorithms would use would be automatically deducted from the number asked for.

7

Increasing the number of points required to make a lab project.

This is just a suggestion to create another goal for players to reach for in the game of EteRNA. As of right now, the number of points required for players to propose a project is 20,000. Perhaps we can increase this total to 30,000, or maybe 50,000 points. This is more or less just something for players to shoot for as a goal, with another badge awarded once the point total is attained. There would be exceptions of course, specifically for classes, developers, and other research groups if they wish to propose a project.

8

Brourd, I was about to type in my own comments – but wow, you included everything on my list and more.

10

Well, if I had known that, I would not have stayed up so late coming up with ideas! :slight_smile:

There wasn’t much happening here, so I figured while I had some free time, I would post some thoughts I have had about improvement of the projects and how much control players will have over them.

12

By the way, I was wondering how much control players will have over the experimental conditions, such as the temperature the RNA folds at, the solution the RNA is folding in, or the concentration of the ligand for multiple state RNA?

13

Example of multiple target structures for proposals

As seen here, there are 3 structures (visualized using the switchmaker)

If I wanted to make a project called “Does increasing distance away from the open loop affect the chemical detection of RNA structural motifs” using all three of these structures and more, I would be required as of right now to make multiple lab proposals, which will then clutter up the active labs. It would definitely be easier if all these target structures were a predefined aspect of the cloud lab proposal itself.

15

This feature could also allow players to weigh the bases differently based on how they want the project scored.

For example, if we look at the lab “Cloud Lab 19 - Big Hairpin Loop by Eli Fisker” We will see that an overwhelming majority of the bases in the design are located in loops. Even if those 8 bases that make up the 4 base pairs of the main design are reactive to chemical probing, the design is guaranteed to score almost a 90 (assuming the barcode hairpin scores perfectly).

If players are given control of scoring, they could determine what the individual weight of any base is in regards to scoring. This would also allow players to focus success or failure on specific structural motifs, if that is the goal of their project.

Let’s say a project was to find successful triloop sequences and only 2 bases out of 79 were closing base pairs for a triloop. If those 2 bases do not match expected chemical reactivity as a closing base pair, the design would still score 97/100, which could indicate false success to some players. However, if I made it where those 2 bases were worth 40+% of the score, any reactivity in that closing base pair would seriously affect the score, and drive players to do better to fulfill the project specifications.

Ultimately, I think this would be a good idea, however, I would recommend that the scoring for any lab or project be placed on the main page of the project itself, whether it uses the current scheme or a custom version by the player.

1 Like
16

Adding to what Brourd said, I would have a couple projects which, at least for what I would call “preliminary rounds”, would require a lot of freedom with designing both sequences and structures, and even with scoring, which in some cases, would not even occur.

In the project I dubbed “CBPP”, I’d like to investigate the experimental data in a similar way that was done in this scientific paper, and hopefully come up a more accurate scoring function, thanks to multiple chemical mappings.

In another possible project, I’d like to investigate a weird idea of mine, related to RNA-RNA interactions, more specifically, the possibility of sequences interacting with themselves. For instance :

I’d be very curious to know how this sequence would fold. Some may recognize the hexaloops on the right side and even guess where I’m going with this. But the details of my hypothesis are not really important here. What really matters is the fact that, as an admin of that project, I’d like to first invite participants to come up with (maybe a dozen of) pairs of sequences+structures like the above one, which would have to match certain general criteria, but could not be scored, because the goal is not a precise target, it’s to discover the rules that may affect the folding in special cases.

17

Search and Sorting Options

I don’t believe I threw this idea out yet.

As we begin to add more and more projects, even if we categorize them all based on the round they were synthesized, or custom categorization from the player, we will reach the point where there will be hundreds of labs to go through. Some basic sort and filter functionality would be great for this. One example I can think of is a basic filter to separate single state and multiple state RNA.

Another great functionality that we could include is the ability to “tag” labs. If there is a tetraloop in the lab, we can tag it with that, which can allow easier browsing and searching for specific structural motifs. These tags could be chosen from a set list, with examples including tetraloops, multiloops, adjacent stacks, bulges, internal loops, symmetric internal loops, lonely base pairs, “short” helices, and so on and so forth.

The tags could be designated either by the developers, or, we could “crowdsource” that job to players, and allow them to go through the 200 previous labs in order to determine which labs have which features :slight_smile:

As of right now, it can be a bit unwieldy to go through all those labs and designs, and I think players would greatly appreciate it if a functionality like this was introduced. If any of these features are already in development, that knowledge is above my pay grade, and I am sure these ideas have probably already been bounced around.

Thanks!

19

experimental devs are a big fan of this idea, since similarly we have several projects in ‘exploratory’ phase where we don’t know the structure. examples are a project to find structured RNA domains in large viral or celular transcripts.
We are working with the site devs to make this option a reality.

21

For common structures like triloop, tetraloop, pentaloop, bulge1, bulge2, loop1-1, loop2-2, etcetera, it might even be possible to have them automatically tagged.

22

Another possibility might be to have a 100% scripted lab, where all sequences were generated by a lab manager’s script. I’m assuming this might be for some goal such as exhaustive testing of pairs in a 2-2 loop, or for testing closing/backing tetraloop closure, where it would be simple to generate them by script and tedious to do it by hand.

This could go several ways: one where all generated sequences are run (assuming the project is approved) or another where players vote for their favorite sequences and get credit for picking the winners.

23

Just wanted to add my list of wishes to an improved lab interface. It holds a long row of fine player suggestions.

Lab Interface Improvements

24

The ability to change the structure of the estimated fold

This comes as a request based on looking at the switch labs.

I think, the ability to edit the structure of the “estimated” fold based on a player input would be highly useful, especially with switch labs. I believe in some cases, a majority of the RNA population folded into the second state of a design, compared to the first, or vice versa. However, I currently have no way to edit the proposed structure of the estimated fold, in order to compare the chemical mapping footprints, and determine if I am correct.

It would also be interesting to score all structures of a target RNA with all experimental conditions. For example, if I have the SHAPE data for an RNA sequence without FMN, I would like to see the score for both the +/- FMN secondary structures. A kind of vague estimation as to which state the RNA favored folding into, which can help players point out sequences where the second state may have been the more favorable position without the ligand, and where the first state may have been favored with the ligand. I assume this would not be a perfect score, since there is quite a bit of leeway in how loop bases are scored, and many bases would share the same score, but I think it would be useful as a general metric for determining to what extent one state existed when it is not supposed to.

25

Hybrid Admin Selection and voting

This is related the idea of separate voting/submission phases.

First, I think whether or not there is a separate phase should be left up to the admin of the project, and the length of the phases as well. In addition, admins should be able to increase the timeline of their project, in case their project is something a bit complex that they feel players would not be able to achieve in a one month time frame.

Second, if I had a lab project, where the request was to use a certain idea or sequence while designing, I would like the ability to “weed out” the sequences that did not match my specifications, and only move those i felt were satisfactory, to the voting phase. This would be useful to players who do not want absolute control of which designs are ultimately synthesized, but do not want the only sequences synthesized to not embody the goal of their research (especially if there are limited slots).

Third, under active labs, I would like all labs to have a timer, and an indication as to which phase they are in. So, for example, a lab project could be in the voting phase, and there are 3 days left until that phase ends, and it is sent off to synthesis. Another example could be a lab project is in the design submission phase, there are 11 days left until voting phase begins. As a final example, if both design and voting are allowed at the same time, the project would have an indication that both are allowed at the same time, and that there are 29 days left until synthesis.

Fourth, automatic notification when a new lab project is moved to the active labs out of the proposed labs.

Finally, if any of these ideas were ever implemented, the default sort of the active labs should be based on time left, with those that have the least time moved to the top of the list, so that players will not miss them.