The current lab round, released early this week, has gotten off with an enormous bang! As I write this, there are already 2288 submissions, which is almost 2/3 of the allocated slots. Yet there are still over 5 weeks before the scheduled cutoff date, and many dedicated players have not even gotten started yet. It looks like this will be the first round since Johan introduced his lab-on-a-chip experiments that players will once again be competing for synthesis slots.
This is tremendously exciting for the future of Eterna. But it does raise the question of how we should best allocate synthesis slots for this round. We discussed various aspects in the dev meeting today, but so many interrelated issues were raised that we decided to get input from more players before making any decisions.
Let me start with some background information:
The puzzles in this round are intended to gather data for one or more scientific papers. So the scientists have been very actively involved in creating new puzzles they would like to get data on.
The plan has been to release a second project this week that has 16 new puzzles that use an RNA reporter as output, instead of the MS2 aptamer used in the currently active project.
7200 synthesis slots have been allocated for player submissions for this round. This is fewer than typical because more have been reserved for designs generated by research bots.
The existing puzzles were posted with a limit of 50 submissions per player. This morning, that was reduced to 30. But some players have already submitted more than that.
The issues that were raised and discussed in the dev meeting included
Is 28 puzzles in one round spreading players’ efforts too thin?
Is 6 weeks too long for a round, given the current submission rate?
Assuming that there will be more submissions than available synthesis slots, how should they be allocated? (For the benefit of newer players, the techniques we used to use were user voting combined with brute force trimming as needed. In the early days, when no more than 10-20 designs could be synthesized for each puzzle, each player could vote for up to three designs, and the designs that got the most votes were the ones synthesized. As the number of designs per puzzle grew, there were more more synthesis slots than designs getting even a single vote, and an algorithm would be applied after the round ended to decided the maximum number of entries allowed per player, and truncate the submission list for those players who had submitted more than that.) If we restore voting as the means for selection, does it need tweaking, given the huge number of designs we have now? Would it help to restore incentives, in the form of points, for voting? If we do that, should we restore points for submissions, as well? And if we do that, should we retroactively award points for all the labs that haven’t been rewarded for? And if we want to make any changes, do we have the developer resources to actually make it happen?
There were probably even more ideas thrown out, but this should be enough to convey the scope of discussion. The objective at hand is to decide how we can best optimize players’ efforts for this round, given the constraints on developer resources and experimental requirements.
I’d say voting on thousands of solutions is unreasonable. I’d love to see bundles implemented, so there is some way to present specific research goals, though I do think having a couple guaranteed slots for individuals would be really beneficial (ie, for new players). The ELO rating system Eterna used at one point would be cool to help with getting players to vote and being more effective (especially with point incentive), but there is the issue of how to teach players which ones to pick (many new players probably don’t have any idea which ones might work over others, and even then there can be benefit to failing designs, as has been brought up in the discussion of that when it was brought up).
Providing points for lab submissions is something I think many have always wanted to see come back (both retroactively and for future puzzles I’d think). Of course, I think there should be larger measures for revamping points, but I’ve gone into that at length before (though my most recent thoughts are only on the player-led development Slack, not on the forum, should probably post them at some point). Points for voting is also a good idea if you want to get people to vote.
So that explains my sudden limit to 30-31 submissions. Rats, I was just getting going.
I often thought it was first come first served. But that doesn’t allow for “better players” to take their time if “lesser players” are eager and early.
Perhaps and algorithm to rank the utility of each players total submissions probability of being of highest quality and then pro-rating the slots on that basis? Someone who very often submits 95% or better scores may have more general utility than me and so might warrant more slots whenever they are scarce.
I think that an important question is what is the time limiting factor in this process. I can think of several candidates
1: Lab equipment time: Experiments can only be performed at a given rate
2: Academic time. The purpose is to provide data for a scientific paper, and theres a limit to how quickly such papers can be written
3: Administration time. Keeping Eterna going involves quite a bit of work, and it would be hard to do this any faster.
4: Player time: Players need 6 weeks to provide the best entries.
I don’t really understand how the voting idea is supposed to work. We don’t know what will and won’t be a good design, right? I mean if there is a metric to determine a better design, then it should be implemented in the game itself. Or is there a metric used, but already brute-forced by research bots and so players are only supposed to come with innovative designs that are not covered in that metric?
I don’t see a point in accepting new designs when you reach the maximum number of submissions you can handle (If there is no way to auto evaluate and sort them). Just close the project or redefine it. In my opinion there should be an extremely hard lab project that is always opened when there is no other project to do… So that players can have fun solving it and there isn’t a two months’ gap with constant countdowns.
Astromon, last night when I took a tally of how many designs you had submitted, you were in no danger at all of not having them synthesized. The few players who be most significantly affected are those that had already submitted 50 designs for a single puzzle, before the quota was lowered to 30.
But even submitting 30 designs for a single puzzle is no guarantee that they all will be synthesized. It all depends on how many others are submitted for that puzzle. We’ve been living in an atypical error, where synthesis slots have gone begging. But in general, and certainly for this round, it’s a good idea for a player to spread her/his effort somewhat evenly among the puzzles. That way, you’ll have few, if any, of yours not synthesized.
Yes, these are all very relevant. The first three are particularly relevant, but beyond players’ control. It’s really the last one that we as players can have control over.
I think we can take it as a given that the six weeks is when we need to have all designs in hand for the next synthesis round. And you are probably right that most players would prefer just to leave all the projects open until then. But, for example, we could instead hold off releasing the new project and do the projects sequentially. The advantage would be that players could be working more concentratedly on a smaller set of puzzles at any one time. We’re always looking for ways to facilitate collaboration between players, and that might have a positive impact in that regard.
Would it be possible to split this single round of experimental targets into two individual sequencing runs. It may also be useful to end this first round in about two weeks to account for the additional time necessary to run the experiments and analyze the sequencing data. This would allow for a greater number of sequences per player in each round, while maintaining a schedule similar to the current plan.
I don’t really understand how the voting idea is supposed to work.
The idea is that players decide what they want to be synthesized. The issue here is the fact that there’s so many solutions, and as you say - we don’t necessarily know what works well. TO be more accurate though, more advanced players have some concepts of what works better than other things that isn’t integrated into the game (some of which is in EternaBot, and some of which has not been quantified due to the lack of developer resources), but newer players don’t have that knowledge, and even failing designs can be useful (it’s the idea that you can learn just as much from what doesn’t work as what does work, and one might even design something particularly to see how it will fail).
I don’t see a point in accepting new designs when you reach the maximum number of submissions you can handle (If there is no way to auto evaluate and sort them).
The point is that at that point, it’s now a first-come, first served race for synthesis slots. As Omei mentioned in the topic, it is beneficial to give particularly experienced players time to work on their submissions, as some may be testing out specific hypothesis, or are fine-tuning.
In my opinion there should be an extremely hard lab project that is always opened when there is no other project to do… So that players can have fun solving it and there isn’t a two months’ gap with constant countdowns.
I’ve actually voiced similar opinions before, though for different reasons. I would like to see different levels of labs so that new players can get involved more quickly (looking in the opposite direction here). I do agree with that point as well though, and I might even extend it to simply say “there should always be something to work on”.
Okay I was speaking in a more general way and many times I have my slots filled so what we are discussing certainly pertains to me. My question is will all 30 slots (or how ever many allowed for any lab) be tested? And anything after that would be subject to vote? Also I would like to add that i like Lfp6’s idea for a hard continual lab that would keep players engaged more. Thanks!
Voting actually played two separate rolls in the “old days”. The obvious one is that it decided whose designs got synthesized. The less obvious one is that it provided a second way for people to participate in the lab, one that earned points. (Points were awarded in proportion to the Eterna scores for voted-for designs.)
Personally, with the huge number of designs involved, I don’t see much promise for voting as a way of choosing designs to be synthesized. However, I have wondered whether voting might be re-imagined as a game facet in its own right. That is, turn it into a challenge for players to figure out how to recognize a good switch. Appropriately incentivized and (eventually, if not immediately) supported by a game UI that served to build excitement, I think it could be a win-win-win scenario.
I guess it will depend on how many players are designing for how many slots are allowed. So i suggest players with winning designs in previous rounds automatically gets a certain number of guaranteed slots. Thanks!
Having a faster turnaround time on returning results to players would certainly be nice! But I think decreasing the time between synthesis rounds probably isn’t realistic right now. A lot of effort and expenses go into each experimental round, and the effort is (to a large extent) independent of how many designs are being processed.
But really, @johan or @rhiju would be better a better source than I for addressing this.
Yup, I completely agree. I would love to see pure analysis incentivized as well - thinking of Eli in particular, promoting this kind of generation of ideas and knowledge, sharing them, and then actually using them by players and the Eterna infrastructure, be it with the strategy market or hypotheses in future labs. The other part of this is that there needs to be a game UI to support it, partially for the facilitation of this kind of workflow, partially to reduce the learning curve, partially to make it more fun (see: gamification).
@astromon In the recent past, the dev team has tried to set the individual quotas so that 1) all submissions got synthesized, and 2) we made good use of the synthesis capacity. (There isn’t a hard limit on how distinct designs can be measured, but the more designs synthesized, the less precise the data is on each design.) But there’s no way to know ahead of time how many players will be participating, and how many designs they will feel like creating. So the admins for different rounds have taken different approaches. For the one lab where Ely and I were the admins, we purposely set the initial quotas low, and raised them gradually on a puzzle-by-puzzle basis as players asked for them. But that took more active attention than making an informed guess at the beginning and living with the consequences.
In this particular round, there were originally fewer puzzles planned, and the informed guess for the quota was going to be 100. But as more puzzles got added, the guess was lowered first to 50 and then to 30. Unfortunately, there was ambiguity about who was going to make the last change, and nobody noticed right away that it been left at 50.
For the few times (in the recent past) that there were just too many submitted designs to get good data on all of them, the selection algorithm was to lower the puzzle quota (after the round was closed) as much as it needed to be in order to “fit” the available slots. Any design that had at least one vote would not be cut. All other cuts were designs by players who had exceeded the ex-post-facto quota for that puzzle.
We don’t have to follow this same algorithm going forward; that’s (one aspect of) what this discussion is about. But in discussing it, we probably do need to distinguish between what we can do in the time frame for this round and where we would like to develop for the future.
" All other cuts were designs by players who had exceeded the ex-post-facto quota for that puzzle" i like this algorithm;
Also i lke your thinking on making voting a game in itself and would help new players and others learn what does make a design look good. Thanks for your answers.
