[Dev Chat Followup] Lab solution bundles, kickstarting collaboration, and making groups useful

[LFP6]

This topic is in reference to discussion from developer chat on July 7, 2015, containing a condensed version of the points raised, with the names of those who raised them, following. Read the original conversation in the dev chat log starting here: http://eternawiki.org/wiki/index.php5/2015.07.31_Dev_Chat#bundlingchat

Here is a condensed (it is still a lot) version of the (really confusing) conversation explaining the idea(s), and some suggestions for how it could be implemented.

• [Nando] This is a general concept for how we work with labs. Groups would submit and voting on designs. It would reactivate the feature of groups, and promote collaboration and discussions. “I noticed that I submitted my 600 designs without ever discussing any of my ideas and hypotheses with anyone. Later I thought, wait a minute, that’s wrong somewhere”
• [Rhiju] The additional piece to this proposal is to eliminate difficulty from designing and voting at one time, through a 3 phase plan, over 4 weeks. Weeks 1-2 is designing bundles, week 3 is voting (bundles would be frozen, players could, in addition to voting, COMMENT on each bundle), and week 4 the top-ranked bundles can be adjusted based on feedback they have received. “This is pretty closely analogous to how expert science is done, with papers/grants submitted; reviewed by peers; and there is a chance for revision. And through the bundle explanations, and comments from players, we’d record quite a bit of our scientific discussion and hypothesis generation.” [jandersonlee]“Two weeks to get O(8000) designs is a tall order, even with teams” [LFP6]"[According to an old GetSat post I found], separate voting used to happen, but people didn’t like it?"
• [General discussion] Groups would create Bundles of solutions, and users would vote on these complete bundles (as opposed to individual solutions), however individual players should be guaranteed at least one submission [Note: I’m assuming that this is speaking about individual submissions here, not in relation to groups].
• [Brourd] Currently, you can join as many groups as you want
• [General consensus] Groups currently are completely underused and basically have no point. [Brourd] Will adding ‘bundles’ really add any collaboration? Wouldn’t this already be happening if it were to happen?
• [Nando]“All it has to be is an incentive for collaborations and sharing of ideas about designing good sequences for the goal at hand”
• [Machinelves]“Groups in foldit seemed to inspire competition and protection of data & info rather than sharing; however within groups and per person, the cookbooks library of strategies helped to share ideas and methods for solving” [Rhiju] “I think all solutions would still be public, but can you think of other ways to incentivize openness and a healthy community?”
• [Rhiju]“The bundles concept is an attempt to solve a few problems. One is that we have so many synthesis slots that it is hard to simply understand the data as a whole. Second issue is that we are not promoting discussion of sequences. The spirit that we had in the early days of eterna wheen there were few synthesis slots and lots of people were incentivized to look at all solutions and all lab data”
• [jandersonlee] 80 bundles is easier to discuss than 8000 submissions [Rhiju]“If each is allowed to present a 140-character theme/summary prepared by the team [it will also help]”
• [Machinelves] What about getting points or badges for applying methods from the strategy market? This would incentivize sharing of methods, and testing of those methods independently, and also reward interaction with other players’ contributions. not a total solution but just an idea"
• [Machinelves/Rhiju] Users may explore various techniques to come up with their solutions, such as a set of iterations exploring changes to that specific sequence, or using a particular bot.
• [LFP6] Having group chats would be one way that would help make groups more useful, where each group has a separate chat channel (the chat interface should probably use tabs).
• [LFP6] Bundles and making groups useful should be treated as completely separate problems. As a starting point, bundles should be available for layers to use, so that when creating a group of related designs (ie testing out a particular hypothesis), they can all be grouped together. In addition, to promote usage of groups, groups should be given the abilities that a user would have (as is already done in some groups which have created a user for themselves). Groups could submit bundles and individual sequences, and submit player puzzles [on prompting from jandersonlee] and labs as well.

Please reply with any additional thoughts to add (or if there’s anything else I can do to clean this up, as I am able to edit my own post with my GetSat permissions). Thanks!

[Brourd]

By the way. The idea of requiring some form of coherent hypotheses for submissions is an excellent idea, and I support the idea of design bundles.

However, I’m going to elaborate further upon my own point in this discussion:

There is literally nothing that stops players from collaboration as of right now. The addition of these tools and features will probably not foster these either.

In fact, the only current motivation I could see with groups is giving players direction in their design efforts. This is most likely going to be a good thing, considering most players feel lost in the labs. This should not be the incentive that players have for this though, since it would be simple enough to incorporate these into the actual labs, and require players to fulfill the objectives in order to make submissions.

Counter argument 1: But groups will have their own dedicated account for submissions, which should help to make this possible.

I believe there is a disconnect between expectations and reality. Who is going to submit these designs? Is it going to be a single person doing all that clicking?

Let’s say you contribute your designs to some sort of pool, and other players can give that design a thumbs up for fulfilling the hypothesis of the bundle, bringing it to the top of the pool and viable for submission in the labs. The administrator for that group then makes the necessary clicks to make it all possible. That is literally all this would be.

A dedicated account for group submissions does not mean group play. And does not guarantee collaboration. It simply puts an additional workload on some other player to be an administrator for the entire process.

Counter argument 2: The addition of group channels on the chat server will facilitate collaboration.

I don’t disagree with this, and I do agree that additional channels of communication can help to facilitate collaboration. However, there are several other methods by which communication may occur, so why haven’t we seen several players submitting designs with similar hypotheses? If we were to implement a more robust group feature into the game, are we to assume that an invisible demand for this collaboration is suddenly being fulfilled? It’s important that we take note of current attempts to share ideas, and determine what’s wrong with those mediums that a player group could fulfill and do better.

I’m too tired to think of the other arguments to counter that the addition of features will do all of this, but will offer my opinion on the technical aspects in a second post. Feel free to reply to what has been brought up here, as well as the next comment posted.

[Brourd]

Now I’ll offer my opinion on the more technical aspects of the matter.

First, I would like to discuss group size limitations.

If we maintain the current model of team building within the Eterna framework, any player could join a group and as many groups as they wish to join. This is not a method of collaboration that would best match our goals for promoting these player associations. It would be better to have player initiated lab projects with strict hypotheses and requirements for administration if that’s the case. Therefore, I believe that there should be a hard reset on the groups, and that we set a few limitations.

The initial thought that came to mind was a limit on size, with groups having a minimum number of players equal to a value of three or more, but less than or equal to 8 (arbitrary choices here), but this may isolate larger groups of individuals (such as classrooms, etc.), requiring some form of special registration or method for them to have all users register. However, if we allow groups with an unlimited number of registrants, our small active user base may all lump into a single supergroup. At which point, we’re right back where we started.

So, let’s approach this from the perspective of solving the problem, which is currently a lack of collaboration. This problem would be best solved through the creation of small groups, where players have the chance to form close associations and professional relationships with those players within their own group. The number of groups you could join is still up for debate, but I believe that a single group would work just fine. It prevents players from arbitrarily joining groups, and allows a player to focus on one thing at a time. 

This aspect of the feature’s design is something we need to determine early on, as it will define how we want to shape the group functionality.

Second, let’s talk about incentives.

In order for the groups to be relevant, the incentives to join a group and submit designs will need to be greater than doing it alone. In addition to that, it must be an experience that engages the player, making them want to submit designs for a hypothesis that the group developed. What these incentives should be is beyond me.

That’s all for now. Please feel free to respond with your thoughts on the subject.

[LFP6]

I think the idea is that having some structure to groups would encourage their usage, personally that makes sense. I think (from my personal observation/experience) people WANT to work together. But they don’t know how,. People want to use groups. But they don’t know what to use them for.

The features encourage discussion and collaborative design, just by the fact there’s a button that can be clicked to do it. I think currently no one has just thought seriously about doing it, as there’s no prompting to do so in a game which has kinda become personal, if you will. This is where it helps to have features for groups, it provides some sort of structure for what could be done.

Perhaps what we need for groups is also a facelift (bear with me, I’m going out on a limb here). Maybe the current visual appearence and structure simply don’t lpromote collaboration, even if the features were there. Because it would still look like a bunch of random comment streams you could join. There needs to be some sort of visual cue to show you, at a basic level, what you can be doing, encouraging group work on analysis and design. I’ll actually start thinking about something like this, and how it might look… I’m thinking of something that clearly says ‘professional lab work happens here’.

[LFP6]

Personally, I don’t think group size is the real issue here, nor how many groups you can join. I think that if an appropriate structure is in place, different groups should form for different reasons. Just because they already reached arbitrary size limit or you’re already working with a group focusing on something else, doesn’t mean your contribution would be less useful. Jensen my comment above about an interface that promotes the feeling of professional science, analysis, etc., kinda forcing you to think about what you’re doing and to do it for a reason.

However, you do have a point. I think in both cases, being a member of many groups or a group having many members, it can get unwieldy. I’d think that the only way to find limits is simply trial and error, not sure if you could theorize that.

There are two other issues that make this difficult though. The first (speaking for Elves here) is elitism. If all of the top 8 players join one group, 9-16 the next, all the way to the top 50 or 100, that becomes an issue for newer players. It’s both a barrier for entry and reduces the community overall as a whole, necessarily breaking then off into small chunks that aren’t encouraged to work together, if anything possibly discouraged. This also means the information/analysis/hypothesis that they never share will forever only be with that small group, whereas it could be useful to others who join in.

The second is inactive members. I can just for see either people going active and filling group space, with admins not wanting to remove, or a rule in place where they’re removed after some time, then they come back to find they can’t join because their spot has been filled. This is more of a small thing, but I’ll point it out.

I like the idea of a minimum group size, to make sure the group will be a group. Maybe you can also be listed in the group (and maybe get a badge?) as a “co-founder”. Something else though, I’m not sure if you need a certain number of points to make a group now, but if not you should, and how should this be calculated? X per co-founder? X combined? X just for whoever becomes admin?

Agreed on incentives. That makes a lot of sense, and probably needs to happen to make this really effective for the broader user group. It’s an interesting situation though of who gets the reward? All group members? Just contributors? How would you accurately measure contribution?

[LFP6]

Would also like to point out that it is still potentially useful to use groups for other things besides analysis as well.

I propose renaming groups to “working groups”. We can discuss what these could be used for (ie analysis, design, scripting, working on a page update for the site or suggested future eterna path *cough*eternacon*cough*, school class, virtual class for players, etc), and then all groups would be given the necessary tools to do all of these tasks, based on the purpose of the group/what they want to do.

[syang]

I like that you pointed this out LFP.  At EteRNAcon, a subject was brought up about having different “paths” that people could follow if they were into labs, scripting, etc.  I think groups would help accomplish this if EteRNA “specialization” is what will happen in the future.

[Brourd]

I do agree that elitism is very much a possibility with this system. And any other system where the total number of members is limited in some way. In addition, let us not fail to consider the elitism of Foldit groups, which have unlimited members with admin privilege to deny or allow entry. Or any groups where all members are not quasi-randomly assigned.

Maybe we should allow anybody to join any group they want? Until the first incident where a player abuses this system, and everybody is up in arms for the inability to prevent these players from joining their group. This is bound to occur when Eterna gets the overwhelming wave of players after our first successful RNA sensor.

But, let’s take a hypothetical look at several different group structures:

First, a group with 50 players. Now, in a perfect world, all 50 of these players are active, and all of them are submitting designs for the sequence bundle. However, the number of designs is limited to 10 sequences. Now tell me, is one of your sequences worth just as much as the player that submits 10 sequences? What if every other player submits 10 sequences? As you can see, it’s already unwieldy, considering there are now 491 submissions for players to peruse and determine if they match the hypothesis of the design bundle. Your voice of reason in a group of 50 is just as worthless as the voice of those 49 other players, and difficult to divvy up tasks and communicate with each other.

Second, a group of 25 players. Now sequence submissions are unlimited. You submit 20 sequences that are a perfect match to the hypothesis. Pristine, beautiful, shining examples of RNA design. And some other player submits 150 shoddy, barely viable, sequences. Whose designs are worth the most in this case? Your sequences are, but nobody will ever get to them as there is a mad rush to get as many sequences to fulfill the requirement. Here, we don’t want to limit the number of sequences submitted, as that will hinder creativity and place undue stress on players to try and be perfect.

Third, we have a group of 8 players, where design submissions are unlimited, and everybody has a perfect understanding of the hypothesis, and they all are master designers of RNA. And the deadline approaches, nobody has submitted anything yet. As admin, you start to get nervous, cheering your group on to make submissions, but they’re busy. One or two have real life obligations. Another handful are members of the other two groups, and find those hypotheses to be far more compelling than whatever this group is doing. Finally, one decides they no longer want to be a member of society and that person goes off to live as some hermit. Deadline is approaching, and being the admin, you make the decision to submit the 100 sequences on your own, with no input from your preoccupied group.

So what do each of these situations represent? A lack of teamwork, vision, engagement, and a sense of relationship for those players you are working with. Maybe that’s the confusion I’m having right now, in that I believe the best way to collaborate is to build a team, and not some circus of players who have all joined a group just to have the name and prestige involved with it.

It’s not impossible to build a team out of a group of unlimited players, but it may be difficult to coordinate and make sure that every player is involved and not just a member to be a member.

It’s not impossible to build a team with players who are in multiple groups, but there is always the potential that they just don’t care about whatever your hypothesis is as much as another hypothesis.

There is a lot of thought that should go into this. We should define what we want out of the group functionality, and what we want users to experience as members of a group, and the best way to implement the features that allow users to experience this vision.

[Brourd]

The creation of a function does not encourage the use of that function. If we have to start naming Eterna features that have been implemented with little to no usage, I’m sure a list could be created. A very long and detailed list.

On the other hand, the creation of a compelling experience built around a function that engages a player will encourage usage of said function. The perfect example of this in Eterna is the puzzlemaker. It’s a beautiful and simple tool that allows any player to design any puzzle within their imagination, only limited by a few constraints to make it easier for other players to see the vision behind the creator. If I had to guess, the number of players who make puzzles before they participate in the lab (if they participate at all) is far higher, even though the puzzlemaker is unlocked at 20,000 points, compared to the lab’s 10,000 point unlock.

This is what I mean when these features don’t define the experience, as the ability to collaborate already exists in forms outside of the game. Even though this will break engagement in some instances, typically these chances to collaborate would be an external enhancement to the experience that already exists.

As you stated, there needs to be a change within the group functionality as it exists, and not just a “facelift,” in my opinion.

[LFP6]

The ability to collaborate is current not guided or promoted. The feature would give that. That’s my personal feeling.

[LFP6]

My concern is limited groups restricting the availability of info by nessesity, rather than by how it’s handled (as in Foldit, which does not allow multiple groups because of a direct competition basis). The issue is of course quality, which is still a potential problem any way. And yes, could be hard to handle potentially. I think the answer is what you state in your last paragraph, finding what we really want out of this.

[rhiju]

Perhaps the idea of requiring bundles to be submitted by groups is overcomplicating things.
How about we forget the question of groups for a little (and maybe move that to a separate thread)?
What are thoughts on having bundles submittable by *individual players*? Everyone would get a freebie synthesis, but the bundles, each tagged with a hypothesis, would be voted upon to fill out the rest of the slots. Any player could submit several bundles.

[LFP6]

I’d like that. The browsing interface would need to be updated though, which would eliminate a good amount of how overwhelming it is. And really, this would really improve organization and communication!

[rhiju]

yes, a redesigned browser is key. 
this would be a good time to (1) collate other getsat threads where players proposed beautiful new mockups of the browser, and (2) maybe even mockup what a bundle viewer would look like.

[Brourd]

Which version of the browser do we want to redesign? Do we also want to take the time to enhance other functions, such as the addition of new columns for the chip riboswitch data (subscores, dissociation constants, etc.), or better sequence/secondary structure searching capabilities?

[Jieux]

Brourd asked me to comment so here are my comments:

Groups had a brief period of activity and all are now inactive to the best of my knowledge. I don’t see this changing and agree that the weight of a group will fall on one individual.

If bundling helps with analysis, I’m all for it. Perhaps a “bundle score” with additional coinage for the best bundles would provide the necessary incentive.  You could perhaps get three bundle votes in addition to your eight design votes.

Hope this is of some interest.

[Omei]

I think bundles are a great idea, for many different reasons.  Since there doesn’t seem to be anyone against them, I won’t try to list all the advantages I see.

My modest suggestion for increasing collaboration is to give the creator of a bundle the power to add additional players as collaborators for that bundle.  Essentially, they become an ad hoc team for the purpose of creating that bundle of designs.  I can image  a bundle team being put together for a variety of reasons, all of them good.  A particular scenario I think fits our community well might be a team of a few people, consisting of one person who has already been participating in the new lab designs, one that has experience in either the SHAPE labs or switch puzzle solving, and one or more who may have little experience but are eager to get involved in the new labs.

There are all kinds of variations in constraints (maximum collaborators in a bundle, minimum/maximum designs for a bundle, how incentives get shared, etc.) and UI features to support better collaboration.  But we can work those out as we go along. 

[LFP6]

I think the idea is that groups could (at least eventually) be used for this. Might be an interesting diea temporarily though?

[Omei]

I’m actually wary about forming long-term “teams”; it could easily stifle collaboration as well as encourage it.  The worst case would be where a player has to choose one and only one team to be a member of.  Then it becomes more of a competition between teams, which can easily slide into non-sharing of ideas between teams.

Also, long term, dedicated, teams don’t fit the characteristics of our community very well, I think. There may be a few players who are consistent in the amount of time they devote to Eterna each month.  But my own experience suggests that for most, it is much more variable than that.  Players have work, education, family, … activities that sometimes take precedence, and they’ll be active and inactive in bursts.  If there were a team of 6 players of mixed experience and the two most active players were both busy with other activites for a month or two, the team could very easily fall apart and leave the other four members “stranded”.

Of course, ad hoc collaboration happens all the time right now.  What I think we should really be looking at here is what, if anything, the developers might do to get more players involved in that collaboration.

So that’s my personal take on things, but it is shaped by 25+ years of experience on collaborative projects via the Internet.  (Yes, that predates the Web.)

[LFP6]

I think treating groups more like ‘working groups’ would be helpful in this regard. It would structure things in a way that is both more fluid, as well as focused more on trying to do something in particular, as opposed to just forming a team to compete.