Thoughts on how to award lab coin for previous labs and future labs

#1

Rhiju asked me to do a sum up of recent chat discussion on how distribute past lab coin. I butchered the chatlog and pulled out what I count the main points.

LFP have been working on getting it to us. Here is what he shared:

‎Using the initial point values from a few months ago, this is how the lab coin pans out http://prntscr.com/jo8yfm The “base” number on the left is as of a few months ago. Then the number on the right is that number plus the lab coin. With updated numbers: ‎http://prntscr.com/jo96ul

We’re actually discussing either boosting the multiplier or rewarding for more than just the top solutions. ‎Ie, top 3 or top 5. ‎Per puzzle. ‎Any thoughts?

Jieux: I think we should just get what we were promised… nothing fancier… moving forward y’all can do whatever you like though :slight_smile: ‎

Jieux continued: ‎giving out coin for labs is more of a motivator than badges for large numbers of designs. ‎‎better reward good designs than large numbers… quality over quantity :slight_smile: ‎

I suggested getting more badges for celebrating training of switch solving abilities. (Mission Switch Badge)

Astromon proposed ‎multi level badge for lab winners too ‎as there are now only one badge, ‎no matter how many winners a player has.

I have noted Jieux’s finishing point: “but my summary for past labcoin is just give us what we were promised… nothing more, nothing less ;)”

Now please add your thoughts

1 Like
#2

Most competitions with multiple participants award gold, silver, bronze. I’d suggest the same for lab results with three point categories 1000, 500, and 200 (scaled up or down) for the top 3 scoring players designs. Only a players top design counts for each sub lab, so they cannot get gold and silver and bronze points. If more than one player scores 100, each player scoring 100 get 1000 lab coin. You must score at least 80 to be considered for lab coin. (Even an 80 point design in the early rounds of a lab can be helpful.) Applying this retroactively for past labs would be nice, but it should at least go forward to new labs. My $0.02.

#3

I think a mixture of both systems could be interesting.

  1. The top three “design scores” get a set number of points, I’m thinking something like 2500/1500/500.

  2. The player gets points for their highest scoring design. which would follow the formula of 500^(floor(highest score)/100). Scores of 100 would get 500 points, 90-99 ~268 points, 80-89 ~144 points, 70-79 ~77 points, and so on. This can also be scaled to give more or less points for higher and lower scores.

  3. The player then gets points for level of participation. This could be equal to 500 points * fraction of designs submitted/submission limit.

This covers the basics of providing a higher score for doing better, fully participating, and getting the highest scores, while providing a reasonable scaling system (i.e. points for scores being exponential, points for time invested in submissions being linear, fixed points for excellence in design).

#5

Also, additional badges is always cool. Thumbs up for that idea.

#6

Coming from a different angle – the nature of the array experiments is that in the absence of filtering, many top scoring designs are statistical flukes, typically because of low cluster counts. When analyzing lab results, one will typically filter out any designs below some threshold, but there is no “right” threshold  – there is no way do really distinguish mediocre designs that get a random statistical boost and truly good designs that just happened to get low cluster counts.

Although we could do the same filtering when choosing a player’s best design, I’m concerned that it will be confusing for most players when their coin reward doesn’t match up with their design that got the best score. 

I suggested awarding coin based on each player’s top 3 designs as a more transparent way of reducing the effect of statistical flukes.  It is equivalent to basing the coin on the average of the player’s top 3 scores, which is much more robust.  The question of how much coin to distribute for a puzzle can be decided independently. 

#7

Glad this is being discussed, and moving forward I think these are all reasonable and fun ideas and I’m happy to roll with whatever y’all decide.  Looking backwards I remember there being a small multiplier added to the player’s best score and best vote. Does this really add up to over $100k in some instances or is there some other measurement being used? Just to reiterate, I think we should get what we were promised (no more/no less).

@ jandersonlee, I think a gold/silver bronze excluded a lot of people from getting rewarded for work in the lab… the old system where all participants get a multiplier of their best score (but only one best score) lets us all know we will get something for our efforts (so even a 20 while not a “good design” still gives a player $200 for their efforts.

@ Brourd. Agreed, badges are cool

@ Omei. I don’t know a lot about clusters but whatever matters in the lab should be taken into account… so some kind of formula that takes the desired attributes into account should be devised.

Thank you to LFP6 for taking the time to investigate this institutional oversight. Would also be nice to get rid of that long “waiting” list in our profiles if scores have been assigned to those designs. feels like I probably have 1000 of those!

#8

@Jieux Re moving forward, I think you are spot on in expecting all the Cloud Labs that still haven’t been haven’t been processed after all these years to be awarded “what we were promised (no more/no less).”

On the other hand, I feel coin awards for the array experiments should be discussed and decided separately, because they changed the nature of lab participation so dramatically.  Instead of being invited to submit three designs per puzzle, the invitation has been in the tens to hundreds.  Along with the fact that switches are significantly harder to design than single state puzzles, the level of effort required to fully participate in labs has gone up tremendously.

I don’t have a specific suggestion for what the awards for the array experiments should be yet – just that they ought to be considered as a category distinct from the Cloud Labs. 

#10

Per the “waiting” status, I recently tripped across a discussion of this happening a while back, and it having to do with correcting the status of those labs. Would you be able to provide a list (or at least a couple examples) of problematic labs?