Daytalk for Newbies

[Toomai]

I am of the opinion that daytalk should not be shoehorned into the Matrix6 setup. If newbie games switch to using it, it should be with a setup designed for it.

[Toomai]

What makes it "shoehorned" rather than just "added to"? What about Matrix6's design makes it particularly incompatible with daytalk?

In general, I think daytalk is a major enough thing that you can't just add it to an existing setup and expect it to work well; it has to be part of the design from the get-go.

[Toomai]

I am of the opinion that daytalk should not be shoehorned into the Matrix6 setup. If newbie games switch to using it, it should be with a setup designed for it.

I'm not sure how seriously daytalk will impact newbie game win percentages; it's a fairly hard advantage to calculate and I think that it would be easier to see what sort of effect daytalk has on these games and then balance around it as opposed to the other way around.

If people do decide to implement it, I would be splitting the stats into "has daytalk" and "doesn't have daytalk" games, and after a few months whatever impact there is should become clear.

I wouldn't characterize a 52% scum win rate as struggling for balance.

The final raw number is pretty good, but it's not a stable balance. Two of the six subsetups just need 1 or 2 scum wins (without matching town wins) and they get pushed to the 40/60 zone.

[Toomai]

Alright, just over a month in, I think it's time for an update on the experiment so far.

There are currently 6 completed newbie games with daytalk, only 1 of which was a town win. While not a good sign, it's obviously not large enough a sample size to draw any conclusions yet.

The replace rates are more interesting though. In the previous 30 games without daytalk, 37.5% of all scum newbie slots (total of 32) saw at least one replacement. In the six daytalk games so far, involving 6 scum newbie slots, there has been 0 replacements. 6 is a pretty tiny sample, but a flat zero is very strong.

However. The "old" rate for town newbies was 35.8% (49/37), and the "new" rate for town newbies is 32.0% (8/25) - that is, basically unchanged. If adding daytalk means we keep scum newbies but town newbies act the same, that may result in a new meta where replacing out is a town tell.

[Toomai]

Okay so now we're ten completed games into the experiment. I'll update the stats topic itself shortly, but this seems important to note here first.

This is the first time in history (of newbie Matrix6 games) that scum has won 9 of the past 10 games (ordered by game number), and "scum wins in past 25 games by game number" is also the highest it has ever been (at a town winrate below 30%).

In addition, here are some interesting stats about replacements:

• The rate of replacing players for newbie scum in these daytalk games is 1/10 (10%). In the previous 30 no-daytalk games, it's 16/32 (50%).
  
• We already have more SE replacements in these 10 games (13/26) than in the previous 30 games (11/71); this is worse than the "all newbies" rate from the previous 30. No IC replacements yet though.
  
• Town newbie replacement rate is effectively unchanged.

Or, if you like pictures: (direct link)


Again, this isn't statistically significant yet, but the emerging patterns are quite strong. If the goal here was "stop newbscum from replacing", I think you have a winner. But I don't really think it's worth everything else that's happening.

[Toomai]

The immediate problem that I personally see is that "newbies replacing out is a decent scum tell" is now "newbies replacing out is a very strong town tell". It's better to have a meta where a major event is a town tell rather than a scum tell, because it's probably less damaging overall to accidentally confirm town than scum, but it's a much stronger case mathematically, potentially making it just as damaging anyway.

[Toomai]

The last few completed newbie games have had a crapload of replacements that have skewed the results quite a lot, so here's what's going on now, after 16 daytalk games (about half done the original 30-game experiment).

Comparing the previous 30 night-talk games with the current 16 day-talk games:

• Town winrate has gone from 14-16 (46.7%) to 2-14 (12.5%). [p=0.02034: significant]
  
• Town newbie replace rate (per slot) has gone from 49/137 (35.8%) to 26/69 (37.7%). [p=0.78716: not significant]
  
• Town newbie replace rate (per player) has gone from 58/137 (42.3%) to 30/69 (43.5%). [p=0.87288: not significant]
  
• Scum newbie replace rate (per slot) has gone from 12/32 (37.5%) to 5/17 (29.4%). [p=0.56868: not significant]
  
• Scum newbie replace rate (per player) has gone from 16/32 (50.0%) to 6/17 (35.3%). [p=0.32218: not significant]
  
• Subsequently, the change in overall newbie replace rate is not significant.
  
• SE replace rate is still through the roof (14 slots so far, compared to 11 slots total in the previous), so much so that it's pulling the overall stats quite hard:
  • Overall replace rate (per slot) has gone from 75/270 (27.8%) to 48/144 (33.3%). [p=0.23800: not significant]
    
  • Overall replace rate (per player) has gone from 88/270 (32.6%) to 58/144 (40.3%). [p=0.11876: not significant but getting close]

[Toomai]

Just eyeballing the conditional formatting of the spreadsheet, I'm seeing no correlation at all between replace rate and time of year. It's an interesting question though, and I'll see if it's possible for me to get a more objective answer.

[Toomai]

Just eyeballing the conditional formatting of the spreadsheet, I'm seeing no correlation at all between replace rate and time of year. It's an interesting question though, and I'll see if it's possible for me to get a more objective answer.

I managed to put something together for this:



Since I don't track the exact dates on which replacements happen (that would be almost impossible - do you call it on the replace in, on the replace out, the player's last post before vanishing, ???), I had to do something a bit cruder, but still seems to be legitimate. Here's the methodology used:

1. Determine which months each game was active for. If a game started on May 15th and ended on July 5th, it covers the months of May, June, and July.
   
2. Count up all the players replaced in each month. At the time of this post, May has 239, June has 272, and July has 302.
   
3. To try and remove the variable of "more games ran in this month", divide by the number of games that covered each month. There have been 58 May games, 70 June games, and 83 July games.
   
4. Therefore, the final values for May/June/July are 4.1, 3.9, and 3.6. The average replacement rate overall is ~3.5 players per game, so this seems legit enough - it should probably be read as relative rather than absolute anyway.

[Toomai]

edit (a way to work against that could be vengefuls or supersaints.)

But this leads to another potential factor: whether a role is appropriate for a newbie game, regardless of whether it's appropriate for fixing a particular issue.

Question, in these games, what percentage of the time is the IC town/scum? How often are the SE's town/scum?

This was the distribution of the two town wins:

• 1657: players: 5Nb-3SE-1IC; town: 3Nb-3SE-1IC; scum: 2Nb
  
• 1668: players: 5Nb-3SE-1IC; town: 4Nb-2SE-1IC; scum: 1Nb-1SE

Of note here is that 5Nb-3SE-1IC games tend to have more scum wins in general.

[Toomai]

For instance perhaps the daytalk has significantly increased the ability for the scum to quickhammer in endgame situations.

Well this is the relevant data if you want to discuss scum quickhammering for the win.

Spoiler: Definitions

• "town hammer": town player made the hammer vote to lose the game
  
• "2-scum hammer": both living scum voted in succession to hammer and win the game
  
• "1-scum hammer": one of the two living scum made the hammer vote to win the game, saying nothing about where their partner's vote is (it could be earlier on the wagon, or elsewhere entirely)
  
• "scum hammer": the only remaining scum made the hammer vote to win the game

• 1652: 3:1 MyLo, town hammer
  
• 1653: 2:1 LyLo, scum hammer
  
• 1654: 3:2 LyLo, town hammer
  
• 1655: 4:2 MyLo, town hammer followed by double unblocked nightkill
  
• 1656: 2:1 LyLo, town hammer
  
• 1657: town win
  
• 1658: 4:2 MyLo, 1-scum hammer followed by double unblocked nightkill
  
• 1659: 3:2 LyLo, 2-scum hammer ~11m apart (can't find PT)
  
• 1660: 3:2 LyLo, no lynch at deadline followed by unblocked nightkill (scum was the leading wagon)
  
• 1661: 2:1 LyLo, scum hammer
  
• 1662: 2:1 LyLo, town hammer
  
• 1663: 3:2 LyLo, town hammer (the two scum went first)
  
• 1664: 3:1 MyLo, town hammer
  
• 1665: 4:2 MyLo, 1-scum hammer
  
• 1666: 3:2 LyLo, 1-scum hammer
  
• 1667: 2:1 LyLo, scum hammer
  
• 1668: town win
  
• 1669: 3:2 LyLo, 1-scum hammer
  
• 1670:2:1 LyLo, scum hammer

And by "discuss scum quickhammering for the win", I apparently mean "toss that out as a potential factor because it happened maybe once in the 17 scum wins".

[Toomai]

Toomai, would it be at all helpful for me to run more games with daytalk, just to have more data? I would have done so until all thirty games ended, but I'm leery because the winrate is so bad. It might help tell us if replacement rates are significant, at any rate.

It might help, it might not. Well I mean more data is always better, but if gathering the data is becoming a problem you're going to have to stop earlier than you'd wish regardless.

Here's what we got playertype-wise so far with 11 games yet to be completed:

• Town newbies: 80, that's enough.
  
• Scum newbies: 22, need 8 more to reach 30, seems like a given.
  
• Town SEs: 40, that's enough.
  
• Scum SEs: 10. The previous 30 games had 19, I don't think we can make 30 with a reasonable amount of games.
  
• ICs: Gonna have 30, but obviously less than that per faction.

[Toomai]

In order to reduce scum's power slightly, why not make the guaranteed goon an encryptor?
This will give town *slightly* more setup information in the event of a goon flip.

In 2-scum games, where offing either scum means no talk anyways, the purpose of Encryptor is to say "You thought this game was night-talk only? Think again". In a world where every game is known to be daytalk, it's pointless.

[Toomai]

This is my understanding of the newbie setup "design rules":

The purpose of newbie games is to:

1. provide a semi-predictable, consistent environment for new players to learn how the game works on this site
   
2. provide a "trial period" for players to determine whether they like the game, saving other games of higher importance/commitment from replace-outs of "nah this ain't for me, bye"
   
3. provide a positive experience in an effort to get players to stick around for future games

To accomplish this, the newbie setup should:

1. consist of simple roles with simple interactions, preferably Normal ones
   
2. be reasonably balanced in town-vs-scum winrate, preferably per subsetup and not just overall
   
3. be low in power, to reduce swing and focus on day play (as the site meta tends to do)
   
4. be a legitimate setup that doesn't compromise on enjoyment (i.e. not something that's so dumbed down it's not fun to play)
   
5. have a chance of either having a Cop or not, if only to demonstrate how this site's meta of the most quintessential role in the game differs from other places (e.g. "actually scumhunt instead of assuming this is in the game")

Obviously point e) of the second list is the least important, but what about the relative importance of the other considerations?

[Toomai]

My opinion on the matter is that matrix6 isn't particularly well balanced and daytalk has reinvigorated the newbies into playing better, causing them to take better advantage of the scum-sided matrix6

There's an argument to be made that it doesn't matter what the theoretical balance is compared to the balance in practice (which without daytalk was about as even as could be hoped for).

[Toomai]

The current stats, comparing the last 40 nighttalk games to the current 29 completed daytalk games, look like this:

• Town wins are way down (21-19; 52.5% -> 6-23; 20.7%) [p=0.00758]
  
• Town newbie player replace rate is unchanged (76/179; 42.5% -> 56/126; 44.4%) [p=0.72786]
  
• Scum newbie player replace rate is a bit lower, but not significantly (24/48; 50.0% -> 12/33; 36.4%) [p=0.22628]
  
• Combined, newbie player replace rate is unchanged (100/227; 44.1% -> 68/159; 42.8%) [p=0.80258]
  
• Oddly enough, SE player replace rate is way up (15/93; 16.1% -> 29/73; 39.7%) [p=0.00062]
  
• IC player replace rate is also elevated, though not significantly (4/40; 10.0% -> 6/29; 20.7%) [p=0.2113]
  
• All told, player replace rate is up overall, though not by much (119/360; 33.1% -> 103/261; 39.5%) [p=0.101]

[Toomai]

I like the IC player. What if there was also a specificly scum private chat advisor?

Somebody who specificly talked to the scum during the night?

That means a third set of eyes to pick up on tells, crumbs, slips, interactions, etc. Which could be even more powerful than daytalk. Not to mention it amounts to adding a tenth player of a new unique fourth category in terms of the queue and replacements.

[Toomai]

Toomai, do you have updated stats on this? It seems as though town have been winning a significantly higher portion of games in the second half of the trial. I'm wondering if the experiment just started during a freak period.

I do but because only one or two games remain I'm holding them until it's all done. But yes, I have also seen this disturbing pattern ("disturbing" in the sense of "how are we supposed to make a decision from this").

[Toomai]

Alright so all 40 daytalk games are finally over. Here are the final results.
[image link, 1821x801 px]

When comparing the 40 daytalk games to the previous 40 nighttalk games:

• Did the addition of daytalk lower newbie replacements without affecting SEs/ICs?

  No.
  Scum newbies did replace out less often (50.0% -> 34.1%). However, town newbies replaced out more often (42.5% -> 50.3%), and SE/IC replacements were significantly elevated (SEs 16.1% -> 37.6%, ICs 10.0% -> 17.5%). And because there are more town newbies than scum newbies, newbie replace rate overall went up (44.1% -> 47.0%).
  
• Is the newbie setup still reasonably balanced with the addition of daytalk?

  No.
  For the first 20 games, "town winrate over the last 10 games" plummeted to 10% for the first time in Matrix6 history, and even dropped down to 0% once 1667 concluded. For this to occur at the same time the daytalk trial began cannot be a coincidence. However, despite this ridiculous evidence, it's not an open-and-shut case, as for the second half of the trial the winrate is exactly 50%. One possible explanation that looks to be quite clear in the linked image is that scum replacements are higher in the first half and town replacements are higher in the second half, but due to small sample sizes (especially for SEs/ICs) I'm not ready to say this is a good reason. A second potential reason is that scum had an advantage until town learned to counter it, but to be honest I haven't really seen anything that suggests this is the case, and the division is way too stark for a gradual improvement to exist.

If there's anything else you want to know, post here and I'll see if I can parse it out of the data.

[Toomai]

why can't it be a "coincidence"?

Six times in Matrix6 history, town's "last 10 games" winrate dipped to 20%, and only once did it stay there for more than one game (it was three). As soon as the daytalk trial began, it touched 20% once, went back up to 30% once, and then fell down to 20% or below for 15 consecutive games. I don't know what kind of math I'd need to back this up, but I find it exceptionally difficult to believe that this happened as a result of pure luck.

[Toomai]

I just can't get around the eyeball test right now to be honest. Something unprecedented happens at the same time as a change to the setup. Yet the fact that the second half of the experiment is pretty much indistinguishable from non-daytalk games is throwing a huge damper on the ability to make any sort of concrete determination here.

Perhaps I should go ask some statistics people what kind of math I need to figure out how much luck would be necessary to explain this result.

[Toomai]

The winrate itself is probably less than 50% and the fluctuations of each half of the experiment are the anomalies. That's why you get the sample size and average the win rate in the first place.

Well when you take the average, the daytalk town winrate is a not-really-acceptable 30%. But the reason I'm kind of making a big deal about the two different halves is because there's a possibility that scum took advantage of the power and then town learned to counter it, meaning that daytalk could be implemented with no long-term winrate effects.

Though that wouldn't really change the fact that replacements were still overall higher, which is kind of against the point in the first place.

[Toomai]

The experiment ran between Icfober and now, right? That's holiday season (1st half) and would raise overall replacement rate.

Did the town eeplace rate remain as high in the 2nd half?

First 40 games: start times between June 2015 and September 2015, end times between July 2015 and November 2015
Trial's 40 games: start times between September 2015 and March 2016, end times between October 2015 and May 2016

But replace rate is actually highest in the summer months, not the holiday season.

Spoiler: month-based replace rates I did mid-January, which have not changed significantly since then

Since I don't track the exact dates on which replacements happen (that would be almost impossible - do you call it on the replace in, on the replace out, the player's last post before vanishing, ???), I had to do something a bit cruder, but still seems to be legitimate. Here's the methodology used:

1. Determine which months each game was active for. If a game started on May 15th and ended on July 5th, it covers the months of May, June, and July.
   
2. Count up all the players replaced in each month. At the time of this post, May has 239, June has 272, and July has 302.
   
3. To try and remove the variable of "more games ran in this month", divide by the number of games that covered each month. There have been 58 May games, 70 June games, and 83 July games.
   
4. Therefore, the final values for May/June/July are 4.1, 3.9, and 3.6. The average replacement rate overall is ~3.5 players per game, so this seems legit enough - it should probably be read as relative rather than absolute anyway.

[Toomai]

It is true that the purpose of the 1xBP is more to balance the matrix than to actually help the town.

I don't think having a Vigilante of any sort in a newbie game is a good idea though. The game only starts out at 7:2; a mislynch, scum kill, and vig kill on town brings it to 4:2 MyLo for day 2. It knocks out the guarantee that all games will have at least one day where you have flips and it's not XyLo.