Daytalk for Newbies

[Psyche]

The underlying complexity fundamentally doesn't matter. You don't need to understand human anatomy to measure someone's running speed and you don't need to look at the games beyond their win-loss record to measure average town win rate. The average town win rate is basically just "suppose we ran a quadrillion games; take the observed win rate, that should be less than 0.0001% away from the average win rate.

If we did actually run a quadrillion games and town won 50% of them, then, if I pick a random game out of those quadrillion games, odds are 50% that it's a town win. If I pick a random sequence of 40 games the odds of the town winning 12 or less games is less than 1%; in particular, if you're going to run a quadrillion games of a perfectly balanced set-up, the odds are less than 1% that the town wins 12 or less games of the first 40 games.

We're obviously not actually going to run a quadrillion games, but there's nothing stopping us in theory and the conclusion simply still holds. If the average win rate were 50%, then the odds of the first 40 games including 12 or less town wins are less than 1%. If you want to argue against that, don't talk to me about personalities, tell me what's wrong with my argument.

Cfj, I tried the two null hypotheses "daytalk matrix6 isn't scum-sided (>50% town win rate)" and "daytalk matrix6 isn't too scum-sided (>45% town win rate)". The relevant data I used was 12 town wins out of 40 and then it's just a matter of using the binomial distribution. You get 0.8% for the first null hypothesis and and 3.8% for the second.

that's a cool way of handling limited information

[Toomai]

The experiment ran between Icfober and now, right? That's holiday season (1st half) and would raise overall replacement rate.

Did the town eeplace rate remain as high in the 2nd half?

First 40 games: start times between June 2015 and September 2015, end times between July 2015 and November 2015
Trial's 40 games: start times between September 2015 and March 2016, end times between October 2015 and May 2016

But replace rate is actually highest in the summer months, not the holiday season.

Spoiler: month-based replace rates I did mid-January, which have not changed significantly since then

Since I don't track the exact dates on which replacements happen (that would be almost impossible - do you call it on the replace in, on the replace out, the player's last post before vanishing, ???), I had to do something a bit cruder, but still seems to be legitimate. Here's the methodology used:

1. Determine which months each game was active for. If a game started on May 15th and ended on July 5th, it covers the months of May, June, and July.
   
2. Count up all the players replaced in each month. At the time of this post, May has 239, June has 272, and July has 302.
   
3. To try and remove the variable of "more games ran in this month", divide by the number of games that covered each month. There have been 58 May games, 70 June games, and 83 July games.
   
4. Therefore, the final values for May/June/July are 4.1, 3.9, and 3.6. The average replacement rate overall is ~3.5 players per game, so this seems legit enough - it should probably be read as relative rather than absolute anyway.

[Cogito Ergo Sum]

assuming day talk was the primary factor in why win rates plummeted seems like a pretty big assumption to me - yeah, it was added and that was a thing that happened, but no, that doesn't (necessarily) mean that it was the cause of the win rate dropping

When there's a change in win rate, there are two clear possibilities - either it's a structural change (and to me daytalk is the only structural change that could've had such a big impace) or something unlikely happened - now this "something unlikely" could still have a more specific form, e.g. if ICs drew scum 75% of the games or something, that would explain the win rate drop but that still falls under the "something unlikely" umbrella.

maybe it's that the setup(s) isn't(aren't) balanced to begin with

We've got a sample size of 287 games without daytalk with a town win rate of 48% (source although you have to subtract the daytalk experiment). That doesn't guarantee the set-up is reasonably balanced but that does seem like the obvious thing to believe.

but other than that, the odds of something being low (1% isn't unfathomably low to begin with) don't mean it can't have happened

Unlikely things do happen but that doesn't mean it's rational to believe that they happened. In statistics, 5% is generally considered to be the bar to clear before something is considered statistically significant. Certainly, drawing the right conclusion from an experiment 19 out of 20 times is a success rate I'd be okay with.

and... i don't think you're flipping a fair and balanced coin every time you run the setup... the setup(s) may be balanced based purely on EV but it isn't being reflected in practice (and if you say that doesn't matter, then i'm just going to shrug and point out that i think skill disparity between any two ICs, between any two batches of newbies, is going to play a bigger factor in who wins/loses than the balance of the setup and this isn't something I would expect to average out over 40 games - it may or it may not, but you're not drawing from a random pool in the first place)

The average win rate might not tell you as much about who's going to win as the playerlist might but as for it averaging out, see my last post. It really fundamentally doesn't matter what factors lead to wins and losses in specific games.

[zMuffinMan]

there is a relstively easy way to check whether you're right rather then relying on the numbers - actually look at the games in question and check things like scum PT length, amount of game-relevant posts that happened during the day, etc

if day talk were having an impact that significant and it's not something else causing, presumably there'd be some significant difference in how the PTs are being used

It really fundamentally doesn't matter what factors lead to wins and losses in specific games

i can somewhat understand this argument in principle, but if youre measuring the impact day talk has on the setup ("the only structural change that could have had such a big impact"), why does the reason for the wins and losses not matter?

[Mizzytastic]

Because he's not measuring that impact. As you said correlation does not imply causation. He doesn't know whether it was day talk that had the impact. What he does know is that something statistically significant happened during the trial. What he has shown is that it is very likely (95%+) that something weird happened during the trial. That weird could be random chance, it could be daytalk, or it could be something else that happened. There are a lot of variables you can't control for unless you did things like choose who was playing in what games.

However, although correlation doesn't imply causation but it can give a wink and a nudge, especially with variables for which you can already see potential mechanisms by which they could modify the results. What you could do now is go look at the games, or look through the stats for any other anomalies during the trial compared to the rest of the data we have and go see if anything else was up - but for now daytalk is a very reasonable suspect.

The question you have to ask is whether it's worth more experimentation, or you think the results are enough to say it's probably better to stick with night talk.

[zMuffinMan]

for now daytalk is a very reasonable suspect

why, though?

why would you assume, "oh, it must be daytalk," rather than looking to see if something else happened or if there were other obvious reasons for it?

edit: also what is the point of even talking about how day talk affected the setup if you're not even looking at whether day talk affected the setup?

[mhsmith0]

for now daytalk is a very reasonable suspect

why, though?

Well, Game Balance is dead, the body is missing and we found Daytalk on his boat tossing a few garbage bags into the ocean. I mean, Daytalk could just be helping out his friend Other Reasons... but I think the case is nevertheless strong, your honor

[zMuffinMan]

well isn't that the point? shouldn't you be asking for evidence that day talk was even at the scene of the crime when it occurred, whether maybe Other Reasons (potentially holding a bloody knife in his stained-red clothes) may have been responsible?

[Mizzytastic]

It's more a case of establishing that something weird happened while daytalk was in place. If it wasn't statistically significant that would also be a useful datapoint. This is why scientific experiments are supposed to be replicable. Unfortunately the nature of this site means it might not be preferrable to attempt to replicate it, so we may have to go with not enough information.

As for why it's a good suspect, it's occam's razor. We know something weird happend during the daytalk trial. We can either assume it was daytalk, or assume it was something else and that daytalk (which you can reasonably imagine could have had an effect) either didn't or is masked by whatever else happened. And most of the something elses that could have happened require lots of individual assumptions of it happening, whereas we know daytalk was in all games no matter what else happened.

Stats often aren't intuitive and we don't have enough experiments to make good conclusions. But it makes more sense to learn towards "that big change we introduced that could reasonably be expected to make games more scumsided made games more scumsided" than "there is no point considering this data because something else might have happened". Establishing something else did happen would of course change that, but we haven't done that yet so there is no point acting like we have.

[Mizzytastic]

well isn't that the point? shouldn't you be asking for evidence that day talk was even at the scene of the crime when it occurred, whether maybe Other Reasons (potentially holding a bloody knife in his stained-red clothes) may have been responsible?

Obviously this would be better but it's not been done yet. Do you have a systematic way of going about this, beyond looking at games and making a judgement?

Also, it's more like daytalk was at the scene of 40 crimes and you are saying "but there might be potentially different things exonerrating it at all those crimes?" There might be, and we should look for it, but don't you want daytalk brought in for questioning too?

[Ircher]

Here's an idea: Get the averages for number of pages in mafia PTs... For both parts of the trial. Also get the standard deviation.

If the numbers are relatively the same, then that would mean daytalk probably didnt have a significant impact, etc.

[Accountant]

VOTE: Daytalk

vig other reasons tonight

you can thank me for solving the game later

[mhsmith0]

VOTE: Daytalk

vig other reasons tonight

you can thank me for solving the game later

VOTE: Accountant for sheeping me.

[Accountant]

its not scummy to sheep people onto real scum

[mhsmith0]

its not scummy to sheep people onto real scum

It is if you're just late onto the wagon

[Accountant]

I'm the first person on the wagon, nobody else has voted daytalk before me

mhsmith0 bussing his partner and trying to misrep me???

[mhsmith0]

I'm the first person on the wagon, nobody else has voted daytalk before me

mhsmith0 bussing his partner and trying to misrep me???

How could I be bussing Daytalk if I'm voting you? Time for a flash wagon on accountant guys!

[zMuffinMan]

masked by whatever else happened

i wouldn't say "masked", considering not one person has actually spent the time looking to see if anything else could have been responsible

we know daytalk was implemented, yes. we don't know if anything else happened that skewed the results. and even then, we do know that some things happened that could have skewed the results - IC replace rate and newbie town replace rate increased - maybe these weren't related to daytalk but did directly impact the win rates. maybe other things were happening that caused it. i haven't checked, and neither has anyone else, and i don't have a good suggestion for how to go about checking, but i don't think saying "well, daytalk and scum win rate can't be a coincidence" is justified in any way

if mafiascum change their banner and there happened to be a significant increase in town win rate, would you then start suggesting it can't be a coincidence? or would you perhaps look to see if maybe something else was causing it? and don't say "well, that's silly, because day talk affects the game and that doesn't" because does it really? how do you know? have you actually looked at the games in question and determined whether scum wouldn't have won without it?

also, maybe the first and second half of the trial shouldn't be treated as connected. maybe toomai is right and town started adapting in the second half of the trial. maybe if the trial went 20 more games, town win rate would have skyrocketed because they learnt to spot something scum was doing when they were using day talk.

hey, maybe it's because there's a rise in town players who see something and assume it must be because of something else without actually doing any real analysis

the point im making isn't so much that any of this is or isn't the case. maybe it is that daytalk is just that imbalanced... but maybe it isn't, and maybe jumping to the conclusion that daytalk was the thing that impacted the win rate isn't justified

Also, it's more like daytalk was at the scene of 40 crimes and you are saying "but there might be potentially different things exonerrating it at all those crimes?"

yeah, but was it really at the scene of the crimes? like, if scum weren't using daytalk or were rarely using it then it maybe happened in the general area that daytalk lives in but daytalk has a pretty strong alibi suggesting it isn't related. if scum were using it, then you've maybe spotted it at the same club on the night the crime occurred - now try connecting more dots before you jump to conclusions.

[Mizzytastic]

I agree with you - saying it can't be a coincidence is too far. I haven't said that, just that the statistics lean in a certain direction. As I said, correlation doesn't imply causation, but it can give a wink and a nudge. Unfortunately the tests that would be required to be definitely testing day talk lie somewhere between incredibly detrimental to newbie games to impossible. Sometimes you are stuck with working with imperfect data and you have to do the best you can.

I'm also not saying that we shouldn't go and look to see if something else happened during those games. We definitely should. However in the absence of that (which we are currently in) we have a plausible mechanism to making games scumsided and evidence showing that it's very likely games were scumsided. If someone changed the banner I would be more cynical because of the lack of a plausible mechanism, but I would still consider it worth looking at more, which is the same position I'm on here.

Do you have a reason to treat the first and second half as different other than the change that took place? Because otherwise that would be cherry picking. It's best to look at the data as a whole unless you can define a reason outside of the results that they should be considered as different groups. If I flip a fair coin a bunch of times then am allowed to split up the results it's entirely possible I can make it look like the coin is sometimes suddenly becoming massively unfair.

At best what we could get by looking at the games (which I support doing) is some other plausible mechanisms with evidence that could also explain the change. But even in that case at best we have data that can't be read in relation to daytalk. This is part of the reason that it's often best to do multiple experiments, and why larger sample sizes are better. The problem we have is this is that we are testing on 'wild' games, and the newbie experience is very important. We have no evidence to suggest that daytalk was causing the desired effects, and some evidence suggesting it could be having other detrimental effects. And that's where we really end up - is it worth in the light of this data point doing more trials with the risk we are introducing a mechanic into newbie games making them more scumsided, now that we have a data point leaning in that direction.

[callforjudgement]

One possible explanation for a change in win rates as a result of daytalk being introduced: a different set of people than normal joining the Newbie Queue in order to see what the new setup was like. The cause wouldn't necessarily have been related to the daytalk itself, just the fact that there was a change. It would also explain the two halves being different.

I'm not saying that this is what happened, just that it's a possibility.

[zMuffinMan]

we have a plausible mechanism to making games scumsided

ummm....

ignoring the fact that there's a lot of issues with just saying "x happened and it could be caused by y therefore it's likely y caused it", let's be realistic here... it's day talk, not any sort of actual imbalance in the setup. the most day talk affords you is the ability to discuss your thoughts on the game during the day (which isn't to say that can't be helpful, it's just not some sort of game-breaking advantage outside of lylo situations), and it probably wasn't even used in a way that would cause the fluctuations that happened even if daytalk did affect the balance

what we have here is something we

know

changed

what we don't have here is anyone doing any sort of examination on whether anything else may have caused what happened

i'm not suggesting i know how best to investigate it. i'm also not suggesting that it's necessarily the case there is something else (hey, maybe day talk just is that imbalanced but apparently only in newbie games). i'm just saying it's incredibly stupid to assume it was day talk itself that caused what happened

edit: actually, it's not so much stupid to make that assumption as it is to base other things off that assumption. if day talk is going to be forever disregarded or thought of as "imbalanced" because of this, then that's incredibly stupid

Do you have a reason to treat the first and second half as different other than the change that took place?

no, but there are certainly arguments that could be made for treating them differently - whether any of them are valid, i don't know. maybe something

did

change. the above post is one of many possible things that might have happened

[Ircher]

Here's an idea: Get the averages for number of pages in mafia PTs... For both parts of the trial. Also get the standard deviation.

If the numbers are relatively the same, then that would mean daytalk probably didnt have a significant impact, etc.

[Zachrulez]

Why would the length of day talk have an impact on win rates?

This might seem like an oversimplified statement, but daytalk was the only thing that was changed in the newbie setup during the trial. The only other thing to blame would really be the players playing the roles. I don't think I've ever seen an argument where players could have a +20% impact on a game, so it seems like a stretch to give the quality of players all the blame for the fluctuations. Daytalk seems to be the only viable suspect.

[Ircher]

Because the more olayers, esp. newbies, cooperate and chat, the more imoacg daytalk has.

[Kagami]

The variance in the size of the scum PT is primarily going to be driven by the players. I would not be surprised if there is a very real effect from daytalk while there is no significant difference in PT length.

I think the more interesting comparison is the posting density of scum players in the main thread with and without daytalk. I suspect the reason for the difference in performance is that scumteams with daytalk are more motivated/invested than their nighttalk counterparts, and that this effect is largely independent of the quantity of game-related content in the scum PT.