The Great Vote Count Analysis (Pre-Discussion)
- Psyche
-
Psyche he/theySurvivor
- Psyche
- petapan
-
petapan Survivor
- petapan
- Survivor
- Survivor
- Posts: 11748
- Joined: May 8, 2012
- Psyche
-
Psyche he/theySurvivor
- Psyche
he/they- Survivor
- Survivor
- Posts: 10698
- Joined: April 28, 2011
- Pronoun: he/they
- N
-
N Jack of All Trades
- N
- yessiree
-
yessiree heMafia Scum
- yessiree
he- Mafia Scum
- Mafia Scum
- Posts: 4389
- Joined: June 6, 2013
- Pronoun: he
maybe you should consider reducing scope/deliverables cuz this looks like a classic case of scope bloatIn post 102, Psyche wrote:i can't tell yall how much i wish i had time for this- Hoopla
-
Hoopla
- Hoopla
- Posts: 10788
- Joined: October 12, 2008
now that all the games i've been in have finished...
this is something i looked into recently.
using the last two years of 3:10 mini normals, i wanted to determine posting frequency of scum vs. town. for this data set, i only used posts on D1, so i didn't have to adjust for shifting town:scum ratios on future days. on D1, scum make up a shade over 23% of the playerpool, so we should expect each post to have a ~23% chance of being from scum.
across 48,094 D1 posts, 8,921 were from scum (18.54%) and 39,173 were from town (81.45%).
someone with better math skills than me should work out how statistically significant that is. perhaps slightly?
i would be interested in seeing this expanded to other queues and future days (though it would be annoying having to control for shifting scum:town ratios on later days).- Hoopla
-
Hoopla
- Hoopla
- Posts: 10788
- Joined: October 12, 2008
on a related note, this is also something i've been considering looking into using the same data set.
intuitively, i feel town tend to be splashier with votes the less information that exists in the game. whereas scum are more inclined to settle on a vote that is well justified. as scum, there is simply less incentive to "reevaluate the game" when you have landed on a hivemind approved vote.- petapan
-
petapan Survivor
- petapan
- Survivor
- Survivor
- Posts: 11748
- Joined: May 8, 2012
definitely statistically significant. the expected mean posts from scum assuming equal posting frequency across alignments would be approximately 11099 posts, with a standard deviation of 92.4. (this is using the normal approximation for a binomial distribution). that means the distribution is 23.5 standard deviations below the mean, with a p-value of basically nothing (3.785x10^-123, wolfram alpha tells me). pretty much impossible for that to happen by random chance. intuitively makes sense that players are less likely to post as scum.In post 105, Hoopla wrote:now that all the games i've been in have finished...
this is something i looked into recently.
using the last two years of 3:10 mini normals, i wanted to determine posting frequency of scum vs. town. for this data set, i only used posts on D1, so i didn't have to adjust for shifting town:scum ratios on future days. on D1, scum make up a shade over 23% of the playerpool, so we should expect each post to have a ~23% chance of being from scum.
across 48,094 D1 posts, 8,921 were from scum (18.54%) and 39,173 were from town (81.45%).
someone with better math skills than me should work out how statistically significant that is. perhaps slightly?
i would be interested in seeing this expanded to other queues and future days (though it would be annoying having to control for shifting scum:town ratios on later days).
i'm not sure if you'd be able to apply anything at the individual game level, because on a smaller scale some people just have different posting styles. like, if you have a 1300 post day 1, assuming players posted in equal frency, you'd expect a mean of 100 posts per player. 2 standard deviations below that mean is 81 posts, which would say a poster with fewer posts than than is >rand scum. but on a practical level, is 20 posts a whole lot? idk.free crypto- callforjudgement
-
callforjudgement Microprocessor
- callforjudgement
- Microprocessor
- Microprocessor
- Posts: 3972
- Joined: September 1, 2011
Your significance test doesn't work because it doesn't take into account the fact that each post is made by a player, and different players have different posting styles. As an extreme example, assume you have one hyperposter in each game who posts all the posts, and the other players don't post at all. Now what the test is measuring is the proportion of time the hyperposter happened to be scum. Your average is still correct, but your standard deviation isn't, because the number of samples is much smaller than the number of posts.
In order to work out if this is statistically significant, we need a workable null hypothesis, and "each post is equally likely to be made by each player, regardless of alignment" (what you have in your post) isn't it; even if there's no connection between posting frequency and alignment, we could still expect there to be a connection between posting frequency and player slot. I'm not sure offhand what the correct null hypothesis is, which is why I can't run the test myself.scum· scam · seam · team · term · tern · torn ·town- petapan
-
petapan Survivor
- petapan
- Survivor
- Survivor
- Posts: 11748
- Joined: May 8, 2012
ah yeah you're right, i knew intuitively that didn't make sense but it's been a long time since i did any of this math. yeah when you take it from an aggregate town/scum binary to different posters with different styles the assumption of being normally distributed usually won't work with 13 data points. still feels like there should be some way it can be used but i have no idea howfree crypto- Psyche
-
Psyche he/theySurvivor
- Psyche
he/they- Survivor
- Survivor
- Posts: 10698
- Joined: April 28, 2011
- Pronoun: he/they
- Gamma Emerald
-
Gamma Emerald AnySurvivor
- Gamma Emerald
Any- Survivor
- Survivor
- Posts: 69101
- Joined: August 9, 2016
- Pronoun: Any
- Location: Hell on Earth (aka Texas)
- Psyche
-
Psyche he/theySurvivor
- Psyche
he/they- Survivor
- Survivor
- Posts: 10698
- Joined: April 28, 2011
- Pronoun: he/they
- Psyche
-
Psyche he/theySurvivor
- Psyche
he/they- Survivor
- Survivor
- Posts: 10698
- Joined: April 28, 2011
- Pronoun: he/they
- N
-
N Jack of All Trades
- N
- Jack of All Trades
- Jack of All Trades
- Posts: 8539
- Joined: August 2, 2012
he unironically likes Mike Bloomberg nowIn post 111, Gamma Emerald wrote:what's the status of this- Psyche
-
Psyche he/theySurvivor
- Psyche
he/they- Survivor
- Survivor
- Posts: 10698
- Joined: April 28, 2011
- Pronoun: he/they
So I've run through the code and reviewed my notes.
My codebase is clearly enough written that I can remember what each part of it wassupposedto do, but it doesn't look like I was careful when documenting new problems/gaps in the actual implementation. They're listed in my issue tracker, but descriptions are pretty terse: a given issue might get short summary sentence and a couple examples outputs proving something's wrong, and that's it. These notes are better than nothing, but the result is that I don't really know where in the 13 currently listed issues I should start, or remember what my approach would've been for addressing these issues. I'll probably have to retrace my steps through the project all over again before I can make changes confidently again. I'll come up with some better practices to avoid this work in the future.
I'll work on this again on Tuesday and then again on Friday or Saturday. Will maybe avoid updates until I, like, have actual new stuff to report.- N
-
N Jack of All Trades
- N
- Psyche
-
Psyche he/theySurvivor
- Psyche
he/they- Survivor
- Survivor
- Posts: 10698
- Joined: April 28, 2011
- Pronoun: he/they
Okay, here's the plan for getting this project back into gear.
The idea is to follow yessiree's advice: scale down and focus my ambitions.
I've spent a lot of time trying to get my automatic votecounter to perform perfectly on every game in my development dataset. I'll accept that its performance now is probably close to the ceiling possible for my particular approach to the problem, and stop substantial efforts to improve the automatic votecounter.
Even though the votecounter doesn't perform perfectly, I do have a solid way to tell if the votecounter has done a good job: I can check if extracted votes accurately predict 1) who if anyone has been lynched in a given game Day and 2) the post number at which a given game Day has ended. Predicting both of these accurately doesn't guarantee that the votecounter has done its job perfectly, but it comes pretty close (I'll try to back this claim up with numbers at some point). So I canvalidatevoting data collected for a particular game, even if I can't develop a votecounter that codes every game perfectly.
We never needed a perfect or near-perfect votecounter for this project. We just needed quality voting data collected over a large sample of games. So far I've been only using ~300 relatively old games to develop my votecounter. I'll collect/preprocess and run my votecounter over data associated with most or all completed games on MS instead. The votecounter will generate valid data for most of those. And if the pool of processed games is big enough, we'll have an adequately sized, validated dataset to support further analyses. And we'll always have the option to add data for further games if new games finish, or if we work out the issues with the votecounter's performance on the games where it has trouble.
I've always thought that the votecounting bit has been the hard part of this project. But this way we can make it a lot easier and get to the fun part a lot faster. And all I have to do is get over myself.Copyright © MafiaScum. All rights reserved.
- Psyche
- Psyche
- N
- Psyche
- Psyche
- Gamma Emerald
- Psyche
- petapan
- callforjudgement
- petapan
- Hoopla
- Hoopla
- yessiree
- Psyche
- petapan