The Great Vote Count Analysis (Pre-Discussion)

[Psyche]

oh off and on

[petapan]

i just find it incredibly funny that something i did as scum that was all BS inspired this

not that I don't think it's valid and worthy to look into, i'd like to see what the numbers are rather than the arbitrary assumptions people usually make

[Psyche]

i can't tell yall how much i wish i had time for this

[N]

he hasn't actually done anything on this

[yessiree]

i can't tell yall how much i wish i had time for this

maybe you should consider reducing scope/deliverables cuz this looks like a classic case of scope bloat

[Hoopla]

- average frequency of town posts
- average frequency of scum posts

now that all the games i've been in have finished...

this is something i looked into recently.

using the last two years of 3:10 mini normals, i wanted to determine posting frequency of scum vs. town. for this data set, i only used posts on D1, so i didn't have to adjust for shifting town:scum ratios on future days. on D1, scum make up a shade over 23% of the playerpool, so we should expect each post to have a ~23% chance of being from scum.

across 48,094 D1 posts, 8,921 were from scum (18.54%) and 39,173 were from town (81.45%).

someone with better math skills than me should work out how statistically significant that is. perhaps slightly?

i would be interested in seeing this expanded to other queues and future days (though it would be annoying having to control for shifting scum:town ratios on later days).

[Hoopla]

- average frequency of town votes
- average frequency of scum votes

on a related note, this is also something i've been considering looking into using the same data set.

intuitively, i feel town tend to be splashier with votes the less information that exists in the game. whereas scum are more inclined to settle on a vote that is well justified. as scum, there is simply less incentive to "reevaluate the game" when you have landed on a hivemind approved vote.

[petapan]

- average frequency of town posts
- average frequency of scum posts

now that all the games i've been in have finished...

this is something i looked into recently.

using the last two years of 3:10 mini normals, i wanted to determine posting frequency of scum vs. town. for this data set, i only used posts on D1, so i didn't have to adjust for shifting town:scum ratios on future days. on D1, scum make up a shade over 23% of the playerpool, so we should expect each post to have a ~23% chance of being from scum.

across 48,094 D1 posts, 8,921 were from scum (18.54%) and 39,173 were from town (81.45%).

someone with better math skills than me should work out how statistically significant that is. perhaps slightly?

i would be interested in seeing this expanded to other queues and future days (though it would be annoying having to control for shifting scum:town ratios on later days).

definitely statistically significant. the expected mean posts from scum assuming equal posting frequency across alignments would be approximately 11099 posts, with a standard deviation of 92.4. (this is using the normal approximation for a binomial distribution). that means the distribution is 23.5 standard deviations below the mean, with a p-value of basically nothing (3.785x10^-123, wolfram alpha tells me). pretty much impossible for that to happen by random chance. intuitively makes sense that players are less likely to post as scum.

i'm not sure if you'd be able to apply anything at the individual game level, because on a smaller scale some people just have different posting styles. like, if you have a 1300 post day 1, assuming players posted in equal frency, you'd expect a mean of 100 posts per player. 2 standard deviations below that mean is 81 posts, which would say a poster with fewer posts than than is >rand scum. but on a practical level, is 20 posts a whole lot? idk.

[callforjudgement]

Your significance test doesn't work because it doesn't take into account the fact that each post is made by a player, and different players have different posting styles. As an extreme example, assume you have one hyperposter in each game who posts all the posts, and the other players don't post at all. Now what the test is measuring is the proportion of time the hyperposter happened to be scum. Your average is still correct, but your standard deviation isn't, because the number of samples is much smaller than the number of posts.

In order to work out if this is statistically significant, we need a workable null hypothesis, and "each post is equally likely to be made by each player, regardless of alignment" (what you have in your post) isn't it; even if there's no connection between posting frequency and alignment, we could still expect there to be a connection between posting frequency and player slot. I'm not sure offhand what the correct null hypothesis is, which is why I can't run the test myself.

[petapan]

ah yeah you're right, i knew intuitively that didn't make sense but it's been a long time since i did any of this math. yeah when you take it from an aggregate town/scum binary to different posters with different styles the assumption of being normally distributed usually won't work with 13 data points. still feels like there should be some way it can be used but i have no idea how

[Psyche]

hope i get the opportunity to replicate this stuff

[Gamma Emerald]

what's the status of this

[Psyche]

stalled when the pandemic put tons of new work on my shoulders but
i might be free again in a couple of days

[Psyche]

Starting this back again tomorrow. Gonna go for a timeboxing approach instead of picking specific dates to hit milestones. First step is just reminding myself what I have and what needs to be done, maybe revising some approaches based on strategies/techniques learned over the past few months.

[N]

what's the status of this

he unironically likes Mike Bloomberg now

[Psyche]

So I've run through the code and reviewed my notes.

My codebase is clearly enough written that I can remember what each part of it was

supposed

to do, but it doesn't look like I was careful when documenting new problems/gaps in the actual implementation. They're listed in my issue tracker, but descriptions are pretty terse: a given issue might get short summary sentence and a couple examples outputs proving something's wrong, and that's it. These notes are better than nothing, but the result is that I don't really know where in the 13 currently listed issues I should start, or remember what my approach would've been for addressing these issues. I'll probably have to retrace my steps through the project all over again before I can make changes confidently again. I'll come up with some better practices to avoid this work in the future.

I'll work on this again on Tuesday and then again on Friday or Saturday. Will maybe avoid updates until I, like, have actual new stuff to report.

[N]

psyche has no avatar... I assume this is done!?

[Psyche]

Okay, here's the plan for getting this project back into gear.

The idea is to follow yessiree's advice: scale down and focus my ambitions.

I've spent a lot of time trying to get my automatic votecounter to perform perfectly on every game in my development dataset. I'll accept that its performance now is probably close to the ceiling possible for my particular approach to the problem, and stop substantial efforts to improve the automatic votecounter.

Even though the votecounter doesn't perform perfectly, I do have a solid way to tell if the votecounter has done a good job: I can check if extracted votes accurately predict 1) who if anyone has been lynched in a given game Day and 2) the post number at which a given game Day has ended. Predicting both of these accurately doesn't guarantee that the votecounter has done its job perfectly, but it comes pretty close (I'll try to back this claim up with numbers at some point). So I can

validate

voting data collected for a particular game, even if I can't develop a votecounter that codes every game perfectly.

We never needed a perfect or near-perfect votecounter for this project. We just needed quality voting data collected over a large sample of games. So far I've been only using ~300 relatively old games to develop my votecounter. I'll collect/preprocess and run my votecounter over data associated with most or all completed games on MS instead. The votecounter will generate valid data for most of those. And if the pool of processed games is big enough, we'll have an adequately sized, validated dataset to support further analyses. And we'll always have the option to add data for further games if new games finish, or if we work out the issues with the votecounter's performance on the games where it has trouble.

I've always thought that the votecounting bit has been the hard part of this project. But this way we can make it a lot easier and get to the fun part a lot faster. And all I have to do is get over myself.

[Ythan]

I am hyped.