Computational Mafia

This forum is for discussion related to the game.
Locked
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Computational Mafia

Post Post #0 (isolation #0) » Tue Feb 12, 2019 5:37 am

Post by Psyche »

I thought maybe it'd be good to have a unitary thread to discuss computational approaches to mafia (running/managing games, playing games, reviewing games) absent any focus on particular implementations. There's a sizable community of people who do this, but i feel that due to differences in strategy we're often get into our own islands and don't really interact much or support one another. Let's use this thread to trade notes from time to time!

We can talk about what we're up to, discuss general/recurring problems, share data, whatever.

Some Relevant Projects
Last edited by Psyche on Thu Jun 13, 2019 9:54 pm, edited 2 times in total.
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #1 (isolation #1) » Tue Feb 12, 2019 6:57 am

Post by Psyche »

Here's a summary of where I think we are. Roughly put, I think the big challenge of computational mafia is how messy and difficult to automatically process game threads are. Natural language processing of player posts is one thing, but even the detection and processing of relatively standardized happenings like player votes, role flips, moderator transitions between game phases, and the like are profoundly challenging things to make a computer do! The result is that the more interesting potential projects in computational mafia require either many hours of coding, many hours of manual data labeling, or forcing players/moderators to change their behavior so that game threads going forward are more streamlined.

We've some success of these approaches, but also a lot of failure. A lot of the necessary hand-prepared data sets exist (e.g. the Game Archive threads), but other data sets often go unprepared due to the huge amounts of work or even unshared after many hours are put into their preparation. Players/Mods have standardized their behavior considerably over the years but a lot remain rightfully resistant to enacting controls on their in-game behavior for the sake of computationalism, and often undisciplined anyway if they aren't; and this method leaves many years of already existing game data abandoned anyway.

As for coding work, some key problems like vote detection/counting under relevant conditions have been solved many times over through distinct projects (including mine) around the site. The result is that in most game threads it's possible to automatically count every vote ever made. Ellibereth famously managed to enhance his Mafia play to near-ceiling levels by formalizing some aspects of his scumhunting strategy. Other tools/formal methods exist or nearly exist for generating and reviewing game setups, timing votecount posts, and other tasks essential to modding. Unfortunately, due to interface issues, secrecy, mod preferences or other challenges, utilization rates for tools like these is rare. Other websites like Mafia Universe cohesively integrate moderator automation tools into the design of their site, and the holy grail of replicating that same success on MafiaScum is a project that remains unfinished but widely anticipated. The more lofty dream of public tools supporting the actual classification of scum and town players from their posts seems at turns substantially further off and all but already here.

Going forward, absent any big changes, we're likely to see more work in fits and starts on these efforts along these lines. Big milestones we might see that might change everything? The site upgrade could happen soon and be successful. Maybe further breakthroughs in thread processing will make it a lot easier for people across the site to build scumhunting tools, this will turn out to be relatively easy, and work will shift to regulating and improving them rather than exploring if it's possible. Sure, there's a chance that instead the site will eventually die out, but that's far off and regardless Mafia is probably here to stay; it's possible that people will just take advantages of sites that implement the concept of forum mafia more cleanly and thus offer cleaner data.
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #2 (isolation #2) » Tue Feb 12, 2019 7:32 am

Post by Psyche »

there are a few ways i think we can make our work smoother, though:

Share Data

I think most people with coding projects share their code and then no one bothers to read it. I believe that there's a much bigger demand for data not captured or just cleaner than what's in the public archives. I think the user who was named Tamuz once hand-coded an archive for every Newbie game that's ever happened, catalogueing player information, game outcomes, all of that; it seems to be totally lost now. Ground truth data is absolutely necessary either to implement practically any potential computational mafia project or to evaluate if one actually works.

Generate Data

So obviously it would be nice if someone would do the tedious work of organizing the ground truth data needed for most computational mafia projects off the ground. Luckily, people volunteer surprisingly often to do this kind of stuff. My interested in computational mafia has been reinvigorated lately because I've finally found an affordable way to pay other people to do this work too in the way that's necessary for our work, something that could hopefully power much more rapid progress in this domain.

But bigger than that, I think a lot of people (including me) have missed the kind of data that their own computer programs could generate for other people to use if they just made a few tweaks and formatted the output. For example, I mentioned earlier that it's now possible to count every vote in most game threads. People not interested in figuring out how to use my votecounter might find a data set including all of these votecounts really useful, either for their own projects or just for in-game meta-analyses. Similarly, my votecounter includes a tool for matching nicknames and abbreviations to particular players in a game - a dictionary of players and their common nicknames/abbreviations might be similarly broadly useful. I suspect a lot of people doing computational mafia have in their codebases affordances like that but haven't made the effort to extract as much value from them as possible by generating, organizing, and sharing it.

Talk Through Common Challenges and Solutions

No one enjoys looking at other people's code, so the old practice of sharing github repositories seems insufficient for supporting each other through our shared challenges. In general in our big "here's my project" threads I think we should spend more time talking about the problems we faced and how we solved them alongside what our projects achieve. And more specifically, hopefully we can talk about computational mafia as a broad effort here!

And idk, drop more of your lives and just spend more time on this. I'm pretty sure the big reason nothing happens on this project is because the work required to work importance ratio is a bit high. That's...understandable.
Last edited by Psyche on Tue Feb 12, 2019 7:46 am, edited 1 time in total.
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #3 (isolation #3) » Tue Feb 12, 2019 7:35 am

Post by Psyche »

Going forward I'm personally focusing on finally organizing a comprehensive data set of all the non-post content information that exists across the game threads on MS that are useful as data. Once that happens, MS becomes much more fun as a playground of behavioral data. Like I said before, I'm pulling from my hobby budget to make the work someone else's problem. Hopefully it's all done in a month. It's so fun to be young and have money.
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #5 (isolation #4) » Tue Feb 12, 2019 7:56 am

Post by Psyche »

at some point i realized that all i really cared about was building the API

speaking of which, look how easy to use and thoroughly documented my Donbot is! https://github.com/MafiaScum-Unofficial ... ter/donbot
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #6 (isolation #5) » Wed Feb 13, 2019 9:41 am

Post by Psyche »

think another way i might help is by writing some kind of review or map of all the progress made so far bc clearly im not aware of it all

Will populate a pile of links and notes here in this post for the time being.

The Mafiascum Dataset
Bicephalous Bob cleans/prepares a dataset of 700 games/10000 documents is freely available for academic use and achieves a good automated role classification model, too. Great leap forward for the effort.

The Newbie 2d3 stats thread
Data could serve work to predict game outcomes, formalize setup balance/generation, etc. Human performance statistics are useful for evaluating computational model performance; Similarly: viewtopic.php?f=5&t=39739

MathBlade's Vote Scrubber
Automated vote detection, even in cases of misspelled/nicknamed players (meaning it's a username matcher, too). Interesting links to other stuff.

etc
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #10 (isolation #6) » Fri Feb 15, 2019 11:16 am

Post by Psyche »

Could you be exact about what you’d like for the API to do?
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #13 (isolation #7) » Mon Feb 25, 2019 5:57 am

Post by Psyche »

shouldn't be too hard implement something at least really close

anyway, now have labels for phase number for every mini normal post in bob's data set (plus any newer games that have finished) and will soon have the same for both normals and opens
once it's fully cleaned up i'll share it with everyone!

this'll enable phase-by-phase analysis of game threads that before was impossible without considerable manual work or tiny data sets
and also more detailed game archives to help people doing metas get where they're trying to get faster
with definite day transition post numbers coupled with a good vote detecter, we'll be able to generate wagon data through every one of these hundreds of games
Last edited by Psyche on Mon Feb 25, 2019 6:36 am, edited 1 time in total.
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #15 (isolation #8) » Mon Feb 25, 2019 6:42 am

Post by Psyche »

i mean at the end of the day if that's your goal then maybe you should join the site redesign effort and just build your votecounter into the system
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #19 (isolation #9) » Wed Feb 27, 2019 10:10 am

Post by Psyche »

whatever happens there’ll always be other sites
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #21 (isolation #10) » Sun Mar 03, 2019 2:07 pm

Post by Psyche »

https://docs.google.com/spreadsheets/d/ ... sp=sharing

Hi guys. The above spreadsheet is unfinished and will need to be cleaned a lot once done, but it will include the post number of every phase transition in all the games covered in Bob's data set, plus any newly completed games in the relevant subforums since. This kind of data should permit or facilitate a broad array of computational efforts tied to mafia and MS. I'll let you know when it's fully "done", but I'm too excited not to share it now. More later.
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #23 (isolation #11) » Sun Mar 03, 2019 4:20 pm

Post by Psyche »

any other kinds of data collection too hard to have a computer do well short of actively reading games? haven't quite figured out what if anything i'll have my coder do after she finishes this thing
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #25 (isolation #12) » Mon Mar 04, 2019 11:38 am

Post by Psyche »

oh i remember somehow loading 200 post pages long ago but forgot how i did that

why'd you put the scraper in js, btw?
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #29 (isolation #13) » Tue Apr 16, 2019 7:12 pm

Post by Psyche »

https://github.com/iodide-project/pyodide

this is really interesting to me as someone who struggled bridging my python-based comp mafia efforts to the web without maintaining a server
i have no idea how it works but it’s totally on my to-do list to explore it
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #31 (isolation #14) » Tue Apr 16, 2019 9:19 pm

Post by Psyche »

neato
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #32 (isolation #15) » Tue Apr 16, 2019 9:45 pm

Post by Psyche »

meh pyodide is far from usable atm i gotta stop listening when people tell me to care about things
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #34 (isolation #16) » Thu May 30, 2019 9:09 am

Post by Psyche »

im gonna clean up that day transition dataset soon and start crunching numbers
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #36 (isolation #17) » Fri Jun 07, 2019 1:45 pm

Post by Psyche »

mix of code mix of tedium outsourced to a nice 20-something mother who lives in the phillipines for $3 an hour
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #37 (isolation #18) » Fri Jun 07, 2019 1:47 pm

Post by Psyche »

i know the pay isn't great but the phillipines are actually a pretty nice place a lot of people choose to live there for free
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #51 (isolation #19) » Thu Jun 13, 2019 9:12 am

Post by Psyche »

have always wanted to write a folder of computational notebooks to support players' breaking down games
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #52 (isolation #20) » Tue Jun 18, 2019 4:44 pm

Post by Psyche »

my supp feedback thread in the speakeasy depends in part on the donbot api we developed last year or so; really eases the challenge of reminding myself every project how to automatically interact w/ the site
really hopeful i can leverage the engagement w/ the codebase the thread required into work on interests in this thread
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #53 (isolation #21) » Tue Jun 18, 2019 4:54 pm

Post by Psyche »

i'll start small
i'll review the donbot codebase for any bugs/gaps and fix them, all the way through the votecounter - maybe reorganize the whole project
from there i'll start actually making stuff (automated game moderation?) or asking basic research questions, depending on my whim

but in the end i might have better things to do (i definitely do...)
still, i'd really like to get back into a hobby that counts as a creative outlet
as long as i don't take things too seriously it shouldn't be too hard to stay energized
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #54 (isolation #22) » Fri Jun 21, 2019 8:11 pm

Post by Psyche »

alright

i'm gonna try to finish to a complete (well, minimum viable product) votecounting bot by the end of the month
given a playerlist and a posting template and a starting post number and assuming no special voting mechanics/roles, it will automatically top every page with an up-to-date votecount and a final votecount plus other optional events when it believes a lynch has been reached
this functionality will be devolved in an API too, enabling people to use the votecounter or pagetopper for other purposes, including research on voting patterns

practically all the relevant code exists, but iirc some debugging is necessary

i don't think anything else about modding a game really needs automation besides perhaps initial handling of role pms
reliably posting votecounts, though? that's incredibly tedious but whether it's done or not has a massive impact on player experience
finishing a bot for that will open me up to modding more games w/o looking like a jackass
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #56 (isolation #23) » Mon Jul 01, 2019 1:00 am

Post by Psyche »

I failed!

did some other work though I swear
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #59 (isolation #24) » Sun Sep 29, 2019 1:05 am

Post by Psyche »

managed to trick my PI into funding my interest in complinguistics so I think I'm gonna return to this world again, though I'm even more focused than before on research over making tools for people
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #62 (isolation #25) » Thu Dec 05, 2019 9:04 pm

Post by Psyche »

hey guys
anyone do anything new w/o telling me?
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #63 (isolation #26) » Sat Dec 07, 2019 12:38 pm

Post by Psyche »

30 days of code for 30 days in a row on my project 30 days let's go 30 days
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #65 (isolation #27) » Mon Dec 16, 2019 3:16 pm

Post by Psyche »

donbot is slow - maybe not too slow since the average mafia thread is not very long compared to the alternative, but something to worry about; at very least i might want to add a progress tracker to curb impatience when threads with like 10,000 posts are being processed
also it explodes if a thread you're processing has one of your foe's accounts' posts in it, but i think that shouldn't be a problem in the mafia subforum. Might add some warning into the code or documentation somewhere...

i think that since i'm leaning toward specializing in data analysis over, like, moderation automation, i'll probably switch to a faster scraping scheme

it does feel quite easy to use! i think i need to bring the documentation closer to the homepage of the repo though
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #67 (isolation #28) » Mon Dec 16, 2019 3:50 pm

Post by Psyche »

yah tqdm very sexy
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #68 (isolation #29) » Tue Dec 17, 2019 4:43 pm

Post by Psyche »

A really concrete overarching goal of mine for a long time now is to build a foundation of preprocessed data and analysis code to ask if there are any reliable patterns in voting behavior that are useful as scum or town tells. The idea is that differences of win condition and knowledge probably drive distinct voting patterns between factions. I've called this effort "the Great Vote Count Analysis".

In theory, the tools and preprocessed data necessary to perform initial analyses of this sort already exist. We have a votecounter that than scan threads for and identify even really messy votes, games archives that track the roles and alliances and deaths of every player, and a hand-coded excel workbook identifying the post numbers of transitions between Day and Night phases across a huge number of games.

It's possible that we'll need even more than that - for example, proponents of VCA often insist that the motives/justifications and context surrounding votes are just as or more important for finding scum in votecounts as the votes themselves. But the basic core foundation of it all is already there and just needs to be tied together.

My hope is that once I tie it together, other people will be interested enough to try out analysis approaches themselves, but I know that's dumb.
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #70 (isolation #30) » Tue Dec 17, 2019 6:03 pm

Post by Psyche »

In post 21, Psyche wrote:https://docs.google.com/spreadsheets/d/ ... sp=sharing

Hi guys. The above spreadsheet is unfinished and will need to be cleaned a lot once done, but it will include the post number of every phase transition in all the games covered in Bob's data set, plus any newly completed games in the relevant subforums since. This kind of data should permit or facilitate a broad array of computational efforts tied to mafia and MS. I'll let you know when it's fully "done", but I'm too excited not to share it now. More later.
i still haven't cleaned it
i paid some filipino lady to make this, was such a great scheme
Last edited by Psyche on Tue Dec 17, 2019 6:05 pm, edited 1 time in total.
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #72 (isolation #31) » Tue Dec 17, 2019 6:08 pm

Post by Psyche »

So I'm steadily working to tie this data with the votecounter (i'll pre-extract all relevant votes) and game archives maybe also a Big Giant Scrape of all the relevant threads for good measure. The result will probably be more than a gig of data, but with that we should really be able to get this show on the road.
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #74 (isolation #32) » Tue Dec 17, 2019 6:45 pm

Post by Psyche »

oh good that's motivating to read
i'll talk up my progress here then

gameplan:
We got to make sure 3 things are in shape and either correct issues or filter out threads where there are issues (e.g. missing/borked posts, weird disruptions like modflaking, bizarre setups or mechanics that might disrupt analysis such as weird voting dynamics or post restrictions) :

- the phase transition spreadsheet above
- the game archives detailed the names, roles, and fates of every player (as well as identifying mods!)
- the votecounter

Think one way to sort of automate detection of issues in these data sources is to build a program that only works if these data sources are solid and will scream at you when it fails. I'll write a program that tries to use these data/tools to report who got lynched each Day and finally who won the game. The votecounter will start at each transition point as identified in the spreadsheet and track votes until it finds a lynch. It'll rely on the game archives to figure out whose votes count on a given Day as well as to check its work. If any one of the three above things is catastrophically wrong, the program should fail and I'll get the chance to either abandon the thread or fix the issue.

This wouldn't catch all errors (eg roles could still be wrong, and the correct lynch could be found by the votecounter even if it misses some votes or starts from the wrong post), but hopefully identifying/resolving the errors it does catch will solve or identify most of the errors that pass silently.

So the next checkpoint is getting this test program running. After that it's working through all the errors. Finally, it's reporting the data set.
But of course after that is the data analysis.

Let's see if I can get it done as a winter break project.
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #76 (isolation #33) » Fri Dec 20, 2019 10:03 am

Post by Psyche »

my fking votecounter requires 32-bit python
how did i let this happen
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #77 (isolation #34) » Fri Dec 20, 2019 10:05 am

Post by Psyche »

oh and here's an example archive: viewtopic.php?f=53&t=29549
had to do quite a bit of cleaning to make this something that doesn't create trouble but it's better than collecting it all myself
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #79 (isolation #35) » Fri Dec 20, 2019 12:33 pm

Post by Psyche »

would be easy to code w a cleaned up (universal) archive
maybe one day
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #80 (isolation #36) » Sat Dec 21, 2019 8:20 am

Post by Psyche »

In post 76, Psyche wrote:my fking votecounter requires 32-bit python
how did i let this happen
this is the absolute worst
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #81 (isolation #37) » Tue Jan 07, 2020 7:10 am

Post by Psyche »

ok i've figured out a fix that isn't "start all over"
it'll still require a bit of work but hopefully it'll result in a better product anyway
and an opportunity to try out some new coding practices
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #83 (isolation #38) » Tue Jan 14, 2020 5:15 pm

Post by Psyche »

found a replacement spellcheckerrr
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #85 (isolation #39) » Mon Jan 20, 2020 1:25 pm

Post by Psyche »

it's been almost a week have an update

i partly cleaned up my two main data sets - one the transition spreadsheet identifying where phase transitions happened throughout each game, and the other basically a computer readable and cleaned-up subsection of the mini normal archives already maintained on this site identifying every relevant player in the game (plus mods), their roles, and their fates
the cleaned up stuff covers only about ~300 mini normals and mostly only handled the prospect of missing information - not inaccurate
furthermore, i still need to get my votecounter working again

As mentioned before, the strategy for identifying errors is to try to combine the three data sources to complete a task that's hard to complete unless they're all accurate:

With the votecounter and the archive indicating which players are still alive on a given Day, i'll try to infer (for every day phase in my data set) where a hammer has happened.
If the next mod post after the hammer isn't the transition point my transition spreadsheet indicates for that Day, then there's a good chance of an issue in my data or code. Similarly, if my player archive and my votecounter disagree on who the hammered player is, that's also a red flag.

If I can successfully infer the transition post and hammered player from voting data and player information for every phase of every game in my data set, I'll proceed to also try to infer which team won the game and compare that against what my archive says (though nothing in my codebase can infer who got killed in a night phase, or handle vote-manipulating PRs, so the code will get some help from the fate-identifying part of the archive).

There could still be issues in the data set or code after achieving all this, but I imagine that if I expand the data set far enough the initial 300 games while trying to use the data set to do more ambitious things, I'll catch them eventually. I'll hire the filipino lady again and help her provide for her children.

And then a side effect of all this testing will be a cleaned data set tracking every vote that happened across all 300 games. That's when we can start the Great Vote Count Analysis. And then we'll find absolutely no reliable pattern in any of these votes even after appling fancy NLP tools to take "context" into account and I'll finally leave mafiascum.net forever. Oh but at least we'll also have a functioning votecounter that works across even old games w/ loose mods. And no one will use it. Well at least I'll be able to move on.
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #86 (isolation #40) » Sun Jan 26, 2020 11:28 am

Post by Psyche »

fixed the votecounter - it can determine the person lynched on D1 across all 300 mini normals in my sample
it's kinda slow though?
i guess im ok with slow as long as it always works

still need to...
extend test to determine transition posts from post# of hammer
extend test to Days beyond D1
extend test to more games
convert test results into cleaned data set
plan out and start analyses

maybe i'll try to solicit ideas a little further down the line
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #88 (isolation #41) » Fri Feb 07, 2020 8:03 pm

Post by Psyche »

god i rlly want to work on this but i should wait i should totally wait
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #89 (isolation #42) » Sat Feb 08, 2020 1:46 am

Post by Psyche »

i did a little work
solved the speed bottleneck
seems the votecounter is no longer perfect now that im using the new spellchecker
i might just switch back to 32bit and return to pyenchant again

EDIT: no the new spellchecker is just as accurate but much faster. my problems are even scarier.
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #90 (isolation #43) » Tue Feb 11, 2020 10:36 pm

Post by Psyche »

Down the Line: Read Extraction

So the Great VCA is of course on the horizon, but we all know that people's votes are only a small portion of the information people produce in a typical game, and of the basis for most people's reads. Understanding Mafia requires engaging with the information in people's *posts*, but it's hard (just engaging fully w/ the posts in one game is a big effort!). People have historically managed this challenge by either focusing their analyses on specific cues/situations where manual coding/interpretation of each case is feasible, or by emphasizing global textual features like comparative post or wordcount while avoiding deep consideration of the content in players posts. Projects that train machine learning classifiers over large corpuses of gameplay text to discriminate alignment seems to be the most sophisticated-imaginable examples of this latter category of work.

To take content seriously without constraining research scale, I want to try leveraging current state-of-the-art tools for extracting structured, machine-readable representations of the information in text. NLP folk call this Open information extraction; tools scan over arbitrary sentences (a simple example might be "Obama was born in Hawaii") and using grammar rules extracts simple subject-object-relation propositions (like ['Obama', 'was born in', 'Hawaii']) much more amenable to automated analysis.

A natural extension of my VCA project that could leverage this kind of tool might be Read Extraction. Votes are themselves a good window into each player's beliefs and status at a given moment in a given Day, and in some contexts they might even be the best window. But they're a limited window - at their best they only represent a player's
biggest
scumread but more often they occur as part of a negotiation between perceived "viable" Day outcomes. Pairing our vote dataset with a reads dataset could enable analysis of how people [pretend to] form and act on their beliefs throughout mafia games that's far more extensive and robust than what might be achieved from studying votes alone.

Skeleton

How do we do that, though? Here's a broad skeleton for a potential Read Extraction tool:
  • Player Identification
    . We need to reliably discern when someone's talking about another player. My VoteCounter already does this to infer the targets of people's votes when the target's exact username isn't mentioned. It already successfully negotiates the ambiguity in abbreviations, acronyms, misspellings and other issues. We'll need more than this - for example, we'll need to tell when players are referenced w/ quotes or replies, and will also need to apply coreference resolution tools to infer who people are talking about when they use pronouns like "He".
  • Claim extraction
    . We'll leverage the best available OpenIE pipeline for extracting people's statements abut identified players and converting them into simple, machine-readable representations.
  • Read inference
    . Finally, we'll have to infer from a person's claims about a player what their professed read about that player is. There are a lot of ways to go about doing this that each have their own precision/recall tradeoffs. Sentiment analysis won't be sufficient - people can have negative attitudes about townreads, and positive attitudes about scumreads. We do have the option to focus on explicit read announcements ("Psyche is scum!"), but we'd miss a lot of reads. It's the toughest part of all this.
No part of this pipeline is likely to achieve perfect performance: players can be identified w/ terms that have nothing to do with their usernames nor that can be inferred from surrounding text. Even the best OpenIE tool I'm familiar with has a substantial error rate. Even with perfect OpenIE, sentences expressing claims aren't even the only way people can convey reads: for example, people very often just post ordered lists. And there are countless ways to call someone scummy, and many of these ways are indistinguishable from other kinds of negative opinions - or even jokes/compliments.

I think I would prospectively focus first on extracting and building a dataset of machine-readable representations of as-close-as-possible-to-*all* mentions of and/or claims about players by players throughout every game I can get good voting data for. That's a substantive problem and in and of itself, and the dataset would be interesting in its own right too.

From there, we explore the challenge of classifying extracted claims, or maybe we choose a unique classification scheme to suit each of whichever research questions we decide are interesting.

Validation

The thing that's made this votecount dataset effort viable was finding a way to get quick feedback on design/implementation decisions. From there I could just incrementally improve my pipeline's success rate, making the project far more manageable. Is something like that possible in this domain?

This is where focusing on a close extension of my VCA project might be relevant. We expect votes to be a partial extension of a person's reads, so we can try to validate a read extraction pipeline against a player's votes. Using our existing vote dataset, we'll track when a vote seems to conflict or occur in accordance w/ the voter's detected attitude about the target. When the time comes to evaluate performance, we can produce discordance and concordance rates, or inspect particular examples of discordance to inform further development. We can potentially exclude votes that occur near the end of a game's Day to avoid the intrusion of social/viability concerns to sharpen the analysis.

There's basically zero possibility of a 100% concordance rate (people don't only vote their professed top scumreads!), and one wouldn't even 100% confirm the quality of the pipeline. But it's at least a strategy for highlighting potential issues in my pipeline and driving improvements.

Final Note

The most substantive challenge I've seen to efforts like the Great VCA is that it's fruitless if one doesn't take the
context
of votes into account. Supplementing our votes dataset with a reads dataset like the one I'm proposing seems a serious way to start doing exactly that. There's definitely other information in people's posts beyond indications of their attitudes about everyone else, but we're starting somewhere.
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #91 (isolation #44) » Wed Feb 12, 2020 8:38 pm

Post by Psyche »

ive never been so proud of a post guys :(
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10048
Joined: April 28, 2011
Pronoun: he/they

Post Post #95 (isolation #45) » Sun Feb 16, 2020 12:05 pm

Post by Psyche »

votecounter now at peak performance on D1
there are about 5 games over my initial 300 game sample that it just gets wrong (ugh!), but that's still a >98% success rate.
i have to see if that kind of performance extends to other days and other games, too, though

what i'll ultimately do is filter out games i can't get good results on according to my validation scheme
still, fixing as many of the errors i catch as possible is important for minimizing the errors i don't catch

big remaining to-dos:
- take power role interactions into account when processing votes and predicting lynches (especially doublevoters and dayvigs and N0s)
- extend votecountertest to predict phase transitions and test the dataset beyond D1
- add more games beyond my original 300, probably focusing on newer games and everything bob collected. god i wonder where tamuz's newbie dataset is. i know it's somewhere...
- fix all the problems in my dataset or votecounter uncovered from the above expansions
- save and share the whole dataset
- first gamut of analyses

and i gotta do it all before month's end
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
Locked

Return to “Mafia Discussion”