Computational Mafia

This forum is for discussion related to the game.
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10056
Joined: April 28, 2011
Pronoun: he/they

Post Post #25 (ISO) » Mon Mar 04, 2019 11:38 am

Post by Psyche »

oh i remember somehow loading 200 post pages long ago but forgot how i did that

why'd you put the scraper in js, btw?
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
MathBlade
MathBlade
He/Him
Technical Support
User avatar
User avatar
MathBlade
He/Him
Technical Support
Technical Support
Posts: 42761
Joined: September 9, 2013
Pronoun: He/Him
Location: Western US

Post Post #26 (ISO) » Mon Mar 04, 2019 2:48 pm

Post by MathBlade »

In post 24, Bicephalous Bob wrote:
In post 14, MathBlade wrote:
In post 9, yessiree wrote:the main issue with trying to automate anything on mafiascum is the lack of an API. (it's in development?) It's FeelsBadMan to work with <document> response types, which means an addition layer of reading/parsing XML/HTML content which makes the whole process more time-consuming and error prone

It would be much much better if you could do something like query "[b]API[/b]/viewtopic.php?f=5&t=78755" and get back JSON instead
Omg so much this.

Like that’s the reason my VoteCount scrubber is slower than I would like.

Getting all that html and crunching it for hundreds of pages is a pita.
The average game should be around 15 light-weight pages if you use &view=print&ppp=200, of which only the last three pages are probably relevant for a vc

A twenty-line artoo.js game scraper: https://bitbucket.org/bopjesvla/thesis/ ... ew-default
Lightweight is a myth when using c#

Even threading it takes a while as it has to node parse.

Plus all the features requested add more lookups.

Mainly glad I looking at doing it in php and potentially removing said bloat.
ScumBlade's eloquent performance left me utterly disoriented, debased, depraved and sent me spiraling into a horrific murky abyss with emotional turmoil and immense despair as my only companions until slowly I suffocate in my own gloom, surrounded by failure. I will never recover. -- Zachstralkita about Mini 1841
GTKAS -- MathBlade
User avatar
Bicephalous Bob
Bicephalous Bob
Mafia Scum
User avatar
User avatar
Bicephalous Bob
Mafia Scum
Mafia Scum
Posts: 3828
Joined: June 4, 2013
Location: I don't know why you're linking me to pictures of babies on Facebook

Post Post #27 (ISO) » Wed Mar 06, 2019 1:39 am

Post by Bicephalous Bob »

In post 25, Psyche wrote:oh i remember somehow loading 200 post pages long ago but forgot how i did that

why'd you put the scraper in js, btw?
I generally like scraping from a web browser because I already know the DOM API. I'd recommend puppeteer, not artoo.js, for anything more serious though.
User avatar
yessiree
yessiree
he
Mafia Scum
User avatar
User avatar
yessiree
he
Mafia Scum
Mafia Scum
Posts: 4349
Joined: June 6, 2013
Pronoun: he

Post Post #28 (ISO) » Tue Mar 19, 2019 8:48 pm

Post by yessiree »

Hi friends, I just found out that you can debug NodeJs apps from chrome dev tools now
https://medium.com/@paul_irish/debuggin ... 4a1b95ae27

makes life a lot easier, so i'll play around with it for a bit
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10056
Joined: April 28, 2011
Pronoun: he/they

Post Post #29 (ISO) » Tue Apr 16, 2019 7:12 pm

Post by Psyche »

https://github.com/iodide-project/pyodide

this is really interesting to me as someone who struggled bridging my python-based comp mafia efforts to the web without maintaining a server
i have no idea how it works but it’s totally on my to-do list to explore it
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
yessiree
yessiree
he
Mafia Scum
User avatar
User avatar
yessiree
he
Mafia Scum
Mafia Scum
Posts: 4349
Joined: June 6, 2013
Pronoun: he

Post Post #30 (ISO) » Tue Apr 16, 2019 8:31 pm

Post by yessiree »

https://github.com/blookvoodow/modbotjs

A simple pagetopper adapted from donbot, re-written in JavaScript with NodeJs (working prototype)
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10056
Joined: April 28, 2011
Pronoun: he/they

Post Post #31 (ISO) » Tue Apr 16, 2019 9:19 pm

Post by Psyche »

neato
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10056
Joined: April 28, 2011
Pronoun: he/they

Post Post #32 (ISO) » Tue Apr 16, 2019 9:45 pm

Post by Psyche »

meh pyodide is far from usable atm i gotta stop listening when people tell me to care about things
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
yessiree
yessiree
he
Mafia Scum
User avatar
User avatar
yessiree
he
Mafia Scum
Mafia Scum
Posts: 4349
Joined: June 6, 2013
Pronoun: he

Post Post #33 (ISO) » Wed Apr 17, 2019 12:20 am

Post by yessiree »

What do u mean by bring python based solutions to the Web? As in hosting it on a domain? Or just having a Web frontend
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10056
Joined: April 28, 2011
Pronoun: he/they

Post Post #34 (ISO) » Thu May 30, 2019 9:09 am

Post by Psyche »

im gonna clean up that day transition dataset soon and start crunching numbers
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
MaryJoLisa
MaryJoLisa
Mafia Scum
User avatar
User avatar
MaryJoLisa
Mafia Scum
Mafia Scum
Posts: 1763
Joined: January 30, 2019

Post Post #35 (ISO) » Fri Jun 07, 2019 1:37 pm

Post by MaryJoLisa »

How are people collecting data from the games?
The Oxford comma is only technically optional.
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10056
Joined: April 28, 2011
Pronoun: he/they

Post Post #36 (ISO) » Fri Jun 07, 2019 1:45 pm

Post by Psyche »

mix of code mix of tedium outsourced to a nice 20-something mother who lives in the phillipines for $3 an hour
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
Psyche
Psyche
he/they
Survivor
User avatar
User avatar
Psyche
he/they
Survivor
Survivor
Posts: 10056
Joined: April 28, 2011
Pronoun: he/they

Post Post #37 (ISO) » Fri Jun 07, 2019 1:47 pm

Post by Psyche »

i know the pay isn't great but the phillipines are actually a pretty nice place a lot of people choose to live there for free
youtube playlist extracter | donbot | game scraper | vca | setupsim | strategist | llm
User avatar
MaryJoLisa
MaryJoLisa
Mafia Scum
User avatar
User avatar
MaryJoLisa
Mafia Scum
Mafia Scum
Posts: 1763
Joined: January 30, 2019

Post Post #38 (ISO) » Fri Jun 07, 2019 1:57 pm

Post by MaryJoLisa »

Is there a way to download game data?
The Oxford comma is only technically optional.
User avatar
Plotinus
Plotinus
Kitten Caboodle
User avatar
User avatar
Plotinus
Kitten Caboodle
Kitten Caboodle
Posts: 7611
Joined: March 13, 2015
Location: UTC+1
Contact:

Post Post #39 (ISO) » Sat Jun 08, 2019 8:37 am

Post by Plotinus »

I use Thesp's votecounter http://www.robandkriskris.com/votecounter/

it's reasonably fast, giving an accurate answer in a few seconds, and it's online so i don't have to run anything.


I have almost entirely automated the task of setting up a newbie game -- posting in all the private topics and randomising the roles and sending out all the role PMs and bookmarking the threads and adding my girlfriend to the dead thread as a spectator and all those little fiddly details, but it still requires human intervention a little bit -- some intentional (it asks you if everything is okay before submitting the role PMs, for example, and you have to type in your password to the browser your own self, it won't do that for you, but there's also a small race condition i'm still figuring out that sometimes requires that i manually hit the back button in the browser to get it back on track.
The failure mode of clever is asshole.

Modding checklists | Sequencer is in Game 5 | Space II is in Day 4
User avatar
Plotinus
Plotinus
Kitten Caboodle
User avatar
User avatar
Plotinus
Kitten Caboodle
Kitten Caboodle
Posts: 7611
Joined: March 13, 2015
Location: UTC+1
Contact:

Post Post #40 (ISO) » Sat Jun 08, 2019 8:39 am

Post by Plotinus »

As of now, though, it can get a game entirely set up in 10 minutes, including time spent waiting for me to google some cool things to use as flavour.
The failure mode of clever is asshole.

Modding checklists | Sequencer is in Game 5 | Space II is in Day 4
User avatar
RadiantCowbells
RadiantCowbells
He/him
Smooth Criminal
User avatar
User avatar
RadiantCowbells
He/him
Smooth Criminal
Smooth Criminal
Posts: 70855
Joined: February 24, 2013
Pronoun: He/him
Contact:

Post Post #41 (ISO) » Sat Jun 08, 2019 11:25 am

Post by RadiantCowbells »

Share >=[
2019 stats: Town WR 76.7%, overall WR 81.667%, 1 scum defeat involving a major mod error in lylo vs 8 scum wins.
User avatar
Plotinus
Plotinus
Kitten Caboodle
User avatar
User avatar
Plotinus
Kitten Caboodle
Kitten Caboodle
Posts: 7611
Joined: March 13, 2015
Location: UTC+1
Contact:

Post Post #42 (ISO) » Sat Jun 08, 2019 6:16 pm

Post by Plotinus »

After the next newbie game starts (I'm next in line), I'll know if I've fixed my race condition, if I have then I'll push my change.

setup a game (requires: selenium) : MafiaMod
also requires these files: files

It's all set up for my own defaults so you would need to:
  • Edit the files in files/public/ to format the OP how you'd like
  • Search for the word "hubris" and remove it if you dont want to add my girlfriend to your deadthread. (This step will be easier in the next version, you'll be able to do --spectators = "person1 person2 person3")
  • If you're not modding a newbie game, put your role PMs in the files/roles directory and add the roles names to the setups dictionary.
  • Also change this to be whatever subforum you'll be putting the game in

    Code: Select all

        if whichThread == 'public':
            url = 'https://forum.mafiascum.net/posting.php?mode=post&f=11'
If you want to increase the size of the deadline smoothly throughout the dayphase, calculate the times that you should do it at: mafiaDeadlines

When should I prod somebody next, given that time passes slower on the weekends and they might be v/la: MafiaProd

Give me a list of countdown timers to post in the mod PT of when everybody should be prodded next, if the nightphase doesn't reset countdown timers, time passes more slowly on weekends, and some people are V/LA: NightProd (has a bug that i haven't figured out yet), requires BeautifulSoup
The failure mode of clever is asshole.

Modding checklists | Sequencer is in Game 5 | Space II is in Day 4
User avatar
Plotinus
Plotinus
Kitten Caboodle
User avatar
User avatar
Plotinus
Kitten Caboodle
Kitten Caboodle
Posts: 7611
Joined: March 13, 2015
Location: UTC+1
Contact:

Post Post #43 (ISO) » Thu Jun 13, 2019 8:04 am

Post by Plotinus »

Okay, the bug is fixed! I used it to set up a game just now and there were finally no showstoppers. My script is ready for public consumption.

So here is how I would set up Game 1933, Stuff I Found Online XII, the last game of mine that finished. The setup was C2 and there were 5 specators, which for the sake of demonstration I will pretend were added at the very start of the game:

Code: Select all

python3 mafiaMod.py -u Plotinus C2 1927 "Stuff I Found Online XII" --spectators="hubris PvtUrist Muffins ZestOfLife chennisden"


It will ask me to paste in the playerlist. Now it will randomise the roles and open a new Firefox window, where it will fill in my username for me, but ask me to type my password into the form myself and hit enter in the terminal once I've succeeded.

Then it prompts me through getting the flavour ready, asking for a picture for the OP, a link, a little bit of text to go with it. You can just hit enter to skip anything you don't want to put in. I like to put youtube videos for my death scenes so it'll ask for a youtube video for the day 1 lynch and save a template for the first flip in the mod PT, but again you could just hit enter to skip it.

It pauses for about 5 seconds between posts it makes, to avoid DDOSing the site and also to avoid the "you can't make posts that quickly" error message that mafiascum's flood protection puts out.

Once it's got all the PTs set up, it'll start sending out role PMs. It won't hit the send button for you on the role PMs, instead it'll ask you "Everything okay?" and you can hit enter in the terminal if the role PM is good and then it'll send it.

Once it's sent all the role PMs, it'll post your rules into the public thread and the generic newbie setup information posts, and the mod IC post, and finally the "All role PMs are out, we'll start in 2 days" message -- it rounds the deadline up to the next half hour or so.

your mod PT will look like this
The failure mode of clever is asshole.

Modding checklists | Sequencer is in Game 5 | Space II is in Day 4
User avatar
Plotinus
Plotinus
Kitten Caboodle
User avatar
User avatar
Plotinus
Kitten Caboodle
Kitten Caboodle
Posts: 7611
Joined: March 13, 2015
Location: UTC+1
Contact:

Post Post #44 (ISO) » Thu Jun 13, 2019 8:08 am

Post by Plotinus »

it also adds the listmod as a moderator to all your PTs, and bookmarks all the threads for you. The default listmod is PenguinPower. You can change it like

Code: Select all

-l Untrod Tripod
, or if you have a backup moderator you could put their name instead.

Also the default spectator is my girlfriend so do this if you don't want that:

Code: Select all

-s ""
The failure mode of clever is asshole.

Modding checklists | Sequencer is in Game 5 | Space II is in Day 4
User avatar
MaryJoLisa
MaryJoLisa
Mafia Scum
User avatar
User avatar
MaryJoLisa
Mafia Scum
Mafia Scum
Posts: 1763
Joined: January 30, 2019

Post Post #45 (ISO) » Thu Jun 13, 2019 8:20 am

Post by MaryJoLisa »

But how do you get the game data, like the information that's relevant to the player, in some kind of raw form?
The Oxford comma is only technically optional.
User avatar
Plotinus
Plotinus
Kitten Caboodle
User avatar
User avatar
Plotinus
Kitten Caboodle
Kitten Caboodle
Posts: 7611
Joined: March 13, 2015
Location: UTC+1
Contact:

Post Post #46 (ISO) » Thu Jun 13, 2019 8:34 am

Post by Plotinus »

I'm not sure what you're asking? My program just automates the process of getting the game set up, so it reads from files and makes the posts. The listmod tells you which setup to use, and then you tell the program which setup it is. It picks who will have which role at random and then sets everything up.

The role PMs are in files like this one. It automatically replaces PUBLICTHREAD with the link to the actual thread.

It only does pregame right now, so it doesn't count votes or make sure everybody is behaving or prod people.
The failure mode of clever is asshole.

Modding checklists | Sequencer is in Game 5 | Space II is in Day 4
User avatar
MaryJoLisa
MaryJoLisa
Mafia Scum
User avatar
User avatar
MaryJoLisa
Mafia Scum
Mafia Scum
Posts: 1763
Joined: January 30, 2019

Post Post #47 (ISO) » Thu Jun 13, 2019 8:41 am

Post by MaryJoLisa »

No, I'm talking about something different. What I'd like to do is download the posts of a game I'm playing and put them into something like a spreadsheet so I can track and sort the content of the posts in a way that is more MJL-compatible.
The Oxford comma is only technically optional.
User avatar
Plotinus
Plotinus
Kitten Caboodle
User avatar
User avatar
Plotinus
Kitten Caboodle
Kitten Caboodle
Posts: 7611
Joined: March 13, 2015
Location: UTC+1
Contact:

Post Post #48 (ISO) » Thu Jun 13, 2019 8:53 am

Post by Plotinus »

oh. well if you just want the literal content of the posts, maybe one column for each player, that should be pretty easy to do. if you want to do something interest with the data then it will get harder depending what you want to do.

I'd recommend the free online book automate the boring stuff with Python. Chapters 11 and 12 will walk you through making something that downloads posts and puts them into a spreadsheet. Depending on your level of background knowledge, you might need to start at chapter 1, which explains the basics in an accessible way.

I have a program that does some calculations about prod timers, and it downloads information from the game overview page. The downloading part is literally just

Code: Select all


def parse_page(url):
    res = requests.get(url)
    res.raise_for_status()
    return BeautifulSoup(res.text, "html.parser")


Figuring out programmatically which time stamp goes with which player and whether or not they are V/LA is a little extra work, but not all that much


I think the easiest version of what you want to do is a really good first programming project. It's achievable in a reasonable amount of time and you'll learn a lot from trying.

I suspect the "sort the content in a way that is more MJL compatible" part might be harder, but it's still worth trying, and something you can refine over time.
Last edited by Plotinus on Thu Jun 13, 2019 9:01 am, edited 1 time in total.
The failure mode of clever is asshole.

Modding checklists | Sequencer is in Game 5 | Space II is in Day 4
User avatar
MaryJoLisa
MaryJoLisa
Mafia Scum
User avatar
User avatar
MaryJoLisa
Mafia Scum
Mafia Scum
Posts: 1763
Joined: January 30, 2019

Post Post #49 (ISO) » Thu Jun 13, 2019 8:56 am

Post by MaryJoLisa »

I think this is what I need, thank you! Once I get the data, I don't think I'll have any trouble sorting it. My test runs have proven effective, but it's time consuming and I knew there had to be a better way.
The Oxford comma is only technically optional.
Locked

Return to “Mafia Discussion”