Beta testing mafia games?

[Sméagol]

Has this been done before?

Are mods interested in this?
Are players interested in this?

I have ideas about this, but I want some initial responses first.
In short: if you have a complicated set-up, you run a trial for say, 3 days, simulating a complete day- and night cycle, in order to get results which are better than random, to check if there are unwanted effects you may have overlooked. Beta testers are obviously encouraged to break your game.

I don't have a need for it currently, but can imagine wanting to do beta tests in the future for some of the ideas I have.

[Ircher]

Eh, I don't think this works better than creating a program that calculates rpEV. Speaking ofvwhich, I did create that goes through all the mountainous setups up to 20 townies.

[Sméagol]

Obviously, I'm not talking about mountainous set-ups..

[AlwaysInnocent]

Seems like a good idea, but you would need to find a dedicated group of people to test your game, and then another group to actually play your game. Twice as much people (unless people can and want to be in both groups).

[Alchemist21]

This would work for open and probably semi-open setups, but it would spoil closed setups.

[hiplop]

I'm assumig beta testers wouldnt be able to participate in the main game?

cool idea but IDK if we have a big enough playerbase to do this effectively

[Sméagol]

That's indeed one of the potential problems. Open or semi-opens are fine, but with closed set-ups you'll need a dedicated beta test team that doesn't have any interest in playing the game for real. I have no idea how large the playerbase on this site currently is, I get the feeling it's dwindling? That said, since people are not expected to play with maximum effort, and probably just one complete day- and nightcycle (depending on your needs), it's possible you may tap in the group of lurkers, ex-players, or other people who are on this site for some reason other than playing mafia games. This includes me for example.. I'm not sure how likely I am to join a game soon, maybe when the Blitz queue is back up. But Beta testing sounds like fun, I'm definitely up for that.

[Alchemist21]

I'm assumig beta testers wouldnt be able to participate in the main game?

cool idea but IDK if we have a big enough playerbase to do this effectively

That's indeed one of the potential problems. Open or semi-opens are fine, but with closed set-ups you'll need a dedicated beta test team that doesn't have any interest in playing the game for real. I have no idea how large the playerbase on this site currently is, I get the feeling it's dwindling? That said, since people are not expected to play with maximum effort, and probably just one complete day- and nightcycle (depending on your needs), it's possible you may tap in the group of lurkers, ex-players, or other people who are on this site for some reason other than playing mafia games. This includes me for example.. I'm not sure how likely I am to join a game soon, maybe when the Blitz queue is back up. But Beta testing sounds like fun, I'm definitely up for that.

I wasn't really thinking of the beta testers joining the main game as much as I was thinking the main game players could just look at the beta thread and see what was revealed. You could play the whole thing in a PT I guess, but you'd probably need special permission and I don't know if it's something they would allow on a regular basis.

[zMuffinMan]

Obviously, I'm not talking about mountainous set-ups..

you can still write programs for more complicated setups, it's just the more roles where there are grey areas with regards to decisions, the more complex it gets

e.g. cops, ICs, etc are relatively easy to account for (assuming optimal play and factoring in an average of X claims a day based on the size of the game where X is some number you'd need to figure out by grinding sample data), non-compulsary vigs on the other hand are an example of a role that is not so easy to account for... and whether or not things like day-talk, neighbourhoods, etc have much of an effect on win cons and (if so) how much of an effect also isn't so easy to account for

this is something i keep saying i'll write a program for one day but never end up having the time for (or wanting to spend the time i have doing it). i imagine i could write one for normal setups without too much difficulty

[Ircher]

Obviously, I'm not talking about mountainous set-ups..

you can still write programs for more complicated setups, it's just the more roles where there are grey areas with regards to decisions, the more complex it gets

e.g. cops, ICs, etc are relatively easy to account for (assuming optimal play and factoring in an average of X claims a day based on the size of the game where X is some number you'd need to figure out by grinding sample data), non-compulsary vigs on the other hand are an example of a role that is not so easy to account for... and whether or not things like day-talk, neighbourhoods, etc have much of an effect on win cons and (if so) how much of an effect also isn't so easy to account for

this is something i keep saying i'll write a program for one day but never end up having the time for (or wanting to spend the time i have doing it). i imagine i could write one for normal setups without too much difficulty

Yes, this was I meant. Also, checking out the "EV Project" page on the wiki may be a good starting point too. I just don't see how the beta testing would help too much.

[Sméagol]

[..]
you can still write programs for more complicated setups, it's just the more roles where there are grey areas with regards to decisions, the more complex it gets

e.g. cops, ICs, etc are relatively easy to account for (assuming optimal play and factoring in an average of X claims a day based on the size of the game where X is some number you'd need to figure out by grinding sample data), non-compulsary vigs on the other hand are an example of a role that is not so easy to account for... and whether or not things like day-talk, neighbourhoods, etc have much of an effect on win cons and (if so) how much of an effect also isn't so easy to account for

Problems:
1] Automatic simulations don't take into account psychology.
2] The whole point is to check for things you may have overlooked.. A script you made will obviously miss the things you have overlooked. A reviewer can help with this, but if it's a complicated role madness game, it's very possible you both overlook things.
3] A script won't try to break your game, which is exactly what you'd be asking of beta testers.
4] Plus this:

this is something i keep saying i'll write a program for one day but never end up having the time for (or wanting to spend the time i have doing it). i imagine i could write one for normal setups without too much difficulty

I'm a lazy motherfucker. If you build a script with the logic for ability interactions.. Then add a forum and voting capabilities, and you have an automated game. I actually came pretty far with a forum with voting capabilities, but never finished it. But that aside. Coding a script that takes into account ability interactions, and adds some rudimentary AI, is a lot harder than asking players to have a quick go at your set-up.

[..]]

Yes, this was I meant. Also, checking out the "EV Project" page on the wiki may be a good starting point too. I just don't see how the beta testing would help too much.

I did check it on the wiki, didn't see anything useful for my potential future needs.

Would probably be interesting to beta test the beta testing. Think I'll just go do it when I have a set-up that could really make use of it, and see if there's any interest in it by then.

[Varsoon]

I've just been testing iterations of mechanics and developing them across the games I mod.
For instance, I wanted to incorporate nuanced neighborhood mechanics in my games, so I made a public mechanic in Steven Universe Mafia that players to mutually neighborize each other and get a role-based power for doing so. That was wildly successful and various iterations of this mechanic have been seen in other games I've run.
With enough moderation experience, there's no need to run a 'Beta' setup. Just run the setup.
Yes, if the mechanic or role distribution doesn't work out, then perhaps things went poorly, but I've not had that issue and I've modded a fair share of large, complex, closed setups.

Furthermore, you shouldn't need to 'Beta Test' smaller setups, especially Micros, because they have such limited interactions that every possible mechanical iteration of the setup can be easily parsed out by a single moderator.

Nevertheless, I have had a good deal of setup anxiety before submitting any game I've run.
The real issue that Beta Testing wouldn't even really account for are the different interations of the game state that you'd be in at late game.
For instance, in my recent SaGa Frontier game, late game could have consisted of a townie who could kill upwards of 8 other players in a single day/night phase while having seven other mixed town/scum that all had kill immunity. That never came to pass, but a single beta test would be very unlikely to lead to that scenario either.

Ultimately, I can understand the urge to 'test' a game, but the aspects of games that need to be tested are often mechanical game scenarios that occur in mid/late game and would require multiple tests that wouldn't even offer truly conclusive data.

Design your setup, balance it to the best of your ability, have an experienced reviewer check it, run every iteration and role interaction through both your heads, then ship it. If something breaks, that's not a bug; it's a feature.

[callforjudgement]

Varsoon's post reminds me of Kill Everyone Mafia. I ran it because it wasn't mathematically broken and I thought it seemed like fun. The game actually worked out, as it happens, although it

should

have broken horribly after the day 1 lynch (I hadn't taken the possibility of town lynching scum into account, I assumed they'd no lynch or lynch a townie). It gave me enough evidence that it's probably not worth running again, but I don't think I regret running it, and if it had been beta-tested (and we found the problems) it probably wouldn't have run at all.

[Varsoon]

Yeah. I think that, after conceiving of every possible interaction that could happen in the game, even if you do have some unexpected results, it's still okay.
I've run games that could've ended in total train wrecks, but they played out to be very fun games for a majority of the players involved. I think that's just the risk that you take when designing a complex setup, and all you can really do is balance as well as possible for it. Players should probably understand that risk when signing up, but games should be run regardless of risk, because games are fun.

[Sméagol]

Improv skills is no doubt another useful skill for a mod to have.. I've had some unforseen situations of my own off-site.