Mini Normal Stats Update

Zachrulez · Post Post #100 (ISO) » Tue Jul 26, 2016 4:27 am

In post 99, callforjudgement wrote:Last time I was in a Normal where two people claimed Cop, they were both scum.

That sounds like fun.

mhsmith0 · Post Post #101 (ISO) » Tue Oct 18, 2016 7:40 am

One setup I think was scum sided:
http://forum.mafiascum.net/viewtopic.php?f=2&t=68243

Town:
2-shot vig
Tracker
2-shot bodyguard
3-shot gunsmith
Ascetic

Scum:
Goon
Doctor
Vanilla cop

IMO the scum power structurally acted against every single edge town had, actively nerfing vig AND gunsmith AND tracker (2 potential false innocents), and giving scum ability to narrow down who was or wasn't PR, and giving doc/vanilla cop believable safe claims if needed (since they were true). Only real town edge was no day talk.

callforjudgement · Post Post #102 (ISO) » Tue Oct 18, 2016 9:28 am

We've been arguing that in the setup thread. Everyone agrees that the scum roles were powerful, but there's a lot of debate about how powerful the town roles are. (For the record, I was one of the reviewers, and thought that town might be

slightly

weak but couldn't be strengthened without making the setup dangerously close to being broken by early claims. I also thought that the setup was easily balanced enough to run.)

It should be noted that in the game itself, town would probably have won trivially if not for the Mafia Doctor, who was pretty much the only thing holding the setup together. I think the reviewers acknowledged that a dead Doctor early would leave scum with very little chance because the town had so much power.

mhsmith0 · Post Post #103 (ISO) » Tue Oct 18, 2016 11:47 am

Let's say doc was vigged n2 after a 2nd vt lynch, with bugs on vt/doc, and nk on say tracker/bg. Is scum really screwed in that case? Gunsmith has a chance for a guilty, but the rest of town power is basically spent. D1/n1 doc death is brutal, but by n2 I'm thinking it's pretty survivable.

mykonian · Post Post #104 (ISO) » Tue Oct 18, 2016 12:24 pm

that does not seem a setup to get upset about.

2 shot vs 3 shot vig is ~ because you end up in an even situation (you can see some curious effects for vig winrates as we went from 12 to 13 players, related). 4 shots you aren't going to get, the game is too small, so there's tons of ways that that won't work out. Same for the gunsmith, might as well be full. On top of that there's a full tracker. BG is a decent role if the gunsmith claims, otherwise probably should stay put till claims. So 3.5 town powerroles, scum get the gf for the gunsmith (the doctor) which does double duty as making one of his partners BP. Potent, but is fragile and they get an info role. Now I might have cut the vanilla cop, but you can leave it in as well. But if that's the whole issue, seriously, is this worth posting about in a MD thread?

mhsmith0 · Post Post #105 (ISO) » Tue Oct 18, 2016 12:45 pm

Well mainly it was as an example of what I felt was a scum sided setup, as that idea came up earlier and iirc no one could give good examples.

mykonian · Post Post #106 (ISO) » Tue Oct 18, 2016 1:48 pm

You misremembered. This was what you were looking for.

In post 74, Hoopla wrote:
In post 59, mhsmith0 wrote:What are some examples of town sided normal games that have happened? My gut was that mini normal 1775 was town sided, and I remember complaints on mini normal 1782 (though I disagreed there). Any that stand out from those who know a lot on those things?
There were no setups that I saw that stood out to me as obviously town-sided. It is rare for town to get more than four PR's in mini normals, and when they do, they tend to feature a lot of weak roles likes neighbours, backup whatevers, 1-Shot-whatevers that don't really do anything and often can't be confirmed, and/or scum getting heaps of power.

An underrated component of what makes a town PR powerful is that when massclaim happens, town can roughly assess how much power they should have and can occasionally confirm (or semi-confirm) a player or two based on balance. Hypothetically, if a mod gives a town more town power than usual (say, 5 strong PR's), it doesn't necessarily equate with a greater town advantage as towns will often lynch these PR's on the belief they can't all be true claims. This is essiantly Stoofer's Law on PR's, but in general I find it much more difficult to create a town-sided game than a scum-sided one.

mhsmith0 · Post Post #107 (ISO) » Tue Oct 18, 2016 1:49 pm

Lolol me

RadiantCowbells · Post Post #108 (ISO) » Wed Oct 19, 2016 4:17 am

I don't think it's necessarily imbalanced but I do think it's a bad setup.

mhsmith0 · Post Post #109 (ISO) » Sun Nov 13, 2016 8:36 am

Is there a database of the mini normal setups with roles, results etc, like there is for newbie games? There's something I'm curious about but would rather not do manual work of inputting into excel if avoidable.

callforjudgement · Post Post #110 (ISO) » Sun Nov 13, 2016 11:49 am

For summary information, there's the List of Mini Normals, which has results, but no information about the setups other than faction balance (it's also slightly outdated but not by much, and only contains faction information on more recent games).

For full information, you want the Mini Normal Archives; the format probably isn't what you'd want but can likely be parsed by computer.

mhsmith0 · Post Post #111 (ISO) » Sun Nov 13, 2016 11:59 am

Yeah was hoping that parsing had already been done. If not I may do it when I'm in the mood, but figured I'd rather find out if it'd a,ready happened.

mhsmith0 · Post Post #112 (ISO) » Sun Nov 13, 2016 2:30 pm

OK I've done a bit of analysis on the mini normal games from 1401 up, ONLY looking at 10v3 games (and excluding a number of ones where the mods didn't bother to update the OP, and didn't bother to post full role sets in the game-ending post or anywhere else that was easy to find).

Overall (now through 1832):
105 games, 36 town wins, 69 scum wins, 34% town win rate (win rates will be town win rates from here on in)

Games with Mafia Traitor
Town is 4-4, 50% win rate

Results by VT count:
3 VT: 0-2
4 VT: 0-3
5 VT: 1-9 10%
6 VT 13-25 34%
7 VT 22-27, 45%
8 VT 0-3

Results by Goon count:
0 goons: 3-4, 43%
1 goon: 14-27, 36%
2 goons: 15-27, 36%
3 goons: 4-11, 29%

The basic conclusion here seems to be that (ignoring odd games with just 1-2 town PRs) there's a pretty consistent over-estimation on value of sheer number of PRs, both for town AND for mafia.

Out of curiosity, I dug into some of the results in a bit more detail:

Spoiler: 5 VT games

Without REALLY digging into the details, one notable tendency here is that there are a bunch of town roles that don't REALLY do much of anything useful (neighbor most notably, also bodyguard), and that having them around doesn't actually seem to help town in any meaningful way.

Spoiler: 0 goon games

Interestingly, in games where ALL mafia have roles of some sort (even if just neighbor or encryptor), town usually seems to have enough actual power to reasonably counter it.

Spoiler: 3 goon games

*1705 is incorrectly marked on http://forum.mafiascum.net/viewtopic.php?f=53&t=29549, it was nea, mason, mason, not nea and all VT's

There's a real variety of power at work here, but in general, even though it seems at first glance like town ought to have reasonably decent power, it's consistently not enough and often not even close to enough.

(counting full roles without modifiers only)
Games with:

Town Cop 9-13, 41%
Town Doctor 10-15, 42%
Town Jailkeeper 7-14, 34%
Town Bodyguard 3-11, 21% (this suggests this role in particular tends to be drastically overvalued)
Town Tracker 7-13, 35%
Town Roleblocker 4-3, 57%
Mafia Roleblocker 7-7, 50% (shockingly, it seems like roleblocker may actually be considered so strong that there's a relative over-compensation here)
2 masons 4-9, 31% (a theoretically strong role that in practice has not been strong)
2 or 3 town neighbors 1-5, 17% (even less useful than bodyguard)

Spoiler: mafia roleblocker games

There's a pretty wide variety of power here, but overall a .500 record suggests that, while some games have probably too much town power (doctor, cop, rolecop, JK vs RB/RC/goon) and others too little (doctor, cop, and miller or 1-shot vig vs RB/goon/goon), in general mafia roleblockers are actually part of setups that are overall pretty balanced.

mhsmith0 · Post Post #113 (ISO) » Sun Nov 13, 2016 3:16 pm

Other stats:

Town Odd night whatever: 9-6, 60%
Town Even night whatever: 6-6, 50%
Town Two-shot whatever: 6-12, 33%
Town One-shot whatever: 8-18, 31%
Town Two-shot whatever: 6-12, 33%
Mafia One-shot whatever: 6-6, 50%
Mafia Two-shot whatever: 3-4, 43%

more stuff tbd

mhsmith0 · Post Post #114 (ISO) » Sun Nov 13, 2016 3:32 pm

TLDR:

1) Non-vanilla roles seem to be generally over-valued, especially roles that don't actually do much of anything useful (town neighbor, bodyguard, 1-shot whatever).

2) 3 goon setups in particular seem to tend to attract nowhere near enough town power needed to make games tossups. One decent power (tracker, JK, watcher, doctor, vig - obviously some of these are better than others) and two mediocre powers (bodyguard, hider, vengeful, 1-shot anything) is simply not enough. Ditto one good power and just ONE mediocre power (like just a night 1 doc in 1701). Even against an all-goon team, you probably need TWO good powers out of 3-4 overall power roles in order to achieve balance.

3) If the goal is to have a balanced win rate in 10v3 mini normals, then towns need to keep getting boosted even beyond the current norms, especially when it comes to what level of power to apply against all goon teams or how heavily to value five town PR's solely for the sake of HAVING five town PR's.

ISO · Post Post #115 (ISO) » Sun Nov 13, 2016 3:49 pm

@Mhsmith
I think you are over-simplifying the problem. You have to remember that while a lot of veteran players play mini normals, I would suspect that a good proportion, possibily the majority, of mini normal players are newer players and that also affects winrates.

That said, more town power is prob needed.

mhsmith0 · Post Post #116 (ISO) » Sun Nov 13, 2016 4:07 pm

I'd definitely agree that the % of vets vs newer players has an impact on mini normal win rates. That said, I think it's interesting to see what the stats are showing in more detail, most notably where there seem to be more consistent imbalances. And it seems like there's a relatively more common imbalance wrt "roles that do something, even if the value of them is very low" being much less valuable than they seem to be given credit for. Whether this is due to the inherent design, or whether those are the kinds of roles that newer players can especially struggle with, I'm less sure.

It's also, FWIW, possible that the randomness inherent in mafia PRs tends to be more pro town in that if a newbie rolls mafia roleblocker, that's much tougher on the scum team than if, say, it's all goons and bussing can be done with much less damage done.

callforjudgement · Post Post #117 (ISO) » Sun Nov 13, 2016 4:20 pm

Bodyguard is a little weaker than 1-shot Cop, and (in most setups) stronger than Doctor. (If it stops a kill, you effectively confirmed yourself via "swapping places with" the best nightkill target, but the nightkill target doesn't become any less confirmed in the process). It's possible that I'm overestimating how good scum are at choosing nightkill targets, though, and thus overestimating Bodyguard as a result.

I suspect your sample size is too small to get really meaningful information about how much roles are being overvalued/undervalued (e.g. 1472 was won by town despite being clearly scumsided; 1590 is a very similar setup that's almost as scumsided (you can do Follow the Tracker in it to get a slight edge) but also was won by town). (There's also the issue of role interactions, e.g. one good reason to put Neighbours in a setup is to make any Rolecops in the setup more powerful by reducing their target pool, as the Neighbours tend to be exposed fairly quickly and a Rolecop rarely has reason to scan them.)

The results for 7 VT games are very interesting, mostly because 7 VT = 3 town power roles is the easiest point on the scale to balance. (It wouldn't surprise me if a major part of the reason more complex setups tend to be scumsided is that the harder a setup is to balance, the lower the bar for allowing it to run seems to be, because eventually nobody can be sure what effect a change would have.)

Finally, the perception of balance affects balance. If players believe a setup is unbalanced, that makes it more scumsided (because it causes them to draw incorrect assumptions from the lylo massclaim). This could well be why we seem to be stuck in a balance rut at 30-40%; it could be that this effect is forcing 50% games down to 40% no matter what we do. (I note that most new mods in the Mini Normal queue submit obviously scumsided setups and have to be persuaded to make them more townsided; it's likely that the site as a whole has similar opinions on balance to the new moderators.)

Anyway, I have a suggestion: we decide upon an intentionally "townsided" balance standard (along the lines of a 1-shot Cop or actiated Innocent Child more townsided than a typical setup), and allow moderators to opt into the more townsided standard in reviews, then advertise it during signups (the previous standard would also be available). After we've had a few games we can compare to see if it's a) making win rates more balanced, b) more fun. (It seems plausible that the two goals will be in conflict; I can imagine that making games unusually townsided might increase the chance of a town blowout, which tend to be unfun. Or to view things through a simpler lens, games that go through 3:2 tend to be more fun than games that go through 4:1.)

mhsmith0 · Post Post #118 (ISO) » Sun Nov 13, 2016 4:36 pm

In post 117, callforjudgement wrote:Bodyguard is a little weaker than 1-shot Cop, and (in most setups) stronger than Doctor. (If it stops a kill, you effectively confirmed yourself via "swapping places with" the best nightkill target, but the nightkill target doesn't become any less confirmed in the process). It's possible that I'm overestimating how good scum are at choosing nightkill targets, though, and thus overestimating Bodyguard as a result.

This is completely incorrect. In zero setups that I have ever seen is bodyguard stronger than doc. Moreover, doc has the potential to outright stop a kill, which can push a game from evens to odds and add a mislynch that town can afford which is a HUGE advantage.

Bodyguard is like the epitome of a trash role that is over-valued in terms of game design, and town's horrific win rate in games with bodyguards largely confirms this. Bodyguard MIGHT be ok in a game with a full cop or a vig or a jailkeeper (really strong town roles that can continue to be useful as the game goes on), but otherwise is very close to worthless. Successfully BG'ing an IC isn't zero value, but is close to it. Successfully Doc'ing an IC is MUCH higher value for obvious reasons (it stops a night kill, someone on town might see you do it, and if it takes it evens to odds, then you just gave town a HUGE advantage).

Town roles with especially horrid win rates include:
bodyguard (3-11, 21%)
IC (2-5, 29%)
1x neighbor (2-4, 33% - neighborizer may be mixed up in this fwiw)
2 or 3x neighbor (1-5, 17%)

and a few more are pushing it:
2x masons (4-9, 31%)
tracker (7-13, 37%)
jailkeeper (7-14, 33%)

mhsmith0 · Post Post #119 (ISO) » Sun Nov 13, 2016 4:59 pm

In post 117, callforjudgement wrote:
I suspect your sample size is too small to get really meaningful information about how much roles are being overvalued/undervalued (e.g. 1472 was won by town despite being clearly scumsided; 1590 is a very similar setup that's almost as scumsided (you can do Follow the Tracker in it to get a slight edge) but also was won by town). (There's also the issue of role interactions, e.g. one good reason to put Neighbours in a setup is to make any Rolecops in the setup more powerful by reducing their target pool, as the Neighbours tend to be exposed fairly quickly and a Rolecop rarely has reason to scan them.)

I'd agree that sample size is limited. However, it's enough to draw some useful conclusions. P-value of the 1-goon, and 2-goon sets are each around 5% independently, and 3-goon data is more like 10% (presuming a base null set probability of 50% win rate), which for that sample size is remarkable.

Similarly, the 5 VT and 6 VT data sets are both well under 10% p-value, which is again remarkable. It's pretty clear that, while individual setups have varying success rates, as a whole, high #s of town PR's (the crappy ones anyway) are consistently overvalued if the goal is 50% win rate. That's a pretty meaningful conclusion when it comes to game design/review IMO.

Neighbors in particular seem to be put into games as a "well that's a town PR that adds to the list" sort of thing, even though scum neighbors are clearly a thing, and there's no particular indication that towns are especially good at sorting out people inside a neighborhood based on the data results.

The results for 7 VT games are very interesting, mostly because 7 VT = 3 town power roles is the easiest point on the scale to balance. (It wouldn't surprise me if a major part of the reason more complex setups tend to be scumsided is that the harder a setup is to balance, the lower the bar for allowing it to run seems to be, because eventually nobody can be sure what effect a change would have.)

Yeah I think it's good that more straightforward setups are by and large being reasonably balanced (7 VT vs 3 goons is 4-6, 7 vs 2 is 9-13, 6v3 is 0-2, 7v1 is 8-8, 6v2 is 5-10, 5v3 is 0-0). Probably means that more focus needs to be put into the more complex ones, since that's where the biggest outlier scum win rates seem to be.

Finally, the perception of balance affects balance. If players believe a setup is unbalanced, that makes it more scumsided (because it causes them to draw incorrect assumptions from the lylo massclaim). This could well be why we seem to be stuck in a balance rut at 30-40%; it could be that this effect is forcing 50% games down to 40% no matter what we do. (I note that most new mods in the Mini Normal queue submit obviously scumsided setups and have to be persuaded to make them more townsided; it's likely that the site as a whole has similar opinions on balance to the new moderators.)

Anyway, I have a suggestion: we decide upon an intentionally "townsided" balance standard (along the lines of a 1-shot Cop or actiated Innocent Child more townsided than a typical setup), and allow moderators to opt into the more townsided standard in reviews, then advertise it during signups (the previous standard would also be available). After we've had a few games we can compare to see if it's a) making win rates more balanced, b) more fun. (It seems plausible that the two goals will be in conflict; I can imagine that making games unusually townsided might increase the chance of a town blowout, which tend to be unfun. Or to view things through a simpler lens, games that go through 3:2 tend to be more fun than games that go through 4:1.)

That might make some sense. One thing I'd like to find is a list of setups that are relatively balanced over a long term in mini normals, and then at least have a reasonable baseline to look at other setups off of that.

callforjudgement · Post Post #120 (ISO) » Mon Nov 14, 2016 9:14 am

In post 118, mhsmith0 wrote:Successfully BG'ing an IC isn't zero value, but is close to it. Successfully Doc'ing an IC is MUCH higher value for obvious reasons (it stops a night kill, someone on town might see you do it, and if it takes it evens to odds, then you just gave town a HUGE advantage).

Let's take the simplest possible example going into night: Innocent Child, Bodyguard/Doctor, VT, Goon. Assume it's a closed setup, players haven't claimed yet, and scum kill the Innocent Child. With a Bodyguard, we're looking at a guaranteed 50% EV the following night. With a Doctor, scum can claim any role that could stop the kill and doesn't directly counterclaim a Doctor (e.g. Bulletproof), and now town has only a 33% EV (assuming that scum will no-kill if town no-lynch). The Bodyguard removes themself from the lynch pool; the Doctor doesn't.

This isn't a contrived situation, either; it frequently comes up in actual games. You obviously remember this incident, given how much we argued about it. Your slot was huge lynchbait at that point in the game (admittedly, you'd only just replaced in, but I'm assuming that you wouldn't have saved the slot just by your presence). If you'd Doctored the confirmed townie and then been lynched, town would likely have been worse off than if you'd Bodyguarded them

even though

it would have taken town back to odds! (Obviously, it would have been even more townsided if town were on odds at the time and there wasn't a vig messing up the odd/even analysis.)

mhsmith0 wrote:Successfully Doc'ing an IC is MUCH higher value for obvious reasons (it stops a night kill, someone on town might see you do it, and if it takes it evens to odds, then you just gave town a HUGE advantage).

"Someone on town might see you do it" is a negative. How are you going to prove you're the doctor, rather than the

scum who tried to make the nightkill

?

mhsmith0 · Post Post #121 (ISO) » Mon Nov 14, 2016 10:38 am

A few responses:

1) That particular incident is a pretty good indication of what I'm talking about, yes. Consider the counter-factual where I'd actually doc saved that night. Now you have 6v1 (instead of 5v1) and town has bought itself an extra mislynch. I'd have made myself very nearly clear from the save and claim, which means that, despite the suspicion the slot was under, it would have become extremely difficult ot mislynch me, from the night action alone.
EVEN IF I had still been mislynched (which again, is pretty dubious given the fact that scum would virtually never pull a no kill gambit in that situation), town would STILL be better off than it was given my BG use. Why? Because you have another day of voting data that can be used to analyze. Scum-hunting primarily comes from catching people making in-thread manipulations, and mislynching a declared doctor with a near-proven night kill save is the sort of thing that needs to be worked hard on to achieve, which then makes it likelier that someone will through that effort become more obviously town or mafia.

2) Using your example: IC, BG/Doc, VT, Goon (presumably going into night), you are correct in that specific instance that it's basically even value, since with a doc you have a no kill loop that only really resolves itself when the doc gets shot the next night. Your example of a scum who fake claims some other kill stopping role, though, gets in trouble here because now you have essentially cleared the VT claim (obviously not 100% since it's POSSIBLE there are two actual kill-stopping roles), and now it's head to head (doc vs fake claiming goon), with two town clears slightly able to better sort the claims since they can work together to solve the game. So either there's a no lynch loop that eventually ends with the goon shooting the doc (or maybe a draw depending on rule set), or you're able to narrow down the field a bit. Either way it's at least a bit more helpful than just having the BG save the IC at night.

3) Perhaps even more importantly, despite the clarity of the real world instance in which doc was massively and obviously more useful than BG, the stats bear it out. Town bodyguard is just about the single lowest win rate town PR out there in mini normals (2-shot vig is worse, and miller/JOAT are around same level, but that's really it), with a pretty reasonable sample size of 14 10v3 games. Town bodyguard is actively useful in limited examples where you can save a town PR with remaining shots (such as cop tracker or vig), but in general it's barely helpful, and in just about zero cases is it ever MORE helpful than doc.

4) "Someone on town might see you do it" is a POSITIVE, at least before LYLO. Someone sees you visit someone, and there is no kill. If pressed on the matter (which usually is inappropriate, fwiw, as most non-killing visitors are town PRs), you claim doctor, you are not counter-claimed, and you are now confirmed doctor (or maybe people might tinfoil you as some kind of visiting scum PR like role cop, but at least you're confirmed to have not taken a shot).

mhsmith0 · Post Post #122 (ISO) » Mon Nov 14, 2016 10:40 am

PS here are some more detailed stats about various town/scum roles that pop up in games, and the resulting town W/L rates (also p-values!). Note that these don't exactly correspond to what I'd shown earlier, since I've made a few spreadsheet corrections (naming conventions haven't always been consistent, so there are manual corrections going on)

Spoiler: town PR win stats

Gamma Emerald · Post Post #123 (ISO) » Mon Nov 14, 2016 6:37 pm

In post 117, callforjudgement wrote:Bodyguard is a little weaker than 1-shot Cop, and (in most setups) stronger than Doctor. (If it stops a kill, you effectively confirmed yourself via "swapping places with" the best nightkill target, but the nightkill target doesn't become any less confirmed in the process). It's possible that I'm overestimating how good scum are at choosing nightkill targets, though, and thus overestimating Bodyguard as a result.

I suspect your sample size is too small to get really meaningful information about how much roles are being overvalued/undervalued (e.g. 1472 was won by town despite being clearly scumsided; 1590 is a very similar setup that's almost as scumsided (you can do Follow the Tracker in it to get a slight edge) but also was won by town). (There's also the issue of role interactions, e.g. one good reason to put Neighbours in a setup is to make any Rolecops in the setup more powerful by reducing their target pool, as the Neighbours tend to be exposed fairly quickly and a Rolecop rarely has reason to scan them.)

The results for 7 VT games are very interesting, mostly because 7 VT = 3 town power roles is the easiest point on the scale to balance. (It wouldn't surprise me if a major part of the reason more complex setups tend to be scumsided is that the harder a setup is to balance, the lower the bar for allowing it to run seems to be, because eventually nobody can be sure what effect a change would have.)

Finally, the perception of balance affects balance. If players believe a setup is unbalanced, that makes it more scumsided (because it causes them to draw incorrect assumptions from the lylo massclaim). This could well be why we seem to be stuck in a balance rut at 30-40%; it could be that this effect is forcing 50% games down to 40% no matter what we do. (I note that most new mods in the Mini Normal queue submit obviously scumsided setups and have to be persuaded to make them more townsided; it's likely that the site as a whole has similar opinions on balance to the new moderators.)

Anyway, I have a suggestion: we decide upon an intentionally "townsided" balance standard (along the lines of a 1-shot Cop or actiated Innocent Child more townsided than a typical setup), and allow moderators to opt into the more townsided standard in reviews, then advertise it during signups (the previous standard would also be available). After we've had a few games we can compare to see if it's a) making win rates more balanced, b) more fun. (It seems plausible that the two goals will be in conflict; I can imagine that making games unusually townsided might increase the chance of a town blowout, which tend to be unfun. Or to view things through a simpler lens, games that go through 3:2 tend to be more fun than games that go through 4:1.)

Oh dear
Do we to drag House here to set you straight