Large Theme Survey Results (up to 9/13)

[zoraster]

One of my missions as the Large Theme Listmod is to try to get more successful games. In order to do that, I've been surveying players after each Large Theme game has finished with a number of questions. You may have received one or more of these PMs asking if you'd take a moment to fill them out. The primary questions of interest to me ask players how strongly they agree with the following statements, 1-10 with 10 being the highest.

Spoiler: Statements

The game was fun
The moderator performed well overall
The moderator did a good job keeping up to date with vote counts
The moderator dealt with interpersonal problems well
The flavor was well done
The setup was balanced
The mechanic was great
I would play a game by this moderator again

I shortened these to: game was fun, mod overall, vote counts, problems, flavor, balance, mechanic, and play in another.

I have received about a 70% response rate last I counted. I really appreciate this, and I hope you all will continue to give such valuable feedback.

Results So Far

Right now I have 12 games in the bank, with a roughly 70% response rate.

Raw Totals

As you can see, balance is the lowest metric as well as the metric that shows the most deviation (1.26 std deviation for those that care).

Keep in mind that just because someone answered that they thought balance was a problem doesn't make it so. There are really too few games to make any definite judgments based on the correlation between factors, but as it stands the mafia winning and players saying that the balance was good is strongly correlated. This is at first surprising (you'd think the more people who lose, the worst the balance gets rated), but it's not a stretch to say our players and mods tend to think games are more town-sided than they are.

Other Correlations

Some of the strong correlation between factors are:

Play in Another-Mod Overall. Hardly a surprise. These two are two sides of the same coin.
Balance-Play In Another
Vote Counts-Mod Overall. This is like the quintessential "are you modding well" test. Interestingly, play in another and vote counts only have a moderate correlation.

The highest correlation for "Fun" is actually the Mechanic. Beyond that, most are only slight or moderately correlated with fun.

Again, there have only been a dozen or so games played and rated so far, so these numbers will likely change.

Qualitative Analysis

As you may have seen, I don't just ask for people to rate games. I ask them for their comments too. This has been

extremely

helpful in getting a handle on what player common complaints are and what they wish we, as mods, would do better. A sampling:

1.

Vote counts.

Even some highly rated games have come in with these complaints. We've also had complaints from a few people that feel mods are waiting too long to edit in a vote count into their posts.

2.

Taking too long to post day and night scenes

. This one I've seen in a number of games. I think everyone understands it can take a while to put together a good scene when you're dealing with flavor, but it's important to know that excited players can become less excited when they have to wait a day (or in some cases as many as 3) after the supposed deadline to get a flip.

Mod Ratings

For moderators of Large Theme Games

Moderators may PM me to get a summary report (with letter grades) by category of what their players said. This will only be used in aggregate, and confidentiality will be maintained. These letter grades are roughly curved with a B as average.

Please keep in mind that almost without exception I think that moderators are performing to a "good enough to mod" standard. But I think we can all strive to do better with what we do. So a sub-average rating does not necessarily mean I or anyone else thinks you're a failure (in fact, even games that are below average still have more people agreeing with the "I'd play in another" statement than not). So take the reports in the manner they're intended: to give guidance on how to improve your players' experience next time.

Also, keep in mind that

this is completely optional.

You are not required to see your results as a moderator, and I will not think less of you if you do not do so.

Conclusion

Overall, mods are doing pretty well. The marks are relatively high on an absolute basis. We could do better, of course, but we're not doing badly at all.

PLEASE KEEP IN MIND THAT ALL RESPONSES ARE CONFIDENTIAL. I WILL ONLY USE THE AGGREGATE RESULTS OF SURVEYS.

[zoraster]

complaints on vote counts varied by games. Most of the comments there were simply "more vote counts" and the like. I don't think people mind the odd missed vote count.

Speaking as a player, the only time it really bothers me is when several times in a single game I feel like the mod has simply forgotten about us and I simply don't know what the vote count is (so with 3-5 players left vote counts aren't as important to me). I think what happens to some mods is that they get a little behind on vote counts and so it becomes a lot more work (updating two pages of votes is not hard. updating 10 takes a while), and so they put it off until they have more time, which leads to even more work, etc.

[zoraster]

Well... This is pretty public already. If you mean am I going to release individual game mod'a results to the public, the answer is no. At most ill do a top 3 or 5 or something when I have enough that not being in the top 3 or 5 isn't seen as a slight. The point is to improve mod behavior, not to be a source of drama and gossip.

[zoraster]

I wouldn't object if mods, especially flavor heavy mods, moved to a system where night lasts for X amount of time, but actions have to be in at X-24 hours. The problem is the feeling of being forgotten almost. in other words, if you say the night is ending on Monday at 20:00 people are going to start waiting for you to post beginning at 20:00. If you post on Tuesday at 20:00, people are going to feel like the mod has taken 24 hours longer than he or she should.

That said, we have had certain instances of people opening days or posting lynch scenes several days after the deadline and that's simply unacceptable.

By the way, once I have 3 or so more games I'll update this. The chart should automatically update.

[zoraster]

which is? I don't think saying "Night Actions must be in by Monday at 8am, Day 3 will begin at 8pm" is bad. I think the problem is if you just say "Night actions in by 8am" everyone assumes you'll post your night scene pretty shortly thereafter.

[zoraster]

How do you NOT do deadlines on the hour?

[zoraster]

1) telling people your thread will open 12 hours "late" is likely to make you look like a bad mod. I start bugging my power roles if they haven't sent me something by ~18 hours before deadline, but they have that time if they need it.

2) Half a day for "mod actions" just kills momentum. If you won't be on in 72 hours because it will be the middle of your work day, then push deadline up (or back) to adjust for it.

3) If your flavor is going to have zero impact on the game, then do your flips on time and edit in the flavor later. But it still shouldn't take more than an hour, once you get home.

1. I don't think it will. Most people understand it takes some time to write flavor (note: I don't usually write flavor, so this isn't something for me). I think people would appreciate that the mod has a deadline too.

2. I don't think this makes any sense whatsoever. Momentum isn't gained or lost by when you say that night actions can be in by. It's by how long the night lasts overall. There is no difference for momentum between saying "Night actions must be in in 60 hours from now, day will begin in 72 hours from now" and "Night actions and day start will begin in 72 hours."

3. I'm fine with this, but I understand that some people feel that flavor plays an integral role to the enjoyment of a game. How many people are really going to go back and read an updated night scene even if they'd otherwise enjoy it? Not many. It doesn't provide the same "joint" experience as posting good flavor at the instant time. If I ever run a flavor game, it sure as hell won't have flavor edited in after the fact because the only time I'd run a true flavored game is if I want other people to read it and think it actually lends something to the game.

On the day? So you'll have an area of about 12 hours where you can open the game (between 2.5 and 3.5 days, for example). That's a promise I am confident I can keep at any time, I should be online somewhere in those 12 hours.

Sometimes you can play forward a little better and find out that on a certain evening you'll be available, etc. But that's not a general thing.

So sometimes if I submit my night action you'll accept it and other times I may just be shit out of luck? I'm not sure how you're using deadline here.

[zoraster]

I mean that's how YOU write flavor, but I think this is one of those cases where there are a variety of ways to end up with a great end product. There are people who like to write as a team, for example, and that requires back and forth. There are people that will want to incorporate other stuff. There may be people who want to use flavor as a game relevant (sounds challenging, but I'm sure there's a way to do it).

Shortening the deadline for players seems perfectly fine. 72 hours is already more than I give my players unless the deadline would fall on a weekend.

One of the problems as I see it is that the only people who need more than 24 hours to decide their action are usually the mafia. And the mafia are the ones who almost always make a kill, which is what works its way into most night scenes. As that comes at the end of the deadline, that makes it harder to pre-write flavor.

What I

don't

like seeing is mods that get too wrapped up in perfecting flavor that they don't even really work on it for a long time. That makes the game drag on.

[zoraster]

Just as an update, as of the newest survey, I'm adding a question about prods and replacements. While Vote Counts may be a little more universally desired, I think players opinions on replacements and prods are important too. The modscore has been adjusted for that. Games before Patator Mafia will not be scored on this survey result.

I'm up to 26 games in the system, so I think things are looking pretty good. Over time, the mod scores have gradually increased. I don't know how much of this has to do with mods trying harder to meet the metrics provided by the survey or some other factor (such as having better mods moderate in the summer, or even less discerning players play in the summer).

One thing I can say anecdotally is that veteran players are much harsher on their mods than newer players. So on a game-to-game basis I expect games that feature well known players to also feature lower scores on the whole.



The Blue line is the mod score average of the first 24 games as time went on, starting with the 6th game (so the first reading is the average of the first 5 games, the sixth point is the average of the first six games, etc). Note this doesn't include a few games that haven't finished their survey results. As you can see, there's a pretty clear trend line up. There are certainly dips in that average, but overall it seems steady up. We'll see if that continues. My guess is that at a certain point we'll reach a plateau, which is okay as long as that plateau is consistently excellent.

The result that mods get when I send them their "report card" are letter grades for each category, with a B being roughly average for that category, and an A being one standard deviation above with a C being one standard deviation below. Right now our average is pretty darn good, so a B average is a well run game. Anything C+ and above for a mod score was acceptable in my eyes. A C and below generally means I make an effort to go through the feedback with the mod and see if we can't identify what kind of problems they're having.

For what it's worth, Borkjerfkin currently has the best mod score. Faraday/Mina, Andrius, Nexus and Vi all are the leaders in at least one category. I don't read too much into this as the upper levels are almost indistinguishable from each other, but certainly these mods are doing at least something right.

[zoraster]

i want to correct myself on the mods listed above. I'm not sure why I read that list the way I did, but it should read:

For what it's worth, Borkjerfkin currently has the best mod score as well as the highest score in two other categories. Nexus, hitogoroshi, Andrius and Vi all are the leaders in at least one category. I don't read too much into this as the upper levels are almost indistinguishable from each other, but certainly these mods are doing at least something right.

I accidentally included Faraday/Mina, who have produced two well rated games, but who are not the leaders in a category.