Mini Theme Survey Results

This forum is for discussion related to the game.
User avatar
Equinox
Equinox
he/they
Shot Count
User avatar
User avatar
Equinox
he/they
Shot Count
Shot Count
Posts: 10105
Joined: April 12, 2010
Pronoun: he/they
Location: Los Angeles, CA

Mini Theme Survey Results

Post Post #0 (ISO) » Wed Dec 23, 2015 7:51 am

Post by Equinox »

After zoraster's success with post-game surveys in the Large Theme Queue, I decided to do the same with the Blitz and Mini Theme Queues. These surveys were intended to measure: (1) player satisfaction with moderator performance in five aspects of the game, (2) player satisfaction with other players' performances, and (3) agreement with positive statements about the game. Additional questions about players' roles were intended to fill possible gaps in the Mini Theme Archives. You can find a non-formatted version of the survey in the spoiler.

Spoiler: Survey
What is your mafiascum.net username?

For what game are you doing this survey?

How did you participate in this game?

Did you play in a hydra?

If you played in a hydra, please list your partner(s).

What was your alignment?

What was your role?

What was your character name, if any?

How satisfied were you with how the moderator handled
(1 = not at all, 10 = extremely)
  1. vote counts?
  2. prods?
  3. replacements?
  4. interpersonal problems?
  5. the game?

How much do you agree with the following statements?
(1 = strongly disagree, 10 = strongly agree)
  1. The game was fun.
  2. The flavor was well-done.
  3. The setup was balanced.
  4. The mechanic was well-done.
  5. I would play a game by this moderator again.

How well do you feel each of the following factions performed?
(1 = not at all, 10 = extremely)
  1. Town
  2. Mafia
  3. Third-party
  4. Other

How could the moderator have improved?

Do you have any complaints about any of the players?

Do you have any additional comments or concerns that were not covered in the previous questions?

In addition to the survey, metrics were calculated for the queue's speed and replacement rates. Because these were not dependent on survey responses, they are current as of yesterday.

For this report, 10 games were surveyed. Of 125 surveys that were sent, 76 responses have been received for a response rate of
60.8%
. One response was excluded for possible abuse.

Moderator Performance

(
N
= 70)

Image

"Personnel" here refers to interpersonal problems. Moderator self-ratings were excluded.

Player satisfaction was high across the board, and there didn't seem to be any significant differences between aspects. The lowest and most variable rating was in how moderators handled interpersonal problems (
M
= 8.07,
SD
= 2.25), followed by prods (
M
= 8.20,
SD
= 2.03) and vote counts (
M
= 8.26,
SD
= 1.83). Respondents were happy with how moderators handled replacements (
M
= 8.52,
SD
= 1.91), which is a good sign given the current replacement climate.

The average satisfaction rating for games overall was 8.49 (
SD
= 1.88).

Player Performance

(
N
= 76)

Image

Town and Mafia ratings would be the more reliable measure here. There was one multiball game run during this period, and respondents were instructed to mark a second mafia team as "Other." However, there was at least one game with multiple neutral parties and that made interpreting the scores for Third-Party and Other more difficult. A future report might exclude the game(s) in question for comparison's sake.

For the most part, people seemed to think that players performed little better than average, with town (
M
= 6.18,
SD
= 2.32) doing slightly worse than mafia (
M
= 6.78,
SD
= 2.29). Given that "town did so bad, etc." happens in almost every game that completes on this site, this is no surprise.

The Games Themselves

(
N
= 70)

Image

It should be noted that these are not ratings
per se
but level of agreement that these aspects were done well. The difference might or might not be significant, but testing this without losing the ability to compare between games involves a much longer survey.

That said, consistent with results in the Large Theme Queue, balance (
M
= 7.21,
SD
= 2.56) remains the lowest metric and has the greatest variation. In contrast, most respondents would play again under the same moderator (
M
= 9.06,
SD
= 1.69). I'm planning on running correlations to see if winning or alignment affects any of these variables for a future report when there is a bigger sample.

Qualitative Responses


I'm not familiar at all with qualitative analysis methods, so this is going to be a census of what issues respondents presented in general. The intent of the free-response questions was to get a sense of what sorts of things players wanted when playing a game beyond the numbers, and most respondents were open and honest with their thoughts, which is greatly appreciated.

  1. The most common complaint was regarding
    frequency of vote counts
    .
  2. Certain games had comments regarding
    balance
    , which is to be expected given the ratings. The Mini Theme Queue currently doesn't require reviews. While most have been good about getting their setups checked on their own, I've considered making reviews mandatory, since the Micro and Mini Theme Queues are the only ones to not make reviews mandatory at this point. Right now, the Blitz Queue is a good test case, so we'll see how that goes.
  3. There were a few requests for
    flavor
    , which is surprising, since I've always expected people to not read them. It might be because this is a Theme queue.

Comments that were specific to certain games were not included.

Queue Speed


As of December 22, the following metrics were calculated:

  • Games took an average of
    50.8
    days to go from /in to sign-ups (
    N
    = 24, median = 56,
    SD
    = 23.39, range: 9-83).
  • Games took an average of
    11.8
    days to fill (
    N
    = 22, median = 10,
    SD
    = 8.43, range: 0-28). Games that have not filled after 28 days are pulled from the queue.
  • Games took an average of
    36.9
    days to run to completion (
    N
    = 17, median = 39,
    SD
    = 14.37, range: 7-58).

Not counting games that were dropped from the queue or from sign-ups, there were an average of
4.1
moderator sign-ups per month (5 in June, 7 in July, 2 in August, 2 in September, 3 in October, 6 in November, 4 in December).

There is a moderate correlation between game number and days from /in to sign-ups with more recent games spending less time in the queue,
r
(22) = -0.48,
p
= .024. This is likely a consequence of the moderator shortage being observed site-wide (see zoraster's player slot census for 2012-2015).

Replacements


As of December 22, the average replacement rate was
3
replacements (mode = 3, range: 1-5) across an average of
2.8
slots (mode = 3, range: 1-4) in 17 games.

Compare this to chamber's report of an average of 3.1 slots replaced per Newbie Game.

Postscript


These results reflect only respondent ratings of things, so it may not reflect reality. What it does say is that moderators, on average, are doing well in giving players the experience they expect, which is what is expected of a game moderator.

Moderators can request a summary report for their games, which will contain the data contained in this post presented as letter grades and general sweeping comments, as above. No individual responses will ever be reported, whether publicly, to game moderators, or to site staff. Confidentiality is taken seriously; I separate username and player information so that they're used
only
to (1) check for abuse and (2) update the Mini Theme Archives, so I can't connect comments or ratings to specific respondents.

Those of you who have also received post-game surveys for Blitz games may have noted improvements to questions and presentation. Depending on how responses to those go, the changes may be applied to the Mini Theme surveys sometime in January 2016. Regardless, I would love any suggestions and feedback from you on how to improve the survey and what additional metrics are of interest to you.

Thank you to everyone who took the time to fill out surveys!
User avatar
Plotinus
Plotinus
Kitten Caboodle
User avatar
User avatar
Plotinus
Kitten Caboodle
Kitten Caboodle
Posts: 7611
Joined: March 13, 2015
Location: UTC+1
Contact:

Post Post #1 (ISO) » Wed Dec 23, 2015 9:43 am

Post by Plotinus »

Neat. Thanks for doing this work!
The failure mode of clever is asshole.

Modding checklists | Sequencer is in Game 5 | Space II is in Day 4
Post Reply

Return to “Mafia Discussion”