Spoiler: Survey
In addition to the survey, metrics were calculated for the queue's speed and replacement rates. Because these were not dependent on survey responses, they are current as of yesterday.
For this report, 10 games were surveyed. Of 125 surveys that were sent, 76 responses have been received for a response rate of
60.8%
. One response was excluded for possible abuse.Moderator Performance
(
N
= 70)"Personnel" here refers to interpersonal problems. Moderator self-ratings were excluded.
Player satisfaction was high across the board, and there didn't seem to be any significant differences between aspects. The lowest and most variable rating was in how moderators handled interpersonal problems (
M
= 8.07, SD
= 2.25), followed by prods (M
= 8.20, SD
= 2.03) and vote counts (M
= 8.26, SD
= 1.83). Respondents were happy with how moderators handled replacements (M
= 8.52, SD
= 1.91), which is a good sign given the current replacement climate.The average satisfaction rating for games overall was 8.49 (
SD
= 1.88).Player Performance
(
N
= 76)Town and Mafia ratings would be the more reliable measure here. There was one multiball game run during this period, and respondents were instructed to mark a second mafia team as "Other." However, there was at least one game with multiple neutral parties and that made interpreting the scores for Third-Party and Other more difficult. A future report might exclude the game(s) in question for comparison's sake.
For the most part, people seemed to think that players performed little better than average, with town (
M
= 6.18, SD
= 2.32) doing slightly worse than mafia (M
= 6.78, SD
= 2.29). Given that "town did so bad, etc." happens in almost every game that completes on this site, this is no surprise.The Games Themselves
(
N
= 70)It should be noted that these are not ratings
per se
but level of agreement that these aspects were done well. The difference might or might not be significant, but testing this without losing the ability to compare between games involves a much longer survey.That said, consistent with results in the Large Theme Queue, balance (
M
= 7.21, SD
= 2.56) remains the lowest metric and has the greatest variation. In contrast, most respondents would play again under the same moderator (M
= 9.06, SD
= 1.69). I'm planning on running correlations to see if winning or alignment affects any of these variables for a future report when there is a bigger sample.Qualitative Responses
I'm not familiar at all with qualitative analysis methods, so this is going to be a census of what issues respondents presented in general. The intent of the free-response questions was to get a sense of what sorts of things players wanted when playing a game beyond the numbers, and most respondents were open and honest with their thoughts, which is greatly appreciated.
- The most common complaint was regarding frequency of vote counts.
- Certain games had comments regarding balance, which is to be expected given the ratings. The Mini Theme Queue currently doesn't require reviews. While most have been good about getting their setups checked on their own, I've considered making reviews mandatory, since the Micro and Mini Theme Queues are the only ones to not make reviews mandatory at this point. Right now, the Blitz Queue is a good test case, so we'll see how that goes.
- There were a few requests for flavor, which is surprising, since I've always expected people to not read them. It might be because this is a Theme queue.
Comments that were specific to certain games were not included.
Queue Speed
As of December 22, the following metrics were calculated:
- Games took an average of 50.8days to go from /in to sign-ups (N= 24, median = 56,SD= 23.39, range: 9-83).
- Games took an average of 11.8days to fill (N= 22, median = 10,SD= 8.43, range: 0-28). Games that have not filled after 28 days are pulled from the queue.
- Games took an average of 36.9days to run to completion (N= 17, median = 39,SD= 14.37, range: 7-58).
Not counting games that were dropped from the queue or from sign-ups, there were an average of
4.1
moderator sign-ups per month (5 in June, 7 in July, 2 in August, 2 in September, 3 in October, 6 in November, 4 in December).There is a moderate correlation between game number and days from /in to sign-ups with more recent games spending less time in the queue,
r
(22) = -0.48, p
= .024. This is likely a consequence of the moderator shortage being observed site-wide (see zoraster's player slot census for 2012-2015).Replacements
As of December 22, the average replacement rate was
3
replacements (mode = 3, range: 1-5) across an average of 2.8
slots (mode = 3, range: 1-4) in 17 games.Compare this to chamber's report of an average of 3.1 slots replaced per Newbie Game.
Postscript
These results reflect only respondent ratings of things, so it may not reflect reality. What it does say is that moderators, on average, are doing well in giving players the experience they expect, which is what is expected of a game moderator.
Moderators can request a summary report for their games, which will contain the data contained in this post presented as letter grades and general sweeping comments, as above. No individual responses will ever be reported, whether publicly, to game moderators, or to site staff. Confidentiality is taken seriously; I separate username and player information so that they're used
only
to (1) check for abuse and (2) update the Mini Theme Archives, so I can't connect comments or ratings to specific respondents.Those of you who have also received post-game surveys for Blitz games may have noted improvements to questions and presentation. Depending on how responses to those go, the changes may be applied to the Mini Theme surveys sometime in January 2016. Regardless, I would love any suggestions and feedback from you on how to improve the survey and what additional metrics are of interest to you.
Thank you to everyone who took the time to fill out surveys!