[Meta] Newbie Experience (Formerly IC replace rate thread)

[RadiantCowbells]

It's not updated tho

[PenguinPower]

Yeah, it's not broken out like that though, IIRC. When I said "more of that," I meant more types of metrics.

[PenguinPower]

Damnit, RC.

[Toomai]

I'll update it today then. And the linked spreadhseet is always up-to-date within a day or two.

[Zachrulez]

I do my best to fill vacancies. For the record I think the 24 hour rule on newbie slots rule is counterproductive since it's mostly just causing newbie games to be short a slot for 24 hours more, and that's something that should change.

Disagree. Nothing turns me away from newbie games harder than having the majority of the game not be newbies at all

If I wanted to play in a game with a bunch of normal users, I'd join a micro. Which is what I do.

There should be

harsher

limitations on replacements into newbies. At no point should se/ic players outnumber actual newbies

If you get to a point where newbies replace out like that the game has essentally served its purpose in filtering those players out. Keeping some kind of newbie integrity is silly. When I need to replace a newbie slot I don't care who takes it. Only se and ic slots need be restrictive with replacements.

[Gamma Emerald]

I do my best to fill vacancies. For the record I think the 24 hour rule on newbie slots rule is counterproductive since it's mostly just causing newbie games to be short a slot for 24 hours more, and that's something that should change.

Disagree. Nothing turns me away from newbie games harder than having the majority of the game not be newbies at all

If I wanted to play in a game with a bunch of normal users, I'd join a micro. Which is what I do.

There should be

harsher

limitations on replacements into newbies. At no point should se/ic players outnumber actual newbies

If you get to a point where newbies replace out like that the game has essentally served its purpose in filtering those players out. Keeping some kind of newbie integrity is silly. When I need to replace a newbie slot I don't care who takes it. Only se and ic slots need be restrictive with replacements.

Do you even mod

[xyzzy]

I think w/r/t any kind of incentive for being an IC, a good metric for deciding who gets that incentive should be the frequency of newbies staying on the site beyond their first game; if someone has a bad experience in their very first game due to a bad IC, they're unlikely to stick around, but if someone has an especially good experience in their first game due to a good IC, they're much more likely to join more games. this isn't a perfect metric, because some people will just leave the site after one game regardless of how the first one went, and some people will stick around even if they had a bad experience the first time, but I suspect that there's a correlation between games with especially good, effective ICs and games that lead to people continuing to use the site beyond their first game.

[Psyche]

Some ICs could think their job is to weed out bad newbies.

[chamber]

That is part of the newbie queue, as long as we define bad as users with activity or behavioral problems. Otherwise it would probably be better to just not even have a newbie queue.

[Zachrulez]

I do my best to fill vacancies. For the record I think the 24 hour rule on newbie slots rule is counterproductive since it's mostly just causing newbie games to be short a slot for 24 hours more, and that's something that should change.

Disagree. Nothing turns me away from newbie games harder than having the majority of the game not be newbies at all

If I wanted to play in a game with a bunch of normal users, I'd join a micro. Which is what I do.

There should be

harsher

limitations on replacements into newbies. At no point should se/ic players outnumber actual newbies

If you get to a point where newbies replace out like that the game has essentally served its purpose in filtering those players out. Keeping some kind of newbie integrity is silly. When I need to replace a newbie slot I don't care who takes it. Only se and ic slots need be restrictive with replacements.

Do you even mod

Whether or not I mod doesn't change the fact that it's probably even less realistic now to expect newbie slots to be replaced only by newbies than it was when I actually was modding. (There was a much more sizable pool of newbies to draw from, and still limited interest in responding to replacement ads.)

[Ircher]

I'm pretty sure there are more newbies now than when I joined.

[fferyllt]

the sign up threads are still out there. it's easy enough to sample how many new players /inned, how many games launched per month. Might be interesting to pick 2 or three months and compare them across 3 or 4 years.

[Sunlit Diamond]

the sign up threads are still out there. it's easy enough to sample how many new players /inned, how many games launched per month. Might be interesting to pick 2 or three months and compare them across 3 or 4 years.

This is a thing I would enjoy doing. *Contemplates*

[fferyllt]

Yeah, I'm thinking about doing it, too. I'd like some input about what would make a decent sample size. And also if there's an easier and more automated way to go about it.

I'd also be interested in doing something to track how many people keep playing after their first game, and what percentage are still playing a year later.

And do that over the same time windows maybe.

It would be a lot of manual labor.

Maybe psyche could make a bot!

[Gamma Emerald]

I'd let Psyche worry about his current bots for now

[PenguinPower]

I'd also be interested in doing something to track how many people keep playing after their first game, and what percentage are still playing a year later.

This is something I think would be beneficial both from a stats pov, and an engagement pov. Meaning, I don't think that this info should just be pulled from historical data, but ongoing and through direct consultation with the players.

[fferyllt]

Yeah, for sure. I'm thinking a baseline trend from a few years past would help put what's happening now into perspective.

[Sunlit Diamond]

How far back would you want to go?

[PenguinPower]

IMO, 3 years prior coupled with ongoing data would be good.

Glad to see you here, SD.

[Sunlit Diamond]

*salutes* I suspect I'll be here a lot. I've been digesting the Mastin Academy thread for the last week or so.

Let me see how automated I can get this, and get a spreadsheet going.

Edit:

My perfectionist mind wants to build a stats sheet for all of the last three years. My realistic mind just reminded me that I have a full time job, a three year old, I'm going to school, and I'm playing mafia. While I could do it, it would take foreeeever. What I can do is dump what I gathered tonight into a Google Sheet so interested parties can collaborate, if interested.

Two random facts I gleaned:

1. Number of Newbie games per month steadily climbed from 2013 to 2015; after that, they began to decline. The difference between 2016 and 2017 from January to June is 20%.
2. We have exactly one player remaining from the first Newbie game of 2013.

And now I toddle off to bed.

[fferyllt]

This is awesome info!

I'm interested in seeing what you've gathered so far, and I am planning to work on this a bit, too. Since you were able to nail the trend down objectively, I'll start on a detailed look at 2017 games.

[fferyllt]

This is a high level summary of the data I've captured about finished 2017 Newbie games.



These games were all announced by the Listmods in the Newbie Queue in those months. There was one game that started in early January, but because it was announced in December, I arbitrarily chose not to put it in the stats. It would be easy enough to add it, if tracking the games by when they actually started is a better approach.

I decided to track replacement data not only because a lot of first-time newbies start their first games as replacements, but because I think unusually large numbers of replacements probably indicate there was something anomalous about the game that may have turned off players from finishing the game and potentially playing additional games here.

As ongoing games finish, I'll update the spreadsheet. I don't plan to make the detailed spreadsheet generally available since I have to capture activity data on individual players in order to calculate aggregates. I feel like there's probably a window after which a first-time newbie who played their first game should be considered an established member and no longer tracked in terms of whether they're still visiting the site, playing games, or whatever.

Thoughts about additional information:

Instead of tracking whether the player is still active "today", track if they're active 1 month, 2 months, 6 months after their first game finishes.
Track how many games they play in the following 1 month, 2 months, 6 months after their first game finishes

There was no way to capture most of this information without doing a bunch of brute-force effort looking at newbie listmod posts, reading games, etc.

At the level of granularity I've captured the info, it would eventually be possible to see if particular mods, ICs, SEs are involved in games with high conversions of first time newbies to established members. It would also be possible to see if certain game "cohorts" go on to play more games together and stick around longer. And I hope it will be possible to tease out some best practices: identify established mods and players whose games have an unusually high conversion rate, and then look at what makes those games different and presumably better for newbies. It may also be possible to see what kinds of second, third, fourth game experiences (which queues, which types of games, etc) correlate with continuing to play on-site months later.

If you have thoughts about summary data that would be useful to track, let me know.

I haven't decided whether to track prior years in this detail. It's a lot of work digging through the data manually.

One correlation of note is that May, the month with the most Newbie slot replacements so far this year (though not by much) was also notable in that 4 players who were in one or more newbie games at the time were sitebanned.

The May numbers were inflated by one game that had 13 replacements, most of which were newbie slots.

[Sunlit Diamond]

One thing I have noticed is that newbie games appear to require a lot more brain power than other games. There's definitely a culture shock shifting from the road to other games and that might contribute to retention rate or lack thereof (for both newbies and ICs, really). I dunno if that's helpful, but ... you never know?

[fferyllt]

There may be something to that, though I think that the need for brainpower varies a lot from game to game even in the same queues. The game design and the player base in each game has an impact.

Some of the most challenging games I've played at MS happened in the Newbie queue. And some of the easiest, quickest wins happened there, too.

[mastina]

One thing I have noticed is that newbie games appear to require a lot more brain power than other games. There's definitely a culture shock shifting from the road to other games and that might contribute to retention rate or lack thereof (for both newbies and ICs, really). I dunno if that's helpful, but ... you never know?

I still maintain my theory that running the same setup--even a semiopen one--with no alternatives may contribute to this because literally every single queue outside the newbie queue is going to have different game setups. Even the queues "closest" to newbies. (Opens for setup, Micros for size, Normals for type.)

They have variety of type and content within which newbies don't get exposure to.