Project ELO (Dataset released)

Realeo · Post Post #125 (ISO) » Fri Jan 20, 2017 9:49 am

Our algorithm seems to disagree.

You put T S O at 0.73, which is a superior town. I put T S O at 1500, which is a completely average town.

Edit: Ignore this. It turns out we agree.

mhsmith0 · Post Post #126 (ISO) » Fri Jan 20, 2017 9:55 am

In post 123, Realeo wrote:The premise of logistic regression is promising.

I have an idea. Is it possible for the algorithm to merge to identical players? Apply cluster algorithm by merging similar player with higher data.

Possibly? I don't have experience with cluster algorithms so I can't really comment much there.

I will say that if you're going to do logistic regression, it's important to have some level of baseline model (for a game full of "other" players), and then build in variables that represent subbing in a "known" player for a random "other" player.

The nice thing about newbie games is you don't have to worry about multiball/sk really screwing with the results, and you always have setups that are all reasonably close to each other in terms of win probabilities (there's a major difference between cop/doc/rb and bp/tracker results, but even there, 40% vs 55% or so isn't wild and crazy, and that's much less than the variation in setup balance for mini normals, opens, mini themes etc, and that's even if you chuck out the multiball/sk setups, which if included without making carefully thought out adjustments potentially skew the results materially).

mhsmith0 · Post Post #127 (ISO) » Fri Jan 20, 2017 10:05 am

In post 125, Realeo wrote:Our algorithm seems to disagree.

You put T S O at 0.73, which is a superior town. I put T S O at 1500, which is a completely average town.

Well, TSO is 7-5 in newbie games, which is a pretty solid record in a format that's a 44.2% overall town win rate. I can't comment on the data you did or didn't include, but I'd note/question:

1) Is TSO's non-newbie game data materially worse than his newbie game data? If the answer is "yes" (presuming your calc was combined data) then that's probably the answer right there.
2) Have you rolled together town and scum games to get a combined rating (IIRC that was your approach)? TSO's 2-2 scum record is fine but a bit below average in newbies, so I could see negatives there. Also not sure what his non-newbie scum game data would be.
3) TSO's actual newbie town win rate is 58%, and his simple modeled newbie town win rate is also 58% (intercept + TSO!town factor = 0.31, which projects a 58% town win rate). Skimming his data, his teammate rands don't seem especially outside the norm (some higher rated, some lower rated, but no particularly apparent major bias), and is almost exclusively up against "other" scum players (1 of his 12 was against nacho but that was it), so that seems to line up in my mind.

So I guess I don't see from my data why he shouldn't be more or less exactly where he is. What in your data suggests that an "exact average" is the appropriate result there?

Realeo · Post Post #128 (ISO) » Fri Jan 20, 2017 10:23 am

TSO showed a 4/5 as town and 2/0 as scum at Normals.

I am on insomnia due to Donald Drumpf, so I would analyze more specific later on after I get sleep.

@mhsmith

is the mhsmith factor(logic) is all you have? May I have data dump? After I get sleep, I want to do algorithm comparison.

mhsmith0 · Post Post #129 (ISO) » Fri Jan 20, 2017 10:28 am

PS cabd is another one where basic W-L (6-8, 43%) and the modeled rating (-0.43, expected 30% win rate) disagree a lot, but that's another one heavily skewed by his town teammates. 2 with GIF, 1 with mala, 2 with ffery, 2 with TSO, 1 with singer, 2 with micc, 1 with drixx, 1 with xayzeck, which is a lot of teammate power in a 14 game sample size.

mhsmith0 · Post Post #130 (ISO) » Fri Jan 20, 2017 10:32 am

Spoiler: ugly data dump

if you want a prettier data dump let me know what you're looking for. The first half (from left to right) is count of town players (so the "anyone else" tally usually is in the 4 to 7 range), and the second half is count of scum players (headers repeat since I count town or scum). Note that the 1/2 column going down the middle refers to "anyone else" for scum players, that may be helpful in terms of organizing / checking data.

I also did the data feed in terms fo
Game 17xx
xxx, Vanilla Townie, lynched day 1
...

awhile back, so that's relatively up to date if you want a list of all players instead of just the ones I'd looked at.

Realeo · Post Post #131 (ISO) » Fri Jan 20, 2017 10:44 am

Ah. So the table earlier is accurate mhsmith factor?

mhsmith0 · Post Post #132 (ISO) » Fri Jan 20, 2017 10:53 am

i updated 1500-1750 or so in post 53, here's 1700 through 1765

Spoiler: 1700-1765

It's kinda piecewise at this point, but that's basically my data. Probably some amount of error in it, but I've put in a pretty solid effort at cleaning it up so I doubt that errors are material.

Realeo · Post Post #133 (ISO) » Fri Jan 20, 2017 6:40 pm

My current priority is this

[ DONE ] Integrate Ne 17xx
[ ] Implement the Dynamic Version of user spell checking
[ ] Fine tune the K factor
[ ] Compare the ELO and mhsmith factor to find which one is more predictive

Despite I already integrate Newbie 17xx, I won't release it now, because my focus is more to algorithm development instead of leaderboard development. Although, I can give a spoiler that what used to be fferyllt-(huge gap)-Vedith is no more. There are now 4 players between fferyllt-Vedith

mhsmith0 · Post Post #134 (ISO) » Sat Jan 21, 2017 12:30 pm

OK so i did a bit of an update to 113, I added data tracking in my table for everyone with 8+ games instead of 10+, which means that I now picked up a couple people with 8 town games but only 8-9 total games. New #s:

Spoiler: Log Results, Newbie 1370-1765

Raskolnikov · Post Post #135 (ISO) » Sat Jan 21, 2017 12:40 pm

I like smith's data better, clearly its more accurate

mhsmith0 · Post Post #136 (ISO) » Sat Jan 21, 2017 12:41 pm

I'm only four more town games away from getting on the table

LicketyQuickety · Post Post #137 (ISO) » Sat Jan 21, 2017 12:58 pm

In post 133, Realeo wrote:My current priority is this

[ DONE ] Integrate Ne 17xx
[ ] Implement the Dynamic Version of user spell checking
[ ] Fine tune the K factor
[ ] Compare the ELO and mhsmith factor to find which one is more predictive

Despite I already integrate Newbie 17xx, I won't release it now, because my focus is more to algorithm development instead of leaderboard development. Although, I can give a spoiler that what used to be fferyllt-(huge gap)-Vedith is no more. There are now 4 players between fferyllt-Vedith

I am torn between wanting you to share your results and waiting until you have everything current.

Realeo · Post Post #138 (ISO) » Sat Jan 21, 2017 3:06 pm

Your face when the algorithm suggests there are 120 misspelling

mhsmith0 · Post Post #139 (ISO) » Sat Jan 21, 2017 3:10 pm

That doesn't even surprise me tbh. I manually cleared out like 50 misspellings, and that was just focusing on people with a bunch of games. i basically didn't put much effort into sorting people with like 2-3 games.

Anyway, if you have a better / corrected feed, then awesome, I'd be happy to use.

Realeo · Post Post #140 (ISO) » Sat Jan 21, 2017 3:13 pm

Sure. I may as well combine the Newbies + Normals to create one clean data.

mhsmith0 · Post Post #141 (ISO) » Sat Jan 21, 2017 3:13 pm

Also with 1767 in the books, new #s

Spoiler: 1300-1765, plus 1767

Huntress, who was winning town in 1767, went up 0.15 (0.42 to 0.57), all other changes were pretty much marginal. Updated intercept: -0.379, predicted 40.6% win odds given entirely "other" players, compares to actual overall town win rate of 44.3%, with the difference being due to the players tracked in the table.

Aeronaut · Post Post #142 (ISO) » Sat Jan 21, 2017 6:35 pm

In post 141, mhsmith0 wrote:Also with 1767 in the books, new #s

Spoiler: 1300-1765, plus 1767
Log Rating Record

Player Town Scum Combined Town Scum Combined

GuyInFreezer 1.64 1.64 12-3 1-1 13-4

Thor665 0.52 -0.36 0.88 12-12 6-3 18-15

Malakittens 0.86 0.86 17-10 3-3 20-13

fferyllt 0.75 0.75 12-5 3-1 15-6

T S O 0.69 0.69 7-5 2-2 9-7

Huntress 0.57 0.57 7-5 1-2 8-7

Micc 0.55 0.55 5-4 2-1 7-5

singersigner 0.55 0.55 6-5 1-0 7-5

Xayzeck 0.48 0.48 6-4 1-4 7-8

Drixx 0.48 0.48 11-10 3-2 14-12

Hopkirk 0.31 0.31 5-3 2-0 7-3

Bulbazak 0.28 0.28 5-4 2-2 7-6

Guyett 0.22 0.22 7-7 2-2 9-9

Raskolnikov 0.18 0.18 4-5 1-2 5-7

RadiantCowbells 0.17 0.17 12-14 3-1 15-15

theslimer3 0.09 0.09 4-4 0-0 4-4

hayatoBL 0.07 0.07 5-6 3-1 8-7

Dierfire 0.01 0.01 7-10 2-2 9-12

TheIrishPope 0.00 0.00 4-5 1-3 5-8

innocentvillager -0.05 -0.05 5-7 3-1 8-8

Wisdom 0.08 -0.08 3-4 6-4 9-8

goodmorning -0.17 -0.08 -0.09 14-23 7-5 21-28

jmo16mla -0.11 -0.11 4-5 0-0 4-5

House -0.14 -0.14 5-7 2-0 7-7

JasonWazza -0.14 -0.14 4-5 3-3 7-8

RachMarie -0.19 -0.19 11-18 2-2 13-20

Nachomamma8 0.68 0.88 -0.19 17-12 5-12 22-24

Nobody Special -0.20 -0.20 6-8 1-2 7-10

Accountant -0.26 -0.26 6-11 2-1 8-12

Sakura Hana -0.27 -0.27 5-6 3-3 8-9

PhantomCobalt -0.29 -0.29 5-8 1-1 6-9

Not_Mafia -0.32 -0.32 6-10 3-4 9-14

notscience -0.30 0.16 -0.46 9-16 5-4 14-20

NicCage -0.46 -0.46 2-6 0-0 2-6

Cabd -0.47 -0.47 6-8 0-0 6-8

BlueBloodedToffee -0.26 0.21 -0.47 9-17 6-5 15-22

Alexcellent -0.50 -0.50 3-5 0-1 3-6

Ms Marangal -0.52 -0.52 2-6 1-4 3-10

enomis -0.52 -0.52 2-6 0-0 2-6

Titus -0.66 -0.66 3-9 0-2 3-11

Flubbernugget -0.75 -0.75 3-8 0-0 3-8

MarioManiac4 -0.86 -0.86 2-8 1-2 3-10

Jake from State Farm -0.92 -0.92 2-10 1-3 3-13

Huntress, who was winning town in 1767, went up 0.15 (0.42 to 0.57), all other changes were pretty much marginal. Updated intercept: -0.379, predicted 40.6% win odds given entirely "other" players, compares to actual overall town win rate of 44.3%, with the difference being due to the players tracked in the table.

:[ why am i not in this

mhsmith0 · Post Post #143 (ISO) » Sat Jan 21, 2017 6:40 pm

because in that span of games you only have four of them counted:
1435 town
1523 town
1566 town
1580 town

you had another three you subbed out of, but I'm only counting the player who ended a game in a slot, and even if they had counted, it's not 8 in a single alignment

1534 town
1652 town
1691 town

Realeo · Post Post #144 (ISO) » Sun Jan 22, 2017 3:55 am

[ DONE ] Integrate Ne 17xx
[ DONE, but like Turing Completeness, we never 100% sure ] Implement the Dynamic Version of user spell checking
[ DONE ] Fine tune the K factor
[ After I get sleep ] Compare the ELO and mhsmith factor to find which one is more predictive

Realeo · Post Post #145 (ISO) » Sun Jan 22, 2017 4:24 am

mhsmith, while I'm going to take a sleep, can I have log loss of your multivariate model predicting the game and classic accuracy (number of correct prediction/ all game to analyze)?

Predicting the game, not the player win rate

Realeo · Post Post #146 (ISO) » Sun Jan 22, 2017 4:27 am

It's a shame that due to the nature of ELO, I cannot cross-validate.

Loopdan · Post Post #147 (ISO) » Sun Jan 22, 2017 2:42 pm

@mhsmith0
I have ten completed Newbies as town and two as scum in the span of 1711-1767. What's a guy gotta do to get on that fancy chart of yours?

mhsmith0 · Post Post #148 (ISO) » Sun Jan 22, 2017 3:22 pm

@loop:

I've got you as 4-3 town, 2-0 scum. My guess is three of the games are your TTTT alt?

town games:
1711
1725
1736
1739
1746
1756
1767

mhsmith0 · Post Post #149 (ISO) » Sun Jan 22, 2017 3:46 pm

@realeo: my combined error function is -248.733 (with my process set to maximize against it).

Error function set as:
W = 0 if scum win, 1 if town win
L = sum of all factors (for a game with nine "other" players, would be = -0.37)
p = e^L / (1 + e^L)
X = p^X * (1-p)^(1-X)
Error = ln(X)

Then i sum up the errors for each line. Average error = -0.627.

It's kinda hard to translate into more intuitive terms, but it would be the (very loose) equivalent of predicting each game correctly at 54% odds (i.e. every game is called town at 54% win odds and they win, or town at 46% win odds and they lose). More accurately, there are a number of games where the model will make a high confidence "prediction" (or, rather, fit to overall results), and most (but not all) of those will be correct.

Two simple examples of good fit:
1) The model translated newbie 1370 (fferyllt and GIF as town, everyone else as "other") as having 88.2% town win odds, and town did win that one.
2) The model translated newbie 1476 (goodmorning, not_mafia, notscience, RachMarie, and Jake From State Farm) as having 90.8% scum win odds, and scum won (this is actually the single "best fit" result, fwiw)

As a simple example of bad fit, the model translated newbie 1508 (Jake from State Farm and Cabd as town) as having 14.5% town win odds, and town ALSO won that one.

Another way of looking at it (as if there weren't enough): there were 17 games where the model "predicted" the result at 80% or better confidence. The model was correct in 16 of them (1508 being the exception).

	Log Rating			Record
Player	Town	Scum	Combined	Town	Scum	Combined
GuyInFreezer	1.65		1.65	12-3	1-1	13-4
Thor665	0.53	-0.36	0.89	12-12	6-3	18-15
Malakittens	0.86		0.86	17-10	3-3	20-13
fferyllt	0.75		0.75	12-5	3-1	15-6
T S O	0.69		0.69	7-5	2-2	9-7
Micc	0.57		0.57	5-4	2-1	7-5
singersigner	0.56		0.56	6-5	1-0	7-5
Drixx	0.49		0.49	11-10	3-2	14-12
Xayzeck	0.48		0.48	6-4	1-4	7-8
Huntress	0.42		0.42	6-5	1-2	7-7
Hopkirk	0.31		0.31	5-3	2-0	7-3
Bulbazak	0.28		0.28	5-4	2-2	7-6
Guyett	0.22		0.22	7-7	2-2	9-9
Raskolnikov	0.19		0.19	4-5	1-1	5-6
RadiantCowbells	0.17		0.17	12-14	3-1	15-15
theslimer3	0.09		0.09	4-4	0-0	4-4
hayatoBL	0.07		0.07	5-6	3-1	8-7
Dierfire	0.01		0.01	7-10	2-2	9-12
TheIrishPope	0.00		0.00	4-5	1-3	5-8
innocentvillager	-0.05		-0.05	5-7	3-1	8-8
Wisdom		0.07	-0.07	3-4	6-4	9-8
goodmorning	-0.16	-0.08	-0.08	14-23	7-5	21-28
jmo16mla	-0.10		-0.10	4-5	0-0	4-5
House	-0.14		-0.14	5-7	2-0	7-7
JasonWazza	-0.14		-0.14	4-5	3-3	7-8
RachMarie	-0.19		-0.19	11-18	2-2	13-20
Nachomamma8	0.69	0.88	-0.19	17-12	5-12	22-24
Nobody Special	-0.20		-0.20	6-8	1-2	7-10
Accountant	-0.25		-0.25	6-11	2-1	8-12
Sakura Hana	-0.27		-0.27	5-6	3-3	8-9
PhantomCobalt	-0.29		-0.29	5-8	1-1	6-9
Not_Mafia	-0.32		-0.32	6-10	3-4	9-14
NicCage	-0.46		-0.46	2-6	0-0	2-6
notscience	-0.30	0.16	-0.46	9-16	5-4	14-20
BlueBloodedToffee	-0.26	0.21	-0.47	9-17	6-5	15-22
Cabd	-0.47		-0.47	6-8	0-0	6-8
Alexcellent	-0.49		-0.49	3-5	0-1	3-6
enomis	-0.52		-0.52	2-6	0-0	2-6
Ms Marangal	-0.52		-0.52	2-6	1-4	3-10
Titus	-0.66		-0.66	3-9	0-2	3-11
Flubbernugget	-0.74		-0.74	3-8	0-0	3-8
MarioManiac4	-0.86		-0.86	2-8	1-2	3-10
Jake from State Farm	-0.92		-0.92	2-10	1-3	3-13