Modbot to Help You Run Games: Now With A Votecounter Demo!

Flubbernugget · Post Post #100 (ISO) » Sat May 12, 2018 6:01 pm

Is there more to that than donbot.getPosts()? I'm trying to sendPM's to a (python) list of people and only the first person in the list seems to be getting it.

Psyche · Post Post #101 (ISO) » Sat May 12, 2018 6:09 pm

I think I've posted about that recently. You need your postdelay parameter to be higher than 1.5 seconds. The site just doesn't let you send too many pms too quickly in succession.

In post 84, Psyche wrote:have found that sequential sendPM() calls require a higher postdelay parameter than 1.5 (the default for donbot). For example, if you call
Code: Select all
bot = Donbot(username='Psyche', password='literallymypassword')
bot.sendPM('testing', 'test', 'Psyche')
bot.sendPM('testing', 'test', ['yessiree', 'vonflare'])
, only the first pm gets sent, but if you call
Code: Select all
bot = Donbot(username='Psyche', password='alsoliterallymypassword')
bot.sendPM('testing', 'test', 'Psyche')
bot.sendPM('testing', 'test', ['yessiree', 'vonflare'], postdelay=5)
, both pms get sent fine

I need to modify the donbot readme to be clearer about how important the postdelay parameter can be depending on what you're up to

anyway yessiree's changes are confirmed good!

You know, if you have a skype or something, maybe we can pick a time to screenshare and go over a few things?

Psyche · Post Post #102 (ISO) » Sun May 13, 2018 9:07 am

my votecounter uses scrapy's selector for finding votes in posts instead of lxml like donbot, and pyinstaller doesn't seem to play well with the former
so i either need to figure out why pyinstaller is misbehaving, or recode my findVotes function to be based on lxml
the latter seems preferable, but more tedious
so we'll see what i do

AnonymousGhost · Post Post #103 (ISO) » Sun May 13, 2018 9:37 am

Ego.

Excitedly watching this!

Psyche · Post Post #104 (ISO) » Tue May 15, 2018 7:44 am

hello everyone
i wonder if any of you have been up to anything lately

ive figured out that lxml's html.fromstring() function does weird things when taking in "partial" html - eg html that isn't wrapped in <html><body></body></html> or something
it doesn't always just add these tags - it can create additional tags, like <span></span> or <div></div> depending on the content of your "partial" html string
depending on how you structure your xpaths, these inconsistencies can create a lot of trouble when processing a series of posts

consequently, i'll probably change donbot.getPosts() to automatically wrap each post in <html><body></body></html> so that we can use lxml without worrying about these patterns

anyway, with this phenomenon noted, i've made a version of my votecounter that doesn't rely on scrapy's Selector (it relies on lxml's instead!) so that i don't have to unnecessarily include scrapy in our client-side app or find hacky ways to exclude it

newest release of the client-side tool is here: https://github.com/MafiaScum-Unofficial ... s/tag/v.20

yessiree · Post Post #105 (ISO) » Tue May 15, 2018 8:03 am

probably better to stick to one html parser anyway

on another note, I upgraded my local to python 3.6.5 finally, and reinstalled all the packages. There seems to be an issue with requests after doing so. after some digging around I found out it was because gevent's monkey.patch_all() was running

after

importing requests
https://github.com/gevent/gevent/issues/1016

this only seems to affect python ~3.6, but anyway, PR soon to rectify this

Psyche · Post Post #106 (ISO) » Tue May 15, 2018 8:07 am

ohhh maybe that was what i was having trouble with
i actually reverted the part with the multithreading because i was getting big delays and then errors
grateful if you can fix that

yessiree · Post Post #107 (ISO) » Tue May 15, 2018 8:18 am

just pushed the fix

yessiree · Post Post #108 (ISO) » Tue May 15, 2018 8:29 am

I don't think we should try to parse through broken tags for votes, and bend over backwards while doing so

1) it's an anti-pattern
2) people will most likely post a followup correcting it so we don't really miss anything anyway

Psyche · Post Post #109 (ISO) » Tue May 15, 2018 8:33 am

if it doesn't parse broken tags, it misclassifies D1 lynches in the data set more often

Psyche · Post Post #110 (ISO) » Tue May 15, 2018 8:37 am

and idk
i think the principle of "if it looks like he's trying to vote this person, i'll treat it as an attempt to vote this person" seems pretty legit modwise, especially in tough situations like LYLO

yessiree · Post Post #111 (ISO) » Tue May 15, 2018 8:42 am

then I would conclude that people should BLOODY STOP MESSING UP VOTES TAGS YEH?

do mods usually accept broken votes? usually people who messed up will at least attempt to correct themselves no?
idk

but it's your vote counter so ultimately it's your call lol

I just dont want to watch another brother being punished for other people's mistakes

callforjudgement · Post Post #112 (ISO) » Tue May 15, 2018 8:48 am

If the modbot is entirely modding the games, and players know that broken tags will be counted, they'll probably use them for some sort of fake-voting shenanigans.

I think that if we're moving to entirely automated moderation, having objective rules about what is and isn't a vote will be important (such rules will effectively exist anyway, so it helps to let people know what they are). That's different from the situation where a computer is backing up a human mod.

Psyche · Post Post #113 (ISO) » Tue May 15, 2018 8:56 am

i can add options to give moderators some latitude over this issue, but it's tough atm discerning what kind of latitude they might want

yessiree · Post Post #114 (ISO) » Tue May 15, 2018 8:59 am

yes I agree, and I also think that you should have zero expectation whatsoever on what the mod chooses to do when you cast a vote with broken format

"eh, whatever, the mod will probably realize that I'm voting XXX"
I dont think this is a good mentality to have, nor a behavior to be encouraged, at any rate

this is when you would consult with "the business people" on what "the clients" would want

Psyche · Post Post #115 (ISO) » Tue May 15, 2018 9:01 am

ok ill start with an option to accept broken tags or not

Flubbernugget · Post Post #116 (ISO) » Tue May 15, 2018 1:05 pm

I think i got the PM aspect of the prodbot working. The next step is to generate a sendto list based on how long a player can go without posting before being prod-worthy.

I have a few items of note though.

I had to change this snippet in donbot:

Code: Select all

216         for user in uids:
217             form['address_list[u][{}]'.format(user.value)] = 'to'
218 
219         for name in ['lastclick', 'creation_time', 'form_token']:
220             form[name] = compose.xpath(postformpath.format(name))[0]

as such:

Code: Select all

---------------------------------------------------------------------------------------------
FROM:
216         for user in uids:
217             form['address_list[u][{}]'.format(user.value)] = 'to']
---------------------------------------------------------------------------------------------
TO:
216         for user in uids:
217             form['address_list[u][{}]'.format(user)] = 'to']
---------------------------------------------------------------------------------------------

My conclusion now is that I am getting the uids as strings, whereas someone else seemed to be getting them as some sort of more complex object. I want to hold off on a PR until I can figure out if/how this would break everyone else's stuff (there will be a push to my fork soon though!!).

Also, the prodbot seems to crap out at around 13-15 players in a single prod with a postdelay of 5 (RIP fortune cookie thread). The errors I have gotten with larger prod sets seem to be either lxml parser errors or http connection refusals. This leads me to believe I'm hitting some limit on the MS server. I'm not 100% sure if it's worth digging into this issue any further than I have, since 13+ prods seems like a lot.

GreenLiquid · Post Post #117 (ISO) » Tue May 15, 2018 1:18 pm

If we ever get to the point of game automation, though, it'll crop up again when there's a need to send role PMs in a large-sized game. How does it work with a delay of 10s?

Also I've been... not working on this at all due to being sick over the weekend and the past couple of days. Once I improve I want to take the PM and autoposting code for a test spin and see what the logic to automate in the context of an actual game would look like.

Flubbernugget · Post Post #118 (ISO) » Tue May 15, 2018 1:47 pm

So, I did a semi-through test of this. I did start sending PM's to a list of 15 people with a post delay of 10. This mostly gave lxml errors. Putting sleeps in sendPM did not mitigate this issue. I got to the point where I just started slicing the sendto variable from [:1] onward, changing postdelay to 5 to speed up testing.

There is definitely something bizarre going on, as there's not enough information in my course of actions to really figure out what's choking up the server. And hell, its code, so there's a good chance I'm wrong in declaring that as the problem anyway

If it's really worth digging into this, I could probably set up a test matrix across a couple of variables (postdelay, number of players, etc...). I don't know what conclusions we'd be able to make from that information, but the results would be comprehensive. The only thing that bothers me about this is spamming users with PM's.

Alternatively, I could try to contact Kison to see if there's any bottlenecks I'm hitting. This might also have the advantage of quickly determining whether or not my looking at server limits is confbias on my end.

Psyche · Post Post #119 (ISO) » Tue May 15, 2018 2:03 pm

push the code if you want; i have some ideas for figuring out what's wrong

GreenLiquid · Post Post #120 (ISO) » Tue May 15, 2018 2:11 pm

Does the result change if you push 13 PMs while the application is running, kill the application, then immediately restart it and send 1 additional PM?

If it turns out this is unavoidable for some reason, I guess we could include a step where the bot checks its outbox / sent box for the message to ensure it actually sent before proceeding to its next action, but that'd really slow things down.

Flubbernugget · Post Post #121 (ISO) » Tue May 15, 2018 2:12 pm

I got side tracked. Prod limit->sendto list, then a push! I promise this time

Flubbernugget · Post Post #122 (ISO) » Tue May 15, 2018 2:14 pm

In post 120, GreenLiquid wrote:Does the result change if you push 13 PMs while the application is running, kill the application, then immediately restart it and send 1 additional PM?

I should only be sending 1 PM to 13 people.

GreenLiquid · Post Post #123 (ISO) » Tue May 15, 2018 2:16 pm

Right, sorry, I meant... is it possible to get it out of its state of not sending PMs by exiting and re-launching the application? Just to see if it's something within that session or not. Might help narrow down the cause.

Flubbernugget · Post Post #124 (ISO) » Tue May 15, 2018 2:17 pm

I'm pretty sure there's a place in hell where you're forced to use vim with no way to tell whether or not caps lock is on