[phpBB Debug] PHP Warning: in file [ROOT]/ext/alfredoramos/seometadata/event/listener.php on line 114: Undefined array key 10146362 [phpBB Debug] PHP Warning: in file [ROOT]/ext/alfredoramos/seometadata/event/listener.php on line 114: Trying to access array offset on value of type null Modbot to Help You Run Games: Now With A Votecounter Demo! - Mafiascum.net
Post
Post #29 (isolation #0) » Thu May 03, 2018 4:54 am
Postby yessiree »
In post 17, MathBlade wrote:The other last warning I would recommend is DDOS defense
I have a troll that bombs the shit out of mine to the point where I had to make a DB for it :/ just can’t find said troll
I don't think this would be an issue for this project. Psyche is not hosting the service anywhere. We are running a local instance to interact with the site.
...If anything, the site should worry about getting DDOS'd with this tool lol
Post
Post #33 (isolation #1) » Thu May 03, 2018 9:03 am
Postby yessiree »
yeah i didnt mean in the sense that someone was gonna take your work and do some serious damage to the server, but it's always good to have that consideration for what
can
go wrong
like just bombarding the server with GET requests is enough to strain it (I hope there's server side caching), but ive noticed that sometimes the server will simply terminate your session if you send too many requests in a short time (not sure if intentional or not)
on semi-related note, it looks like you're working with document response types and just parsing the HTML. I hope we get a more streamlined API once the upgrade happens (fingers crossed)
Post
Post #51 (isolation #2) » Sun May 06, 2018 7:52 am
Postby yessiree »
In post 49, Psyche wrote:after a bit of research, i think the solution here is to give clients an application that starts a private, local server (like a pyinstaller-packaged flask app)
that delivers our html/css/js/etc-based user interface in their browser at http://localhost:.../
and talks w/ the interface to execute whatever python functions the clients need for their goals
have you considered Node.js? there is a built in http-server module that launches a local server
then you can use w/e js framework to hit your python endpoints
Post
Post #54 (isolation #3) » Sun May 06, 2018 8:19 am
Postby yessiree »
node.js would only be hosting the web app that has the UI, not the main app
the python code base stays as is, the only change that's needed is to expose the current methods as endpoints so the clients can access them with XHR requests; there are a couple ways of doing that (the endpoints package comes to mind)
Post
Post #56 (isolation #4) » Sun May 06, 2018 8:39 am
Postby yessiree »
vanilla python is pretty shit when it comes to implementing a UI. you also dont want to be dealing with any os related quirks that's guaranteed to come up. web based UI will probably give you the most bang for your buck, time investment wise
Post
Post #62 (isolation #5) » Tue May 08, 2018 10:53 am
Postby yessiree »
I have some ideas about a way to get all the most recent votes from living players
1. get a list of living players (most likely from config or a mod post)
2. get the ISO of all living players (through the activity overview screen), run these requests asynchronously, sort by timestamps in DESC order
3. parse the content and get the most recent vote/unvote, return the post #
4. wait for all the ISO requests to finish
5. compile the results
Post
Post #66 (isolation #6) » Thu May 10, 2018 6:55 am
Postby yessiree »
In post 62, yessiree wrote:I have some ideas about a way to get all the most recent votes from living players
1. get a list of living players (most likely from config or a mod post)
2. get the ISO of all living players (through the activity overview screen), run these requests asynchronously, sort by timestamps in DESC order
3. parse the content and get the most recent vote/unvote, return the post #
4. wait for all the ISO requests to finish
5. compile the results
Post
Post #68 (isolation #7) » Thu May 10, 2018 7:26 am
Postby yessiree »
we get a little boost in performance if we run code in parallel (ie. if we're getting the user id for 10 users, we can send out 10 separate requests simultaneously then wait for all to finish, VS sending one, wait for response, send next one, wait for response etc...)
for python people either go with asyncio or gevent
Post
Post #78 (isolation #8) » Thu May 10, 2018 2:30 pm
Postby yessiree »
In post 69, Flubbernugget wrote:What's the difference between gevent and asyncio? I know what a coroutine is, but don't have much experience with either library.
both libs do the same thing, just in different ways. it really comes down to preference
Post
Post #89 (isolation #11) » Fri May 11, 2018 10:43 am
Postby yessiree »
@GreenLiquid, I skimmed over your code and think I got a good sense of what you're trying to do. The thing is a beast to say the least.
That said, I can't help but feel that we are approaching this with fundamentally different mindsets. I see that you are trying to record perfect history and store them in JSON files. Then you can generate whatever you need from these data. Whereas I'm only interested in data that are relevant to my needs, so most of the time it will be grabbing the most recent votes, generating a VC, and moving on.
Your approach is perfectly fine. I'm trying to think of what you'd need to automate the data generation process. You could write a crawler that monitors the thread and polls for every action that's worth collecting data for. Or you could write a program that starts scanning the thread only when you're asking something from it. You could keep track of the post number of where it left off last time it is ran though. Either way it's just something I wouldn't do, but if you could do this it would be pretty impressive, not to mention the data you collect would be invaluable.
Post
Post #92 (isolation #12) » Fri May 11, 2018 2:01 pm
Postby yessiree »
why not just push what you have onto a separate branch and have other people look at it? git is literally the best tool for collaboration for situation like this
also, someone wise once said this: "premature optimization is the root of all evil". dont get too hung up on performance, focus on correctness and meeting your goals first. optimize only when necessary
Post
Post #105 (isolation #13) » Tue May 15, 2018 8:03 am
Postby yessiree »
probably better to stick to one html parser anyway
on another note, I upgraded my local to python 3.6.5 finally, and reinstalled all the packages. There seems to be an issue with requests after doing so. after some digging around I found out it was because gevent's monkey.patch_all() was running
Post
Post #114 (isolation #17) » Tue May 15, 2018 8:59 am
Postby yessiree »
yes I agree, and I also think that you should have zero expectation whatsoever on what the mod chooses to do when you cast a vote with broken format
"eh, whatever, the mod will probably realize that I'm voting XXX"
I dont think this is a good mentality to have, nor a behavior to be encouraged, at any rate
this is when you would consult with "the business people" on what "the clients" would want
Post
Post #132 (isolation #18) » Wed May 16, 2018 4:36 am
Postby yessiree »
I think I know what this is.
1. The input field that you're storing the counter has type=text, it should be type=number
2. JavaScript has problems representing floating point numbers
3. Shit hits the fan when you're trying to multiply it by 1 in the setInterval() function
suggestion (optional): since I don't think people will be staring at the pagetopper page constantly, instead of setInterval(), use setTimeout() and use the counter as the timeout should probably be cleaner
Edit: probably should not be converting string into numbers by multiplying it with 1, use parseInt() instead
Post
Post #139 (isolation #20) » Thu May 17, 2018 7:38 am
Postby yessiree »
In post 64, Psyche wrote:me and radiantcowbells found a bug in getNumberOfPosts() that makes it fail when there are unread posts in the thread. disgusting.
Post
Post #141 (isolation #21) » Thu May 17, 2018 9:03 am
Postby yessiree »
ok so, I fix the bug mentioned above, because that's needed for the pagetopper to work
also finished a prototype for the pagetopper. It takes care of the scheduling part in python, so effectively eliminating the need to do it on the client side /w JavaScript.
to run it, you need to initialize it with a thread, and the current page number of the thread, then call the run() function
I'm currently testing in a game im modding so we'll see how that goes lol
Post
Post #153 (isolation #26) » Mon May 21, 2018 10:57 am
Postby yessiree »
it occurred to me that we could just write batch scripts that run the python files, then WindowsTM users can download the source code with the .bat files, and simply run the batch scripts
this is much much simpler than the whole web app thing with flask
In post 153, yessiree wrote:it occurred to me that we could just write batch scripts that run the python files, then WindowsTM users can download the source code with the .bat files, and simply run the batch scripts
this is much much simpler than the whole web app thing with flask
it's worth serious consideration
This is a good idea. One of my big concerns with the web gui interface was that it never seemed like it would make for good UX no matter how good the UI could be done. In the scheme of trying to attract more users to what is now a bit of a "clunky" site that was a bit of an issue. Now instead of trying to spin up a server it's literally just clicking an icon.
i have the POC for this ready so will probably polish it then send it for review some time tomorrow
basically, as a user, you just need to:
1) download python
2) download the source code of this repo
3) change some settings in a text-based configuration file, such as thread url, username, pw, etc...
4) click on a .bat file to run a specific feature of donbot
Post
Post #171 (isolation #30) » Mon May 28, 2018 12:09 pm
Postby yessiree »
In post 169, davesaz wrote:Would it help if the "server" part was integrated right into the site?
I don't know if that's possible, just asking...
it's definitely possible, but it wouldn't benefit the average user at all, as it would only be useful for developers. So there isn't much value for something like that, especially given they are already short on resources, and the migration is probably the highest priority anyway
Post
Post #179 (isolation #32) » Tue May 29, 2018 3:09 am
Postby yessiree »
What roadblocks did you run into with JS? I've already written a vote counter in vanilla JS that can be ran in the browser console; I can show it to you if you'd like. I'm sure if I turned it into a proper web app and made use of all the libraries out there, the entire code base can be converted to JS pretty easily.
Post
Post #181 (isolation #33) » Tue May 29, 2018 7:31 am
Postby yessiree »
like HTML/DOM parsing? that's literally what JS is made for though. There might be semantic differences but I don't think you will have too much trouble with it since you already have experiences with tools like lxml and scrapy
but I like this google colab thing, it is also exactly what we're looking for, so I think this is the way to go?
Post
Post #184 (isolation #35) » Tue May 29, 2018 7:46 am
Postby yessiree »
or maybe are you running into an CORS issue? yea you are running into an CORS issue.
I think you are requesting a resource from a domain that does not have the 'Access-Control-Allow-Origin' header set (which is the case for any mafiascum.net pages), then you can only do so while on the same domain
Post
Post #186 (isolation #36) » Tue May 29, 2018 7:51 am
Postby yessiree »
unlike our python implementation, where we log in before sending any requests, and the 'request' package automatically attaches the session token for every requests we send afterwards
on the other hand, i dont think vanilla javascript has automatic session management. so you'd have to use a library for that, Axios.js for example, but I think if you followed the same model it will work
Post
Post #189 (isolation #38) » Tue May 29, 2018 8:18 am
Postby yessiree »
but then again, if we use google colab to run ipython notebooks we dont need any of this
I'm completely new to this but I think python-javascript resource sharing is possible, so you can do stuff like: getting web resources with python, then letting javascript handle any html parsing work, storing the result texts, then accessing those results from the python side again, etc...
pedit: ok, it'll take a lot of work though, i dont know how well the packages you used would translate to js (well the regex part should be fine, but not sure about the other ones)
Post
Post #208 (isolation #43) » Wed May 30, 2018 9:30 am
Postby yessiree »
In post 189, yessiree wrote:I'm completely new to this but I think python-javascript resource sharing is possible, so you can do stuff like: getting web resources with python, then letting javascript handle any html parsing work, storing the result texts, then accessing those results from the python side again, etc...
Post
Post #213 (isolation #44) » Thu May 31, 2018 8:48 am
Postby yessiree »
im just gonna jot down a list of things im doing so we dont have duplicated efforts
1. Make a more efficient version of pagetopper that leverages the google colab vm
2. Integrates it with ur version of votecounter
3. UI mocks
It's a NodeJS app. So you would need to install Node to run this.
I've finished authentication. Fortunately NPM's request package is similar to the requests package we used in python, so we get persisting cookies between requests (authenticate once and done). This means everything that was done in python is possible in NodeJs, so converting the codebase to Js should only be a matter of time.