robots.txt

[Tamuz]

Does MS have one?
I've done a brief search and sent a few crawlers around but I can't find any, but the site knows Google[Bot] and Bing[Bot] are watching. Where would I go to not be a dirty cracker and abide by good ethics here?

[Mr. Flay]

I have no idea what your last sentence is asking.

[Tamuz]

If you're a good person with a crawler you should abide by webmaster's requests which they show using ./robots.txt
I don't want to be a bad person, but this site doesn't have anything of the type that I can find.

[Mr. Flay]

robots.txt isn't really what you're after anyway. You want an API or a lightweight interface for scraping, right?

phpBB handles most of its security through user roles, not robots.txt. Guests and Bots are locked out of some things.

[Tamuz]

I have the means to scrape. I was looking for the admin control flow. Your point re phpBB answers that, just means that if I want to scrape something from a non-guess forum I'll have to do so via my actual account, I also understand that there is a reason bots are locked out of things, so I probably shouldn't do so. Question answered as well as I need, thanks.

[Mr. Flay]

We can also create a category of userbot that has similar privs to you, but doesn't have to use the regular skins, etc. Might help, though an API is better.

Won't be soon, though. We need to upgrade the forum anyway.