Forum Moderators: open

Message Too Old, No Replies

tScholarsBot

"tScholarsBot [beta] is now up and ready..." and checks robots.txt.

         

pendanticist

7:41 am on Jan 17, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



[tscholars.com...]

it has been running around the internet crawling for educational websites/webpages yesterday night. So far it has index approximately 50,000 web pages. However, visitors wouldn't be able to test it out until March. Screenshot will be out soon.

tScholars.com sole purpose is to establish an educational and well managed website while at the same time creating an atmosphere of fun and excitement for the scholars, with reliable educational resources to please the online scholars.

Works for me.

DoctorIanJ

4:01 am on Mar 1, 2005 (gmt 0)

10+ Year Member



This bot is not polite - it read my robots.txt file and then went ahead and scanned the disallowed directory.

Their contact email doesn't work.

And they have changed the Host IP from 205.209.170.130 early in February to 216.69.161.175 as of Feb 28th. They should be scholarly enough to follow the rules and have an email that works.

pendanticist

4:31 am on Mar 1, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yup. That and the bulk of the site is dead now. I banned it via IP Number as it does not obey robots.txt. Looked good on the surface, but it acts just like a rouge.

Bye-bye tScholarsBot. Don't go away mad, just go away...

While I'm thinking of it. There are just too many 'bots' out there and one day the entire Internet is going to be one big highway of nothing but bots and those trying to control them. Hell, I get at least two a week and trying to protect my work is getting to be much more of a hassle now-a-days to be sure.