Forum Moderators: open
63.241.61.8 - - [09/Jan/2005:01:37:22 -0600] "GET /my_dir/my_page.html HTTP/1.0" 200 3697 "-" "ConveraCrawler/0.4"
Not sure who or what this is? Wondering whether to ban it. Doesn't seem to be any info here. Doing a google search doesn't help much other than bringing up other sites' log pages.
Any info would be appreciated.
63.241.61.8 - - [20/Jan/2005:15:23:14 -0800] "GET /robots.txt HTTP/1.0" 200 1751 "-" "ConveraCrawler/0.5 (+http://www"Don't ask me why the link stops at 'www'.
But I do slam the doors for 24 hours on any bot that hits the site faster than a given rate.
And this little beastie gets blanked every two or three days. It hits the site spidering at 10+ pages a second. It rarely gets more than 50 pages before the ban activates.
Then it comes back at most a week later to try again.
If it spidered at a decent rate, it could have been through the whole site by now.
Definitely an impolite spider.
To ban or not?
As a rule this forum has not a whole either gathered or determined an consensus of good or bad.Rather, it has been a tradition the each webmaster determines personally what bots are either beneficial or detrimental to their own sites (s).
There is a lot to consider in making the determination as to ban or not.
Your site, the relevance of the engine to your site, how behaved the bot is, does it request robots.txt, does it download files too fast, to name a few.
To ban or not?
To add to what Glenn provided. . . a parallel.
If a thief breaks in to your home, you have but two options?
1) interact with that thief.
2) hide under the bed until the police come and await their advice and assitance. ;)
Don
How can I ban them? (windows server)
If your referring to shared hosting on a Apache:
[webmasterworld.com...]
