homepage Welcome to WebmasterWorld Guest from 54.211.201.65
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Idbot
IDBOT hammered our site, now banned
JohnCanyon




msg:3382314
 5:21 pm on Jun 29, 2007 (gmt 0)

This is a nasty bot that requested over 40 pages per second. We quickly added it to our .htaccess and encourage others to do likewise.

According to BOT website:
ID-Search.org is a research project started by a group of Russian candidates from the cities of Saint-Petersburg, Nizhnii Novgorod and Novosibirsk.

This BOT does NOT follow robots.txt so only way to stop it is to either ban the IPs at the firewall level or add its agent to your .htaccess

Companies site:
IDBot can recognize the directives from robots.txt files only partially, which is the result of the scantiness of our resources. Full support of robots.txt will be launched soon.

For quick reference here is our .htaccess entry.
RewriteCond %{HTTP_USER_AGENT} ^IDBot/1\.0 [NC,OR]

Cheers,

JC

[edited by: JohnCanyon at 5:22 pm (utc) on June 29, 2007]

 

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved