homepage Welcome to WebmasterWorld Guest from 54.197.65.82
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Idbot
IDBOT hammered our site, now banned
JohnCanyon

10+ Year Member



 
Msg#: 3382312 posted 5:21 pm on Jun 29, 2007 (gmt 0)

This is a nasty bot that requested over 40 pages per second. We quickly added it to our .htaccess and encourage others to do likewise.

According to BOT website:
ID-Search.org is a research project started by a group of Russian candidates from the cities of Saint-Petersburg, Nizhnii Novgorod and Novosibirsk.

This BOT does NOT follow robots.txt so only way to stop it is to either ban the IPs at the firewall level or add its agent to your .htaccess

Companies site:
IDBot can recognize the directives from robots.txt files only partially, which is the result of the scantiness of our resources. Full support of robots.txt will be launched soon.

For quick reference here is our .htaccess entry.
RewriteCond %{HTTP_USER_AGENT} ^IDBot/1\.0 [NC,OR]

Cheers,

JC

[edited by: JohnCanyon at 5:22 pm (utc) on June 29, 2007]

 

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved