Burf

Ignore Robot.txt

[frontpage]

Just had a new robot caught in our bad spider trap.

Just added to the firewall for our servers.

A bad robot hit /spidertrap/ 2006-01-30 (Mon) 02:41:26 
address is 212.1.139.52, agent is Norbert the Spider(Burf.com)

[frontpage]

Just logged another hit in the spider trap by this nuisance.

The following ip just got banned because it accessed the spider trap.
212.1.151.182
Norbert the Spider(Burf.com)

[burf2000]

Sorry about that, its is my spider. Sadly it must not be reading your robot.txt very well

[frontpage]

Just got hit again and routed it at the firewall.

212.139.6.126
Norbert the Spider(Burf.com)

[frontpage]

Our robots.txt is not too complex.

User-agent: *
Disallow: getout.php
Disallow: /img/
Disallow: /trap/

User-agent: Teoma
Crawl-delay: 30

User-agent: psbot
Disallow: /

User-agent: bumblebee@relevare
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: NPBot
Disallow: /

User-agent: SurveyBot
Disallow: /

User-agent: SlySearch
Disallow: /

User-agent: [almaden.ibm.com...]
Disallow: /

User-agent: e-SocietyRobot
Disallow: /

User-agent: Nutch
Disallow: /

[burf2000]

I will look in to this tonight.

[Dijkgraaf]

Hi frontpage
Is that first disallow
Disallow: getout.php
Exactly how you have it in robots.txt?
If so it should actually be
Disallow: /getout.php

[thetrasher]

frontpage wrote:
A bad robot hit /spidertrap/ 2006-01-30 (Mon) 02:41:26

frontpage wrote:
Our robots.txt is not too complex.

User-agent: *
Disallow: getout.php
Disallow: /img/
Disallow: /trap/

Maybe you should add the following line to your robots.txt:

Disallow: /spidertrap/

[frontpage]

That is how it is.

Don't forget there are many websites on this server with different paths to the spider trap.

Some are 'trap' and some are 'spidertrap'.