Forum Moderators: open

Message Too Old, No Replies

Picsearch bot loves 403 forbidden requests

         

SEOPTI

3:15 pm on Nov 10, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



This picsearch bot is ultra dumb, I can't imagine why MSN works with them.

psbot/0.1 (+http://www.picsearch.com/bot.html)"

This bot hits 20k+ 403 forbidden at my site and continues to do so. What a dumba**.

Mokita

11:34 pm on Nov 11, 2006 (gmt 0)

10+ Year Member



Have you tried giving it access to robots.txt and disallowing the psbot? Works well for me. I have it disallowed in all our sites, and I rarely ever see it now.

User-agent: psbot
Disallow: /

Possibly because it isn't finding robots.txt, it thinks it's OK to crawl and index your site, so keeps on trying.

SEOPTI

6:47 pm on Nov 12, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks for the info. I need to start using robots.txt, I ban them first because most of them do not respect robots.txt

The next poor coded pest in my logs I see is:
CazoodleBot/Nutch-0.9-dev (CazoodleBot Crawler)

This pest tried to crawl 40k 403 sites.