Forum Moderators: open

Message Too Old, No Replies

A pair of WordPress crawlers

No idea what purpose they serve

         

GaryK

6:10 pm on Feb 22, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



WordPress-Do-P-/2.6.2
and
WordPress-B-/2.6.2NOFALSE

Neither one read robots.txt. Both crawled a bunch of pages related to kit reviews.

I couldn't find anything about either bot here or elsewhere.

Might anyone here have some idea what purpose these bots serve?

wilderness

9:24 pm on Feb 22, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I believe the sole purpose of their existence is to eat 403's.

With the above in mind?

75.126.12.zz - - [29/Dec/2007:09:53:50 -0600] "GET /MyFolder/MyPage.html HTTP/1.1" 200 13184 "-" "WordPress/2.3.1"

One year later:
72.32.122.zz - - [26/Dec/2008:22:59:20 +0000] "GET /SameFolder/SamePage.html HTTP/1.1" 403 998 "-" "WordPress/2.6.2"

The only conclusion that I'm able to surmise (considering this particular page) is that some internet forum has a participant who placed a link to the page.

incrediBILL

10:10 pm on Feb 22, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



The only purpose I could see for WordPress to crawl would be link checking.

If it goes beyond a page or two, no clue.

GaryK

1:06 am on Feb 23, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks. I'll keep an eye on them. For now they haven't done anything I'd ban them for. They appeared to be checking a bunch of kit reviews which is fine with me cause I get lots of inbound links that way.

leadegroot

2:37 am on Feb 23, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I suppose its possible that somewhere there is a wordpress plugin that checks links in posts are still valid?

And indeed - a quick bit of googling found at least one - but a quick check of the code shows that it is setting the agent to "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)", so its not that one. :)

I suppose someone could have customised it...