Forum Moderators: open

Message Too Old, No Replies

SlugBug Spider

a new bot on the block

         

Dijkgraaf

1:01 am on Jul 9, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



UA: SlugBug Spider/0.1 beta (SlugBug.com search engine; http://www.slugbug.com)
robots.txt: Yes

Fetched only a single page after getting robots.txt

Their web page is rather uninformative (but then I loaded it with JavaScript disabled, and the page is written in such a way as to need it).
Seems to be a news aggregation of some sort from it's Meta keywords.

[edited by: Ocean10000 at 1:58 am (utc) on Jul 9, 2010]
[edit reason] Beaking link to make ua readable [/edit]

keyplyr

2:30 am on Jul 9, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Saw it also, same UA string, same behavior. They got 403 after the robots.txt request due to whitelisting. They currently use the major search engines as their search so their crawl purpose is yet unknown.