Forum Moderators: open

Message Too Old, No Replies

bloglines

crawler?

         

Bewenched

3:45 am on Apr 10, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



65.214.44.29 (Unknown)4/8/2008 21:31/robots.txt512200 - OK

Gives No Referrer
Gives no user agent.
grabbed the robots.txt

resolves to
crawler.bloglines.com

Ocean10000

5:27 am on Apr 10, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Bloglines is a FREE online service for searching, subscribing, creating and sharing news feeds, blogs and rich web content.

Question for you. Do you have an RSS feed someplace on your site?

It will download existing RSS feeds, and look for new ones to add to its list. From my reading of there site, a member of there site will subscribe to one of your feeds, so it can be viewed by there service. It does check robots.txt. But other then that I do not know. I block it since it does not supply a User-Agent when reading Robots.txt.

[edited by: Ocean10000 at 5:32 am (utc) on April 10, 2008]

Bewenched

5:36 am on Apr 10, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Nope .. it asked for my robots.txt file

incrediBILL

7:11 am on Apr 10, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



The crawler.bloglines.com is owned by and used by Ask.com for all sorts of tasks including making screen shots that show up in Asks SERPs.

[edited by: incrediBILL at 8:40 am (utc) on April 11, 2008]

Bewenched

4:25 am on Apr 13, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Here's the UA string it created the most recent time it came through.

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a1) Gecko/20070308 Minefield/3.0a1