TailsweepBlogCrawler/Tailsweep

[JAB Creations]

It doesn't seem to be an actual search engine though rather a 'hey I exist, come look at my services' sort of request as there was only a single request for the entire month.

79.136.112.nnn - - [01/May/2009:17:47:09 +0000] "GET /robots.txt HTTP/1.0" 200 761 "-" "TailsweepBlogCrawler/Tailsweep-2.6-SNAPSHOT (http://www.tailsweep.com/; bot at [tailsweep] dot com)"

Thoughts on this "spider"?

- John

[edited by: incrediBILL at 2:01 am (utc) on July 17, 2009]
[edit reason] Obscured IPs [/edit]

[GaryK]

TailSweep is an advertising network for blogs and social media. The SNAPSHOT part of the user agent string likely means this particular bot is making screenshots/thumbnails of the page it took. It might also be attempting log spam. That is, hoping your logs are public so their link will get indexed.

[JAB Creations]

Thanks Gary...I don't make my logs available to the public though I have seen log spam before though I don't actually consider this log spam (not if you compare it to some of the stuff that bots have done in the past!)

Is it worth allowing or not?

- John

[marcusherou]

Hi Guys.

The Tailsweep crawler looks for blogs to be indexed in our search engine.

We both have an advertising network for blogs and a blog search engine.

The snapshot part is just a versioning hint. 2.6-SNAPSHOT basically means version 2.6 development branch.

Hope it makes sense

Cheers

//Marcus Herou, CTO Tailsweep

[GaryK]

Welcome to WebmasterWorld, Marcus.

Thanks for letting us know about your crawler.

I'd like to suggest selecting a different term for your versioning hint as many of us here will ban a crawler simply for including such a term. You know, if it looks like a screenshot tool we'll treat it as one.

Also, the URL in the user agent string really should lead to a page that describes what your bot does and how we can restrict what it does on our sites.

Thanks again. :)