flatlandbot/allspark

[wilderness]

Claims to be a 3rd party Vertical Search for customers only.
(Note; I've diabled links in UA)

robots and root on two sites.

74.62.161.zz - - [19/Feb/2009:13:55:29 +0000] "GET /robots.txt HTTP/1.1" 200 5023 "-" "flatlandbot/allspark (Flatland Industries Web Spider; [flatlandindustries...] dot com/flatlandbot; jason @ flatlandindustries dot com)"

[dstiles]

Been blocking this for some time. My IP & UA compare to yours.

Their web site says (in H tags)...

Providing New Revenue Streams for Web Publishers
... search solutions for business, education & government
... Your Own Vertical Search Engine
... Valuable Service for Your Users
... Monetize Those Search Results

In other words, make money from the sites we scrape.

[wilderness]

I've added their IP. however they denied themselves with the word "spider" ;)

[Pfui]

From last Fall, an earlier incarnation of their (still) ridiculously long, name ID-heavy UA:

great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; [flatlandindustries.com...] [e-mail])

Asked for robots.txt, then promptly ignored it.

Looks like the same HOST, too, because my info appears to match the OP's IP (if the last two digits are this year minus 1934):

rrcs-74-62-161-zz.west.biz.rr.com

[edited by: Pfui at 1:57 am (utc) on Feb. 20, 2009]

[keyplyr]

My experience is that flatlandbot has always obeyed robots.txt and I have not had any issues with it. However since they are not forthright with bot info I have since denied this UA.

[enigma1]

I ban them by hostname, so it should take care of all their ips. As far I can see they don't offer a public search engine on their site and I agree with dstiles, seems they're scrapping content to service their own little business.

[Pfui]

They're baaack --

crawler.flatlandindustries.com
flatlandbot/baypup (Flatland Industries Web Spider; [flatlandindustries.com...] [e-mail])

robots.txt? YES

Methinks six -- count 'em, SIX -- mentions of "flatland" in every single hit is egomaniacally over-the-top (not to mention added logspam).

[GaryK]

I've seen a ton of UAs that included flatlandbot.

Like you this is the most recent one:

flatlandbot/allspark (Flatland Industries Web Spider; [flatlandindustries.com...] [e-mail])

Others from 2008 include:

flatlandbot/baypup (Flatland Industries Web Spider; [flatlandindustries.com...] [e-mail])

flatlandbot/flatlandbot (Flatland Industries Web Spider; [flatlandindustries.com...] [e-mail])

flatlandbot/flatlandbot (Flatland Industries Web Spider; [flatlandindustries.com...] [e-mail])

great-plains-web-spider/flatlandbot (Flatland Industries Web Robot; [flatlandindustries.com...] [e-mail])

great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; [flatlandindustries.com...] [e-mail])

great-plains-web-spider/gpws (Flatland Industries Web Spider; [flatlandindustries.com...] [e-mail])

EDIT: Oops. The baypup one was last seen on Jan 31, 2009.

[edited by: GaryK at 11:58 pm (utc) on Mar. 10, 2009]

[wilderness]

Gary,
Fortunately, the they don't seem capable of getting past the "spider" hump ;)

[GaryK]

Yeah, I know you block anything with spider, crawler in it, LOL.

[baypup]

My name is Jason and I live in Kansas City. This is my search engine, it has been in a lab in Kansas City for a couple years now. All the pages fetched by 74.62.161.76 and 74.62.161.78 are searchable by the public at www [dot] baypup [dot] com. We have a 12 million page web index and are planning a 500 million page index. We would like to be able to crawl your site. Regarding This:

>Been blocking this for some time. My IP & UA compare to yours.

>Their web site says (in H tags)...

>In other words, make money from the sites we scrape.

We make money (laugh! not yet) the same way Google and Yahoo do. Our beta ad delivery network is at: ads [dot] baypup [dot] com.

Apologies if my bot was causing anyone problems.

All my bots obey robots.txt, and if they don't I would like to know about it.

Call my NOC phone if I can help - 816-309-1463

[blend27]

Welcome to WebmasterWorld Jason!

NetName: RCWE

Boy, I have tried to resolve several BOT Issues with RR(RoadRunner?), -Scooby-Doo is my witness:)

74.62.0.0/16 is banned on my end due to masive Site Abuse.

But it's just me.

[keyplyr]

New UA today:

74.62.161.76 - - [19/May/2009:07:42:50 -0700] "GET /robots.txt HTTP/1.1" 200 3907 "-" "baypup/1.1 (Baypup; http://www.baypup.com; [e-mail])"