FlightDeckReportsBot

update

[keyplyr]

RE: older thread [webmasterworld.com...]

I've been watching this bot. Althought it really hasn't been much of a pest, I was curious to see if in fact it would follow robots.txt standards especially after the bot owner chimed in and said he'd fix it.

I haven't been able to get their info page to load: www.flightdeckreports.com/pages/bot , but reading this from the home page I decided not to allow access.

Flight Deck Reports is a next generation data mining service. We provide a cloud-based group computing platform in order to enable decision makers, research analysts and writers to process vast amounts of data found in the source code of web documents. Our goal is to enable discovery and measurement of the trends, business relationships, and Internet technologies that impact your business

So with the announcement from the bot owner (above older WW thread) that robots.txt is now being respected, I added:

User-agent: FlightDeckReportsBot
Disallow: /

Today it came, requested robots.txt and promptly disobeyed requesting index.html.

Now blocked.

[Staffa]

Though no bots get into my sites, any bot having a try and their site mentions "data mining" gets blocked.

I only allow bots which could send me visitors, anything else is a waste of my money for the benefit of theirs.

[keyplyr]

I only allow bots which could send me visitors

Sometimes there are other benefits besides direct traffic. However, this bot is not in that category for my interests.

[Staffa]

Sometimes there are other benefits besides direct traffic.

Pardon my ignorance but what would these other benefits be ?

[keyplyr]

Pardon my ignorance but what would these other benefits be ?

*Benefit* of course is subjective. What is of benefit to a specific business plan may also change periodically, so your ignorance may in fact be appropriate for your niche.

For me - A year ago I was very reluctant to get involved with any social networking, until of course I started to see my market share drop to those that did. So I began a campaign to proliferate every social site with our company/product info. I've received multiple orders because one of our product pages went viral at a social site. We are also now are seeing significant direct and indirect traffic by users adding our links to their personal sites due to the social presence.

I also see high value in defensive branding/product presence, even though the resource will never bring direct traffic. I block most traffic from China & Upper Asia but I want our company & products represented in the regional SERP.

I allow several bots because they feed other resources that may bring traffic/sales (example: Speedybot, Exobot.)

I allow several bots to crawl and data-mine because they report traffic, website reach, web presence, etc. These independent reports are valuable when I sell advertising.

The list continues...

[Staffa]

Thank you keyplyr for taking the time to spell it out for me, it is indeed instructive.