SBIder/0.8-dev

Anyone getting this?

[youfoundjake]

SBIder/0.8-dev (SBIder; [sitesell.com...] [support.sitesell.com...] hitting the site.
I would block it how?
RewriteCond %{HTTP_USER_AGENT} ^SBIder/0.8-dev [OR]

side note [OR], is that a conditional statement? once there are no more [OR]s, does that mean to continue on in the .htacces?

[volatilegx]

I'd use

RewriteCond %{HTTP_USER_AGENT} ^SBIder [OR]

To catch more of them, assuming they change version numbers.

[bobothecat]

side note [OR], is that a conditional statement? once there are no more [OR]s, does that mean to continue on in the .htacces?

I'm sure Jim will correct me if I'm wrong... :) , but I beleive the "[OR]" statement is used to tell .htaccess to go the the next directive. ( "or" something like that ).

[youfoundjake]

Perfect, thats what I though about the operator, but wasn't sure.
a week ago i was playing with the file and forgot to check afterwords and left the site down for 14 hours. oops
I will make it more general to catch different versions.
How do you guys feel about a general Mozilla version also being prevented?

[Mokita]

I disallow SBider using robots.txt which it seems quite happy to comply with:

User-agent: SBIder
Disallow: /

My preference is to block compliant bots that way, rather than increase the size of my .htaccess file unnecessarily and slow the server response for normal visitors.

[youfoundjake]

I have thought about that. I have read in these forums that there are bots that once they see that they are disallowed in the robots.txt file, they will make one last grab into the site for a file. So I thought I would just stop it cold. I don't get enough visitors yet to worry about server load so maybe I'm lucky in that aspect.

[incrediBILL]

FWIW, SBIder seems to operating from a narrow range of IPs from 64.34.145.194 - .198

Not sure if anyone is seeing it outside this range of IPs.

[malachite]

A quick update on this one - hadn't seen it for quite a while, came back yesterday and it isn't obeying robots.txt

[Mokita]

This bot has been back in our sites in the last few days too. It asks for robots.txt but then, unlike previously, it immediately asks for one, or at most, two pages.

This is not a burden, but trust and the principle is being flouted. So now it will eat only 403s, if it asks for anything but robots.txt.

[starhugger]

SBIder is a crawler used by Site Build It! (SBI). They're a legitimate website development and marketing methodology/service. Their spider is fairly new and gathers information about what websites are out there, their topics and other information, which can help their subscribers make informed choices when building their (also legitimate) websites. The founder of SBI, Ken Evoy, is a strong believer in not using dirty tricks or questionable methods in website building and marketing, so I can't imagine that SBIder would cause any problems for anyone. If you feel it's behaving badly, I'm sure Ken would want to be informed about it. You can reach him at SiteSell.com.

Starhugger

[Pfui]

I appreciate your loyalty, Starhugger, but no matter how nice Ken Evoy [google.com] is or respectful (or not) his bot, there's absolutely no benefit to my site(s) from his using my bandwidth to spider my stuff for his purposes (including to glean market data to sell to his customers).

That's more than enough to make any bot block-worthy in my book.

Plus SBIder is not really new -- it's been a topic of discussion [webmasterworld.com] for going on a year or so. And we're not the only ones talking about it, so all of this SE-findable thread is 'no PR is bad PR' manna from heaven to a marketer like Evoy.

(Finally, from the Nit-Pick Dept.: His never-ending home page looks like something straight out of Multi-Level Marketing 101. Sorry but UGH.)

[starhugger]

It is your choice, of course, to decide whether or not to block any crawler, whatever your reasons. :-) My intention was to say this is not simply a tool of the "bad guys" whose only purpose is to hack your system, steal your content or some other unscrupulous purpose. And also to say that if someone has noticed it behaving badly or questionably, I would invite them to report this to SBI.

For the record, beyond the fact that SBIder's gathered info is included as one part of the many SBI tools and very broad development system, he doesn't sell the info to his customers, per se.

However, to each their own. :-)

Starhugger

[keyplyr]

I disallow SBider using robots.txt which it seems quite happy to comply with... My preference is to block compliant bots that way, rather than increase the size of my .htaccess file unnecessarily and slow the server response for normal visitors - Mokita

I had it in robots.txt also until it disobeyed it, so now it's banned via mod_rewrite.

[Mokita]

I had it in robots.txt also until it disobeyed it, so now it's banned via mod_rewrite.

My experience exactly - see message # 9 in this thread above.

[Staffa]

I didn't even bother with robots.txt.
The bot from this site, like from any other site that comes crawling without a benefit to my site in return gets banned.