Welcome to WebmasterWorld Guest from 54.158.65.139

Forum Moderators: Ocean10000 & incrediBILL

SMBot

Crawling from Amazon.com in South Africa

   
8:44 pm on Dec 24, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



SMBot/1.1 (www.specificmedia.com)
216.182.230.228
domU-12-31-33-00-03-2E.usma1.compute.amazonaws.com
-----
OrgName: Amazon.com, Inc.
OrgID: AMAZO-4
Address: Amazon Development Centre South AFrica

Did not request robots.txt. Not sure if it needs to. It appears to be related to affiliate advertising.

8:12 pm on Dec 25, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Update: Bill commented about this on his blog.
5:11 am on Jan 9, 2007 (gmt 0)

5+ Year Member



This bot is worse than you think.

User-Agent: "SMBot/1.1 (www.specificmedia.com)"

I'm a little ticked off. Specific Media has using SMBot to rip complete web sites, repeatedly. In the last couple days it hit 2 of my sites several hundred times each. Specific Media is using this bot to data-mine your site for information that can be used by their advertisers and advertising network. They are making a profit off your hard work! Without your permission of course. Can someone say Digital Millennium Copyright Act violations? SMBot completely disregards Robots.txt standards as well. At first their bot was crawling around without a user agent. (Fellow bot-hunter IncrediBILL has more info on SMBot.)

[edited by: volatilegx at 8:59 pm (utc) on Jan. 9, 2007]
[edit reason] trimmed post to remove call to action [/edit]

6:59 am on Jan 9, 2007 (gmt 0)

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



hybrid,
As previously suggested, redirecting bots is a BAD idea.

plain old denial is much more effective.

1:40 pm on Jan 9, 2007 (gmt 0)

5+ Year Member



wilderness...please elaborate.
2:31 pm on Jan 9, 2007 (gmt 0)

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



redirecting "bad" bots to either their own websites, anothers website or alternative images is a bad practice.

Any attempt at such redirecting of bad bots simply causes more problems for yourself in the future.

Suggest changing your lines to a simple denial of access.

RewriteCond %{HTTP_USER_AGENT} ^SMBot [NC]
RewriteRule .* - [F]

Jim (as I recall) has an alternative that sends them off to a page which uses less kb's than even a simple 403.

Don

7:26 pm on Jan 9, 2007 (gmt 0)

5+ Year Member



Hmmm...you may be right...It's just getting ridiculous with these guys though. Hammering people's web sites with a bad bot is a bad practice too.

I'll revert to a 403. I'd be interested in Jim's alternative method.

7:32 pm on Jan 9, 2007 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



I don't send them away and I don't send them 403s either.

I feed them a page with breadcrumbs so I can see where the data shows up if it's ever indexed by a search engine.

10:49 pm on Jan 9, 2007 (gmt 0)

5+ Year Member



Interesting...what do you mean by breadcrumbs?
7:56 am on Jan 10, 2007 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Breadcrumbs - a piece of data so unique that it has NO search results and contains a session ID that ties the data back to the original IP and user agent.

For instance, a nonsense code to search like "AAVVQQAA" plus the key to link them back to the crawling event. Hyphenate the code so the search engine will see the first part as uniquely searchable so it would look like "AAVVQQAA-12276021092" or however you do it.

Looks kind of like a part # when it's assembled :)

I never show breadcrumbs to search engines, just cloaked to the rest of the world to track my data.

I use CSS to hide them in the browser, people never see these, but the crawlers strip out the html and VOILA! they are exposed on the scrapers websites.

[edited by: encyclo at 1:12 am (utc) on Jan. 12, 2007]
[edit reason] fixed typo per request [/edit]

3:03 pm on Jan 10, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



> I never show breadcrumbs to search engines, just cloaked to the rest of the world to track my data.

Yet another fantastic reason to cloak! ;)

8:51 pm on Jan 10, 2007 (gmt 0)

5+ Year Member



IncrediBILL: Brilliant. :)

volatilegx: Yes, very true. I'd say that's definitely a valid use.

9:06 pm on Jan 11, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



When I initially learned about this it was marked, for your eyes only. I think posting it in public will ultimately be self-defeating. Scrapers now know to check for CSS that makes an element seem invisible. From there it's not a lot of work to strip out anything that's invisible.

Please always remember it's not just white hats who read this forum! :)

9:27 pm on Jan 11, 2007 (gmt 0)

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



I think posting it in public will ultimately be self-defeating.

Gary,
Many of us are on agreement along these lines, however closing our doors and insights to others, makes it quite impossible to share something that most webmasters do not utilze. Seems a do or die dilema?

This a good example of my first awareness of such monitoring of Webmaster World some three years ago.

[webmasterworld.com...]

Personally, I'm unable to recall when a bad-bot was able to spider my entire sites successfully. Nor am I aware of it (spidering) being done in cloaked and/or unidetified manner.

I'm not saying that complete spidering doesn't still occur, only that successful use of rewrites with htaccess has been successful in deterance on my sites.

Should we close our doors to newcomers willing to learn and create a "good ole boys" method of communication?

Don

10:10 pm on Jan 11, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Should we close our doors to newcomers willing to learn and create a "good ole boys" method of communication?

No of course not. :)

Still, I feel so exposed when information like this becomes common knowledge.

I'm a conflicted man. No doubt about it. ;)

10:16 pm on Jan 11, 2007 (gmt 0)

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



I'm a conflicted man. No doubt about it.

Gary,
Jim's in charge of the dispensary ;)

[webmasterworld.com...]

If no comfort there?
Try here:
[herbalrescue.co.nz...]

If no comfort either location?
try some regular "herb" ;)

Don

10:26 pm on Jan 11, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I could have stopped with Jim. I loved that thread. But since you offered so many choices I think I'll go directly from the top of the list to the bottom. ;)
11:29 pm on Jan 11, 2007 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



I think posting it in public will ultimately be self-defeating

Gary, what was for YOUR EYES ONLY was the specific codes I'm using, that's still just for you! :)

Early on I was just experimenting with the tactic and have since shown the results of how this works at both SES and PubCon, so the cat is out of the proverbial bag. I also came up with other ways of implementing or randomizing, including not using CSS, so that it's virtually impossible for scrapers to code for this technique.

For instance, you can even embed a specific visible phrase and use it to track content such as "aardvark and centipede farts" which currently return no results. You could make it completely visible in small type on the page such as "Silly factoid#12276021092: Did you know that the aardvark and centipede farts?"

Then just hit the SE's looking for the exact phrase "aardvark and centipede farts" and "silly factoid" and sure enough the related code pops out.

The only problem I've run into since busting some scrapers is that they took a page out of my book and started using NOARCHIVE so you can't snoop their cloaked pages in search engine cache. The solution to this problem was to bind the session ID code to a specific word so that Google will display the word plus the session code in the snippet. That's why I always have a phrase plus a keyword so that if the scraper scrambles the content, which many do, I can pull it back together in the results.

So what next, they start looking for long numbers and filter them out?

Fine, I can switch to HEX or BASE36, or a completely alpha variant so it will look like a word instead of a number.

Besides, scrapers are like rats, the smart ones take your cheese and leave an empty trap, but we can still enjoy snaring all the stupid ones while it lasts.

[edited by: incrediBILL at 12:07 am (utc) on Jan. 12, 2007]

11:42 pm on Jan 11, 2007 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Scrapers now know to check for CSS that makes an element seem invisible.

Maybe it's a font tag changing the color opposed to CSS
Maybe I embedded it in a 1 pixel iFrame lurking in the page
Maybe I faked the code honeypot-style into the URL designed for scrapers to followed to a special page so I can search for anyone that links to "you_stupid_scraper_12276021092.html"
Maybe the code is hidden in the title as "PAGE 122760 of 21092".
Maybe it's all of the above!

You never know what I'll do next ;)

The point is that there are many tricks and this is just scraping (pun intended) the surface.

[edited by: incrediBILL at 12:08 am (utc) on Jan. 12, 2007]

1:12 am on Jan 12, 2007 (gmt 0)

5+ Year Member



Again, Brilliant. thank you. I'm current refining my anti-bot strategy and will be working on a unique implementation of that concept.
1:14 am on Jan 12, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



You never know what I'll do next

You and I both know how true that is. ;)
1:35 am on Jan 12, 2007 (gmt 0)

5+ Year Member



Bill, Don, Gary, Dan: I know I'm new to this forum, but thank you all for your open advice...You guys rock. I'm already a pretty good bot-fighter, but there is a ton more to learn and you guys have been at it longer. It seems like the more I learn the more I realize there is to know. :) Again, thanks.
5:14 pm on Jan 12, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



the more I learn the more I realize there is to know

The sign of a wise /(wo)?man/.

(sorry for the regex, couldn't resist)

8:08 pm on Jan 12, 2007 (gmt 0)

5+ Year Member



LOL. :)
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month