Welcome to WebmasterWorld Guest from 54.163.158.163

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

Does amazon have a crawler or are all Amazon AWS IPs fair game?

     
9:53 am on Nov 2, 2012 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 13, 2005
posts: 128
votes: 0


There is no valid reason that someone should hit my site from an amazon AWS box. I'd love to just block the whole of amazon AWS/etc straight at the firewall. But I'm worried that I'll somehow block an amazon crawler/spider and that would possibly impact the sales I do on amazon.

Anyone have any suggestions/links for this sort of thing? I want to allow, if it exists, the actual amazon company to browse the site but block amazon's AWS/etc customers who spin up a server then scrape/spam with it.
6:09 pm on Nov 2, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5459
votes: 3


Pfui has a long and dedicated thread [webmasterworld.com]
6:14 pm on Nov 2, 2012 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 13, 2005
posts: 128
votes: 0


Yes, I've seen that - thank you. I'm not sure though whether Amazon has its own crawler?
6:25 pm on Nov 2, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5459
votes: 3


Nor am I, unfortunately their hosting business customers have a proven record of abuse, as does Amazon AWS' acceptance of these customers.

Perhaps the Amazon FAQ (NOT Amazon AWS) provides the answer.

The easiest explanation is within your raw visitor logs and the image references to your own Amazon pages.
What are those IP's?
Simply separate them from the Amazon AWS IP's.
8:09 pm on Nov 2, 2012 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:7563
votes: 244


It used to be call "A1" but haven't seen that UA for a while.

Then there were versions of "AWSpider" (AWSpider 0.3.2.12 last hit my logs in 2011.)
9:35 pm on Nov 5, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 26, 2006
posts:1619
votes: 0


I've busted several amazon "bots" scraping our site for images.... makes me wonder if they are actually stealing product images from sites for their own use.
10:52 pm on Nov 5, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5459
votes: 3


bots don't exactly leave resumes ;)

Harvesting, plagiarizing or simply indexing, who knows the why?

The AWS customers hit us all, that's why the long threads exist.
5:54 am on Nov 6, 2012 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:7563
votes: 244



The AWS customers hit us all, that's why the long threads exist.

Yes, but we're discussing "Amazon" bots.
6:57 am on Nov 6, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5459
votes: 3


My bad, hope it's just a full moon ;)
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members