homepage Welcome to WebmasterWorld Guest from 54.226.213.228
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
1900 hits from images.yahoo.com
slipkid




msg:4493626
 6:04 am on Sep 11, 2012 (gmt 0)

Yesterday, my website
came under continual attack (DoS?) from a yahoo referrer/user agent listed below. The hits occurred every 10-15 seconds and were continuous when they stopped in the early am. At least 1900 hits (I think).

The is the referrer/ua. (URI etc. changed)

24.190.103.173 - - [07/Sep/2012:02:57:48 -0400] "GET
/cgi-bin/referers.cgi?http://images.search.yahoo.com/images/view;_ylt=A0PDoTHcCUlQh38AzK.JzbkF;_ylu=X3oDMTBlMTQ4cGxyBHNlYwNzcgRzbGsDaW1n?back=http%3A%2F%2Fimages.search.yahoo.com%2Fsearch%2Fimages%3Fp%3Dkeyword_one%2Bkeyword_two %26_adv_prop%3Dimage%26va%3Dkeyword_one%2Bkeyword_two%26fr%3Dyfp-t-701%26tab%3Dorganic%26ri%3D122&w=750&h=500&imgurl=www.example.com%2Fpicture_gallery%2Fimages%2Flocation_of_image%2image.jpg& rurl=http%3A%2F%2Fwww.example.com%2Fpicture_gallery%2Fimage_location.html&size=81.9+KB&name=image_title%29&p=keyword_one+keyword_two&oid=c8a97c65e40bca9a6331f36da03145c4&fr2=&fr=yfp-t-701&tt=image_title%2529&b=121&ni=112&no=122&ts=&tab=organic&sigr=123rbb8eu&sigb=14545eqhn&sigi=13aikubif&.crumb=NZ.bhUZyY2s
HTTP/1.1" 404 486
"http://www.example.com/picture_gallery/image_location.html"
"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.5;
.NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 4.0; .NET CLR
2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"

According to my logs, this user agent seems to want the image contained in the folder www.example.com/picture_gallery//images/location_of_image/image.jpg.

Weird.

Hosting company said no impact to their system because bytes served were low and server was returning 404.

The "GET" references a perl logging script using a 1px by 1px web beacon.

[edited by: incrediBILL at 2:45 am (utc) on Sep 12, 2012]
[edit reason] broke up long referer [/edit]

 

wilderness




msg:4493673
 9:38 am on Sep 11, 2012 (gmt 0)

deny from 24.190.103.173
or
RewriteCond %{REMOTE_ADDR} ^24\.190\.(9[6-9]|10[0-3])\.

If you'd like to lessen the innocents?

#UA contains GTB and comes from Optimum WRRNNJ
RewriteCond %{HTTP_USER_AGENT} GTB
RewriteCond %{REMOTE_ADDR} ^24\.190\.(9[6-9]|10[0-3])\.
RewriteRule .* - [F]

keyplyr




msg:4493914
 6:30 pm on Sep 11, 2012 (gmt 0)

@slipkid

I wouldn't block 24.190.103.173 because it's a cable ISP and you'd be blocking real visitors.

See if you can block something unique to the UA.

slipkid




msg:4493917
 6:41 pm on Sep 11, 2012 (gmt 0)

@ keyplyr

I kind of figured it had something to do with a user's mouse. Researched the URI and found as indicated that it was coming from New Jersey.

Don't use Google ToolBar... so not familar if it had anything to do with the constant stream of hits.

My pics are the more popular pages on my site, so don't want to exclude them in robots.txt.

I consdier this a one-off problem, and will monitor to see if it happens again.

Thanks all for the help.

lucy24




msg:4493994
 9:58 pm on Sep 11, 2012 (gmt 0)

Yesterday, my website came under continual attack (DoS?) from a yahoo referrer/user agent listed below.

Putting a search engine in the forged-referer slot is a tried and true approach. Most of the time the exact wording is wrong, so you can block them even if you don't want to block the honest users coming in from real searches.

Now, personally I don't care much for yahoo so their image search goes straight into the hotlink bin without checking to see whether it's real or not. But ymmv.

I kind of figured it had something to do with a user's mouse.

Huh. Most people would blame the user's cat. But to each his own :)

slipkid




msg:4494015
 11:15 pm on Sep 11, 2012 (gmt 0)

Partial to dogs, hate cats. Would not give credit to a cat's intelligence to hit a mouse button every fifteen secods...

keyplyr




msg:4494024
 11:43 pm on Sep 11, 2012 (gmt 0)


Now, personally I don't care much for yahoo so their image search goes straight into the hotlink bin without checking to see whether it's real or not. But ymmv

I get triple digit daily traffic from Yahoo/Bing/Google image search, but I guess if you don't want traffic coming to your site then blocking them is an alternative. And BTW, once again this has nothing to do with hot-linking, at least not from the major SEs.

lucy24




msg:4494040
 2:12 am on Sep 12, 2012 (gmt 0)

once again this has nothing to do with hot-linking

Your server can't tell the difference between a hotlink and a "google image search sent me". (Uh... You did know that, didn't you? :() They both come through as referers, so any routine aimed at one kind will automatically pick up the other. Which is why at least half of my current hotlink exemptions are for assorted legitimate* google functions. Conversely, certain image directories are roboted-out because I know by direct experience that people aren't interested in the pages; they're just collecting hotlink fodder.


* For a given definition of "legitimate". I know some people have serious issues with Translate, but mine are perfectly respectable and there's no reason to block them.

wilderness




msg:4494051
 2:57 am on Sep 12, 2012 (gmt 0)

[quote]For a given definition of "legitimate". I know some people have serious issues with Translate, but mine are perfectly respectable and there's no reason to block them. [quote/]

Opinions are like. . . . ;)

keyplyr




msg:4494103
 6:56 am on Sep 12, 2012 (gmt 0)

Your server can't tell the difference between a hotlink and a "google image search sent me".

Sure I can because Google does not hotlink my images, at least not what I consider hotlinking. They are doing my bidding :)

I use a script that checks a few things any time a request is made for a file residing on my server from a remote source. It also busts the display of said image if the referrer isn't my site, and the human instantly gets pulled to my page where the image is.

lucy24




msg:4494132
 7:35 am on Sep 12, 2012 (gmt 0)

Sure I can

YOU can. Your server can't. You didn't actually read my post, did you?

the human instantly gets pulled to my page

Well, that's one way to use Image Search to generate traffic.

thetrasher




msg:4494287
 1:16 pm on Sep 12, 2012 (gmt 0)

I'm a little confused.

yahoo referrer/user agent

Sorry, but I see neither a Yahoo referrer nor a Yahoo user agent.

wilderness




msg:4494328
 2:20 pm on Sep 12, 2012 (gmt 0)

A0PDoTHcCUlQh38AzK.JzbkF;_ylu=X3oDMTBlMTQ4cGxyBHNlYwNzcgRzbGsDaW1n?back=http%3A%2F%2Fimages.search.yahoo.com%2Fsearch%2Fimages%3Fp%3Dkeywor

lucy24




msg:4494465
 8:12 pm on Sep 12, 2012 (gmt 0)

I see neither a Yahoo referrer nor a Yahoo user agent.

Technically you're right. But I think the sample line is a request sent to the OP's analytics program. In that case his own page would be listed as the referer, while the referer for that page would go into the request's query string. Go back a few lines in the logs and you'll find the original page request, with Yahoo in the referer line.

keyplyr




msg:4494510
 11:26 pm on Sep 12, 2012 (gmt 0)


YOU can. Your server can't. You didn't actually read my post, did you?

I read your entire post. I answered it accordingly. You didn't actually read my post, did you? LOL

e.g. my SERVER can tell the difference because of the script I have in place. Sorry, not going into any more detail on a public forum.

Anyway, as stated above, I enjoy the traffic resulting from image searches and do not consider it hotlinking since they have my full approval to do so.

slipkid




msg:4494541
 1:10 am on Sep 13, 2012 (gmt 0)

I see neither a Yahoo referrer nor a Yahoo user agent.


I agree with what Lucy24 has pointed out.

I am still learning how to frame issues on the forum.

Elsmarc




msg:4494563
 3:00 am on Sep 13, 2012 (gmt 0)

Goodness. All you need is an .htaccess file and use the RewriteCond %{HTTP_REFERER} bit. Just Google something like stop hotlinking. It's not rocket science.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved