homepage Welcome to WebmasterWorld Guest from 23.22.128.96
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Microsoft or not?
70.37.130.8
Bewenched




msg:4533174
 8:09 am on Jan 4, 2013 (gmt 0)

I keep getting image requests from 70.37.130.8 with a blank user agent.


I can't block blank UAs because of some of the services that we use dont send the UA info... go figure..

Anyhow, it appears to be from
NetName: MICROSOFT-DYNAMIC-HOSTING

but is that really Bing/Microsoft or what?

 

wilderness




msg:4533187
 10:12 am on Jan 4, 2013 (gmt 0)

I keep getting image requests from 70.37.130.8 with a blank user agent.


I can't block blank UAs because of some of the services that we use dont send the UA info... go figure..


Sure you may.
Just used multiple condition focused upon both the UA & IP.


There's an old thread around on MS Dynamic Hosting [webmasterworld.com]

wilderness




msg:4533188
 10:26 am on Jan 4, 2013 (gmt 0)

# deny blank UA; IP ONLY
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteCond %{REMOTE_ADDR} ^70\.37\.([0-9]|[1-9][0-9]|1[0-8][0-9]|19[01])\.
RewriteRule .* - [F]

FWIW, it's my understanding if your server has installed the latest version of Apache (2.4-?) that you may use CIDR.

My el-cheapo still has 2.2 installed.

wilderness




msg:4533189
 10:29 am on Jan 4, 2013 (gmt 0)

And not intending to hi-jack this thread, however I had some email correspondence via a website inquiry and the email header contained the following:

65.54.190.zzz
Microsoft Hosting

Anybody have a clue if MS has a similar hosting range in the 52-55 Class B?

Bewenched




msg:4533308
 6:36 pm on Jan 4, 2013 (gmt 0)

I'd tried blocking some of these hosting ranges of theirs only to find that bing shopping was pulling their images from that range.... pisses me off that they aren't supplying a user agent.

wilderness




msg:4533313
 6:43 pm on Jan 4, 2013 (gmt 0)

do bing shopping visitors show a refer?

keyplyr




msg:4533440
 1:12 am on Jan 5, 2013 (gmt 0)

I've blocked blank UA for close to 10 years:

RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteCond %{REQUEST_METHOD} !^HEAD$
RewriteRule !^(favicon\.ico|403\.html|w3c/p3p\.xml|w3c/policy1\.xml|robots\.txt)$ - [F]


But allow HEAD requests for the above files.

wilderness




msg:4533476
 5:45 am on Jan 5, 2013 (gmt 0)

I've blocked blank UA for close to 10 years:


As have I and many others (standard practice even at the suggestion of SE's and confirmation of DNS), however apparently his situation is unique.

not2easy




msg:4533477
 5:56 am on Jan 5, 2013 (gmt 0)

You can verify IPs at bing dot com/toolbox/verify-bingbot
Not the most useful way to do things.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved