homepage Welcome to WebmasterWorld Guest from 54.83.133.189
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Accredited PayPal World Seller

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Mozilla/4.* (Compatible
to block or not?
Seedy




msg:4429096
 4:26 pm on Mar 14, 2012 (gmt 0)

I've noticed MSN sometimes uses Mozilla/4.* plus one or two ISP bots like StalkStalk and a couple of educational institutions.

Other than that the rest of its' users seem to be bad news.

Does anyone know of any other reasons not to block 'Mozilla/4.* (Compatible' please, perhaps to add as exclusions?

Thank you

 

dstiles




msg:4429221
 9:34 pm on Mar 14, 2012 (gmt 0)

I assume you mean "block from MS IPs", since Mozilla/4 is still a major browser UA-part.

There is quite a bit of coverage on MS and various UAs in the Search Engine Spiders forum at [webmasterworld.com...]

Seedy




msg:4429450
 10:51 am on Mar 15, 2012 (gmt 0)

Well to be honest, no. I was considering creating a blanket block for my Mozilla/4. users with exception rules to allow those I know are genuine because from what I can see, the very large majority of visitors to my sites using it are unwelcome.

dstiles




msg:4429743
 10:47 pm on Mar 15, 2012 (gmt 0)

Most browsers IE, FF, chrome, Safari etc) begin their UAs with either Mozilla/4 or Mozilla/5 depending on browsaer age and a few other things. It's a stupid historical thing brought about by MS wanting to crash the web scene.

If you block most Mozilla/4 or 5 UAs you will lose a lot of real people.

I really recommend reading though the past year's Search Engine Spiders forum.

Seedy




msg:4429744
 10:50 pm on Mar 15, 2012 (gmt 0)

I wouldn't even consider blocking /5 but I just can't help but notice the very high proportion of /4 users that aren't legitimate visitors.

I will do as you suggest though and read up more.
Many thanks

g1smd




msg:4455631
 7:04 pm on May 20, 2012 (gmt 0)

I'm almost at the point of blocking all "Mozilla/4.0" instances, especially when combined with "MSIE 6.0".

Seedy




msg:4455642
 7:22 pm on May 20, 2012 (gmt 0)

I'm with you there, and Mozilla/4.0 coupled with all MSIE versions prior to 6.0 too (with an exception for SE's)

Perhaps something like:
RewriteCond %{HTTP_USER_AGENT} Mozilla/4 [NC] #AND
RewriteCond %{HTTP_USER_AGENT} MSIE/\ (1-6) [NC] #BUT NOT
RewriteCond %{HTTP_USER_AGENT} !(google|msn|bing) [NC]
RewriteRule 403.php [F]


?

g1smd




msg:4455645
 7:29 pm on May 20, 2012 (gmt 0)

Yes. I meant to say "MSIE 6.0 and before".

I see a lot of scraper bots, form filling spammer bots and little or no evidence of real human accesses.

If you mean versions 1 to 6 then
(1-6) should be [1-6] here.

MSIE/\ should be MSIE\ too. The slash should not be escaped.

Finally,
RewriteRule 403.php [F]
should be
RewriteRule !^403\.php - [F]
or similar.

dstiles




msg:4455672
 9:01 pm on May 20, 2012 (gmt 0)

Blocking MSIE6 is ALMOST acceptable.

The browser has been deprecated for a few years now and most accesses with it are bot/exploit attempts; although a few genuines still come through from people running Windows 2000, which cannot run any MS browser higher than that. In fact 2000 is also obsolete: I still use it here for a variety of reasons with Firefox but normally use linux.

g1smd




msg:4455677
 9:12 pm on May 20, 2012 (gmt 0)

Opera sometimes?/often? spoofs as MSIE 6.00. That's my major concern - and their list of UA strings used over the years is massive and unwieldy.

dougwilson




msg:4455721
 11:17 pm on May 20, 2012 (gmt 0)

I block msie 2-6. No loss in people visitors seen. Tons of critters turned away.

Wouldn't consider blocking mozilla 4

Seedy




msg:4455831
 7:49 am on May 21, 2012 (gmt 0)

If you mean versions 1 to 6 then (1-6) should be [1-6] here.

MSIE/\ should be MSIE\ too. The slash should not be escaped.

Finally, RewriteRule 403.php [F]
should be
RewriteRule !^403\.php - [F]
or similar.


Thank you for cleaning that up, I was being hasty.

dstiles




msg:4456136
 9:25 pm on May 21, 2012 (gmt 0)

A few "mobile" devices (mostly, I think, old phones) use Moz 1.2. Whether worth enabling is another matter.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved