Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Can a very restrictive firewall get your site banned from Google?

         

1script

8:10 pm on Aug 23, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Trying to get to the bottom of this issue with a multiple site ban. (more info here [webmasterworld.com]) One thing that links all these sites is the firewall policy. Lately, I've been adding entire subnets to my iptables' DENY rules. I've been monitoring suspicious behaviors in my sites' logs and have been adding subnets that belonged to AWS, Web hosting companies and some especially often offending internet providers from Far East. The reasoning was that a real visitor can't actually come from a cloud (AWS) or a web hosting IP unless they are browsing behind a proxy and if they are behind a proxy, they're usually up to no good.

In any case, I was very careful to avoid banning IPs belonging to Google, Yahoo, Bing and a handful of other important SEs. Indeed, Googlebot is still feasting happily on my pages, 50,000+ a day. However, I would have no idea if the infamous Google manual reviewers are browsing via a proxy or a VPN. Are they known to be evasive or are they guaranteed to come from a Google IP?

Does a scenario like this sound probable to you guys: a site gets queued for review for whatever reason. The reviewer can't get to it and bans it just in case?

dstiles

9:32 pm on Aug 23, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I operate a similar blocking policy with no deleterious effect. On the other hand, I have no idea if any of my sites have ever been google-reviewed.

I allow most (though not all) google IPs even if they are not bot IPs, although I keep bot IP ranges tied with bot UAs to ensure valid crawling. So if a reviewer came in on a non-bot range (and it would have to be a non-bot IP) with a valid browser UA (and various other valid things) then the visitor would get access. Some of the prohibited IPs include certain feed/preview services.

levo

10:18 pm on Aug 23, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



There is a difference between blocking with 403 etc. and blocking with firewall - dropping (instead of denying) the connection. Google chrome or toolbar might report the timeout as page-speed stat..

1script

10:39 pm on Aug 23, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Google chrome or toolbar might report the timeout as page-speed stat..
Do you mean toolbar timeouts can be considered equally as important a signal as Googlebot's? I guess it can be detrimental to some degree but I'm not so sure about an outright ban...

levo

6:22 pm on Aug 24, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Well, it is just a theory. Those timeouts can also be considered as the website is down/not reliable (frequently?).