Forum Moderators: goodroi
Does anyone have a list?
i think you might want to instead block american ips so you dont get visitors from america. let us know what you are trying to achieve to make sure we are give the right suggestions.
My reason for excluding the american search engines is to better define where I get my traffic from. To be honest their are too many bots and even with the 404 hits on my site are too much traffic. I can't process american orders and I can't ship my product to the united states.
So the I lower my expenses on bandwidth and raise my overall profit margin.
I thought this would be good opportunity to greate a projet of a master search engine list by country.
This method people could further define what traffic they want by where it comes from.
This method people could further define what traffic they want by where it comes from.
i understand what you are saying and it is helpful to have the option to block traffic from different geographic areas. some products can not be exported for a variety of reasons.
the unique issue with banning the american marketplace is that many search engines started in the us. thus the .com domain is intended for generic searches and not only specific to a country like the .co.uk, .de, .fr etc. domains. for example google.com and yahoo.com are mostly american users but they do have significant international traffic since those domains are the generic domain.
my vote would be for blocking by ip. imho this would provide better geographic control.
(But it is not perfect.)
What I have done is located the IP addresses to the bots that I do not want and deny them in my .htaccess file. In my custome 403 page I use the " option explicit response.Status " and also send a 404.
the bandwidth falle has been very noticable.
I am only currently allowing the bots for these search engines
HenriLeRobotMirago, seek and voila.
Profit margin has increased.
Next project I must compile a list of well known proxy servers in Canada, France, Switzerland, luxemburg and belgium. Once I do that i think I will have finished.
google.com and yahoo.com are mostly american users but they do have significant international traffic since those domains are the generic domain.
Does google.com really have *significant* international traffic?
I've tried visiting www.google.com when visting various European countries and without exception I've been redirected to the relevant www.google.TLD (.co.uk / .de / .fr / .it / .se / ...)
Do you have any statistics on non-US users of www.google.com?
just to be clarify when i said significant i did not mean a majority of traffic. google.com and yahoo.com are not composed of 100% american traffic.
I need to generate a new robots.txt file and I pretty much need to deny american search engine traffic.
You really don't need a list to block.
After your list of allowed robots just add this and all the rest will be blocked:
User-agent: *
Disallow: /
