Where do bots come from?

geography...

[followgreg]

Hi,

I am trying to understand where bots are coming from, I mean what cities in particular, in know that:

Yahoo = sunnyvale, CA

But except for this one (I am using geotrends from webtrends) what robots come from what cities, What about Google, MSN, Ask and the like?

[nmattheij]

I doubt they come from just one point. Google and MSN probably have bots all over the planet.

[gregbo]

I am trying to understand where bots are coming from, I mean what cities in particular ...

Even if you get a mapping from the IP address of the bot to some country or city, the bots may actually be dispersed around the globe, running in the private address space of the company operating them. The IPs you see may correspond to where the bot traffic enters public IP address space.

What exactly are you trying to accomplish? Perhaps there is another way to solve your problem.

[Receptional]

followgreg asks an interesting question.

If I want to deliver UK content to Brits, US content to Americans and French to the Canadians (;)) then I don't want the bots getting different stuff to the intended audience. After all - that would break guidelines!

So - how do you tell a bot that you have different content for different locations? Is there a way? does Yahoo or Google have a way to figure this out? Is the robots.txt stuff that could help?

[followgreg]

Since receptional brought up an interesting point :) I think that this thread needs a little push.
Geoip vs. bots analysis might tell us a little more about how SE see our sites...
does anyone have some experience/thoughts to share?

[gregbo]

If I want to deliver UK content to Brits, US content to Americans and French to the Canadians (;)) then I don't want the bots getting different stuff to the intended audience. After all - that would break guidelines!

What guidelines?

So - how do you tell a bot that you have different content for different locations? Is there a way? does Yahoo or Google have a way to figure this out? Is the robots.txt stuff that could help?

I'm not sure you can tell a (generic) bot to do this. They spider based on links, generally speaking. Even if you could do this, you can't be sure that the people who are searching for your site are constructing queries (or proffering other information) that would cause specific results to be displayed.

You might consider looking into something like paid inclusion with SEs that offer tools and features that (attempt to) geotarget a user to search results based on their location.