Forum Moderators: open
Our programmers came up with a solution - identify all spiders and use a meta redirect to one particular server (using an internal IP address). They identified spiders by domain. Unfortunately, when they made this go live all of my AdWords traffic started hanging (organic continued to deliver correctly). Once I brought this to their attention, they quickly changed it back.
Now that I've learned what they are doing, something about the approach is strking me as wrong. They are talking about now identifying the spiders by IP address rather than domain. Isn't there a better way to segment out the spider traffic and move it to a specific server?
A huge portion of our e-commerce site's pages are generated using our search utility. The upside is very accurate results pages, the downside is high server load. The only time this becomes really problematic is when we are being heavily spidered.
The reason for the above is your own previous inquiry; i.e., 3rd paragraph, which I have relocated as 2nd paragraph.
Requests to verify domain names are very server side intensive.
Now that I've learned what they are doing, something about the approach is strking me as wrong. They are talking about now identifying the spiders by IP address rather than domain. Isn't there a better way to segment out the spider traffic and move it to a specific server?
IP address recognition is the accepted method:
This website is kept current:
[iplists.com...]
Our programmers came up with a solution - identify all spiders and use a meta redirect to one particular server (using an internal IP address). They identified spiders by domain. Unfortunately, when they made this go live all of my AdWords traffic started hanging (organic continued to deliver correctly). Once I brought this to their attention, they quickly changed it back.
Meta tag redirects have long been outdated and were never really accepted by the major SE's.
Rewrites or Redirects would be more in perspective, however you may get into page listing problems with the SE's because your serving a different content to them than the general populus.
I believe there exists a Webmaster World Forum for AdWords?
Perhaps that forum could address this portion more readily?
[webmasterworld.com...]
The mod_rewrite could make the secondary server transparent to the spiders (depending on how it was written) and therefore making your Adwords problem moot.
