Directing Spiders to specific Servers

[TheSeoGuy]

We have a web site that receives a large amount of traffic which requires the site to be load balanced accross four servers.

Is there a way to use robots.txt to allow/disallow specific servers?

Thanks in advance.

[goodroi]

Simple answer is no. Robots.txt is not designed to allow or disallow specific servers.

Side note - I am not sure why you would want to allow or disallow specific servers. My first guess is that you are trying to deliver different content to the search engines. If that is your goal you should become very knowledgeable on ip delivery otherwise known as cloaking.