Have been searching forum for a while, but can't find answer to this one - can anyone help please? I have a page with images that's getting hit excessively. Is it possible to write the robots meta tag on a page to allow Google (for Adsense etc.) and disallow all other bots?
jomaxx
5:15 pm on Mar 25, 2004 (gmt 0)
Yes. Searching Google for "robot exclusion" should probably bring up lots of examples. Google's explanation of how to tell Googlebot what to spider is filed under "Jobs, Press, & Help", then "Webmaster Info".
Another good idea is to keep images in a standard directory, and exclude that directory for all robots if you don't want the images spidered.
