Forum Moderators: open

Message Too Old, No Replies

Robots exclusion

How to exclude selectively robots?

         

specter

7:36 am on Mar 14, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Is it possible to exclude (or allow) a particular robot for a web site?
F.I. Can I allow the crawling only to Altavista robot and block Google robot?

If yes,how?

Thanks to all.

wilderness

4:26 pm on Mar 14, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Both Google and AltaVista honor robots.txt

[google.com...]
[google.com...]
[google.com...]

[help.yahoo.com...]
[help.yahoo.com...]

wilderness

5:00 pm on Mar 15, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



specter,
There are many examples across the internet of robots.txt. See google.

You may go to most any website and add "/robots.txt" to the domain name and view their file.
You'll see a variety.
These are regular text files.

There is a small software which may be acquired for free which will create a robots.txt for you:
RoboGen
[rietta.com...]

Don