Question about blocking 1 robot, but allow all others

[capsmaster]

Just so I don't screw this up, if I wanted to block yahoo but allow all others would this be the right code?

User-agent: Slurp
Disallow: /

Do I need to add anything else so all other robots still crawl my site?

Thanks in advance.

[Sanenet]

Yes, that looks fine.

Try adding
User-agent: *
Disallow:
at the top, this allows the bots to go anywhere (although most assume this by default?)

[robotstxt.org...]

[capsmaster]

Thanks a bunch Sanenet!

[Clint]

I was just about to ask this EXACT same question for SINGLE pages. ;)

Does anyone know the correct meta tag to stop the Yahoo bot from indexing/crawling SINGLE pages? Their site mentions <META NAME="robots" CONTENT="noindex"> but that of course is not going to work since that will block *ALL* bots. So, I need to know the meta tag JUST for Yahoo that I can add on certain pages. Or, can this/should this be done in the robots.txt file? I already have this in my robots file:

User-agent: *
Disallow:

Thanks.

[Sanenet]

Do it in robots.txt

UserAgent:*
Disallow:

UserAgent: Slurp
Disallow: page1.htm
Disallow: page2.htm
Disallow: dir/dir2/page1.htm

[Clint]

Hey thanks a bunch. ;)

Leave a space below this?
UserAgent:*
Disallow:

[jdMorgan]

Most robots will accept the record which contains their robot name or the first record that contains "*" whichever comes first. Some are smarter than that, but don't count on it.

Therefore you will usually want to put the robot-specific records first, and follow them with the wild-card robot record:

User-agent: Slurp
Disallow: /page1.htm
Disallow: /page2.htm
Disallow: /dir/dir2/page1.htm

User-agent: *
Disallow:

This disallows Slurp from fetching the indicated resources, and allows all other robots unrestricted access. If you reverse the order, then Slurp may fetch only the wild-card record, and assume that it too is free to fetch all resources.

All files should be specified starting with a slash, since robots.txt uses prefix-matching.

Each record should end with a blank line, including the last record, as shown.

"User-agent" and "Disallow" should be spelled and captialized exactly as shown to avoid problems.

After making any changes, validate your robots.txt file here [searchengineworld.com].

Jim

[Clint]

Thanks a lot Jim for the info. :)