Robots.txt - Disallow: or Disallow: /

[phred]

What's the correct universal go-away:

User-agent: *
Disallow:

or

User-agent: *
Disallow: /

Cheers,
Phred

[g1smd]

The first example blocks nothing.

The second example blocks everything.

[phred]

Thanks.

Phred

[lixsmk1]

Hi g1smd what would this say about the a robot.txt file,is it good enough? when it contains the following syntax

User-Agent: *
Allow: /

and if we want to specify the sitemap location would it help spiderability and indexing. If yes then how would we include the sitemap if you could tell the proper syntax for it please,thanks

[g1smd]

Most searchengines do not understand "Allow".

Use Disallow as in the post above.

[lixsmk1]

Hi g1smd,

Thanks for your reply mate.Why don't they understand it, would googlebot get it and what about using sitemap in robots.txt?

any reading resource to know more about it please.

[phranque]

1996 Robots Exclusion Standard [robotstxt.org]
Allowing Googlebot [google.com]
Yahoo! Slurp [help.yahoo.com]
Bing [bing.com]

[lixsmk1]

Cheers Phranque

[toplisek]

I have some errors in robots to show sitemap:
SITEMAP: http://www.example.com/sitemap.xml
How to do that validation will pass.
Error notice:
The following block of code contains some errors. You forgot to add a Disallow line in this block of code. You MUST insert at least one Disallow command. Please, remove all the reported errors and check again this robots.txt file.

[edited by: engine at 5:06 pm (utc) on Aug. 5, 2009]
[edit reason] please use example.com [/edit]

[lixsmk1]

User-agent: *

Sitemap: http://www.example.com/sitemap.xml

is this a correct format ?

[edited by: engine at 5:07 pm (utc) on Aug. 5, 2009]
[edit reason] please use example.com [/edit]

[phranque]

SITEMAP: http://www.example.com/sitemap.xml
How to do that validation will pass.

it might be case-sensitive so try "Sitemap:" instead.

is this a correct format ?

from [sitemaps.org...]

This directive is independent of the user-agent line, so it doesn't matter where you place it in your file.

so you probably don't need the User-agent line but it won't hurt.

[g1smd]

If you add the 'User-agent' line, you must add the 'Disallow' line (even if the 'Disallow' line disallows nothing).

[lixsmk1]

cheers lads thanks alot for useful information. i'll revise it

[toplisek]

Yes, it is regardless of CASE SENSITIVE.
It is important to use User-agent' line and ALSO use of Disallow line.

[lixsmk1]

ahaan i'll add in Disallow, cheers thanks

[Michael william]

Hi I am Michael william

Please use the below syntax for correct code

User-agent: *
Allow: /
Disallow: /cgi-bin/