I want entire site indexed

[brnm98105]

Hi all, Im a bit confused. This is my current robots.txt

User-Agent: *
Allow: /

I want my entire site indexed by any and all robots. Is this the correct robots.txt to use?

I see people also using

User-Agent: *
Disallow: /

Which one should I use?

Thanks in advance

[jdMorgan]

Neither.

The "Allow" directive is only supported by *some* of the major search engines' robots, since "Allow" is not part of the Standard for Robot Exclusion [robotstxt.org] "specification." Other robots may ignore it (resulting in them crawling your entire site, as you wish), or some of them may treat it as a fatal error and not crawl your site at all -- There's no telling which.

The second one will Disallow robots from fetching any URL on your site which starts with "/" -- In other words, it will Disallow *all* URLs on your site.

You have three choices:

1) Delete your robots.txt file, and put up with all of the 404-Not Found errors in your logs and the skewing of your "Site Statistics" reports because of these errors.

2) Upload a blank robots.txt file. This is perfectly-acceptable and allows all robots to crawl the site, while preventing the aforementioned 404 errors.

3) Use the correct syntax to explicitly allow all URLs to be fetched:

User-agent: *
Disallow:

Note that the Disallow argument is blank, and that there is a blank line at the end of this file.

Jim

[Siteman]

Agree. But option 3 would be preferable.

.S.

[brnm98105]

So like this :

User-agent: *
Disallow:

or

User-agent: *
Disallow: /

[Habtom]

Like this :)

User-agent: *
Disallow:

[brnm98105]

thanks all

[g1smd]

Not the extra blank line after the last line of visible text.