Welcome to WebmasterWorld Guest from 54.226.32.234

Forum Moderators: goodroi

Message Too Old, No Replies

I want entire site indexed

     

brnm98105

3:51 pm on Dec 16, 2008 (gmt 0)

5+ Year Member



Hi all, Im a bit confused. This is my current robots.txt

User-Agent: *
Allow: /

I want my entire site indexed by any and all robots. Is this the correct robots.txt to use?

I see people also using

User-Agent: *
Disallow: /

Which one should I use?

Thanks in advance

jdMorgan

4:31 pm on Dec 16, 2008 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Neither.

The "Allow" directive is only supported by *some* of the major search engines' robots, since "Allow" is not part of the Standard for Robot Exclusion [robotstxt.org] "specification." Other robots may ignore it (resulting in them crawling your entire site, as you wish), or some of them may treat it as a fatal error and not crawl your site at all -- There's no telling which.

The second one will Disallow robots from fetching any URL on your site which starts with "/" -- In other words, it will Disallow *all* URLs on your site.

You have three choices:

1) Delete your robots.txt file, and put up with all of the 404-Not Found errors in your logs and the skewing of your "Site Statistics" reports because of these errors.

2) Upload a blank robots.txt file. This is perfectly-acceptable and allows all robots to crawl the site, while preventing the aforementioned 404 errors.

3) Use the correct syntax to explicitly allow all URLs to be fetched:

User-agent: *
Disallow:


Note that the Disallow argument is blank, and that there is a blank line at the end of this file.

Jim

Siteman

5:13 pm on Dec 16, 2008 (gmt 0)

5+ Year Member



Agree. But option 3 would be preferable.

.S.

brnm98105

5:45 pm on Dec 16, 2008 (gmt 0)

5+ Year Member



So like this :

User-agent: *
Disallow:

or

User-agent: *
Disallow: /

Habtom

5:51 pm on Dec 16, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Like this :)

User-agent: *
Disallow:

brnm98105

6:07 pm on Dec 16, 2008 (gmt 0)

5+ Year Member



thanks all

g1smd

8:38 pm on Dec 19, 2008 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Not the extra blank line after the last line of visible text.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month