Welcome to WebmasterWorld Guest from 54.224.5.186

Forum Moderators: goodroi

Message Too Old, No Replies

I want entire site indexed

     
3:51 pm on Dec 16, 2008 (gmt 0)

Junior Member

5+ Year Member

joined:Sept 10, 2008
posts:79
votes: 0


Hi all, Im a bit confused. This is my current robots.txt

User-Agent: *
Allow: /

I want my entire site indexed by any and all robots. Is this the correct robots.txt to use?

I see people also using

User-Agent: *
Disallow: /

Which one should I use?

Thanks in advance

4:31 pm on Dec 16, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Neither.

The "Allow" directive is only supported by *some* of the major search engines' robots, since "Allow" is not part of the Standard for Robot Exclusion [robotstxt.org] "specification." Other robots may ignore it (resulting in them crawling your entire site, as you wish), or some of them may treat it as a fatal error and not crawl your site at all -- There's no telling which.

The second one will Disallow robots from fetching any URL on your site which starts with "/" -- In other words, it will Disallow *all* URLs on your site.

You have three choices:

1) Delete your robots.txt file, and put up with all of the 404-Not Found errors in your logs and the skewing of your "Site Statistics" reports because of these errors.

2) Upload a blank robots.txt file. This is perfectly-acceptable and allows all robots to crawl the site, while preventing the aforementioned 404 errors.

3) Use the correct syntax to explicitly allow all URLs to be fetched:

User-agent: *
Disallow:


Note that the Disallow argument is blank, and that there is a blank line at the end of this file.

Jim

5:13 pm on Dec 16, 2008 (gmt 0)

New User

5+ Year Member

joined:Dec 16, 2008
posts: 7
votes: 0


Agree. But option 3 would be preferable.

.S.

5:45 pm on Dec 16, 2008 (gmt 0)

Junior Member

5+ Year Member

joined:Sept 10, 2008
posts:79
votes: 0


So like this :

User-agent: *
Disallow:

or

User-agent: *
Disallow: /

5:51 pm on Dec 16, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 15, 2004
posts:1867
votes: 0


Like this :)

User-agent: *
Disallow:

6:07 pm on Dec 16, 2008 (gmt 0)

Junior Member

5+ Year Member

joined:Sept 10, 2008
posts:79
votes: 0


thanks all
8:38 pm on Dec 19, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Not the extra blank line after the last line of visible text.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members