homepage Welcome to WebmasterWorld Guest from 54.167.75.155
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
I want entire site indexed
brnm98105




msg:3808635
 3:51 pm on Dec 16, 2008 (gmt 0)

Hi all, Im a bit confused. This is my current robots.txt

User-Agent: *
Allow: /

I want my entire site indexed by any and all robots. Is this the correct robots.txt to use?

I see people also using

User-Agent: *
Disallow: /

Which one should I use?

Thanks in advance

 

jdMorgan




msg:3808657
 4:31 pm on Dec 16, 2008 (gmt 0)

Neither.

The "Allow" directive is only supported by *some* of the major search engines' robots, since "Allow" is not part of the Standard for Robot Exclusion [robotstxt.org] "specification." Other robots may ignore it (resulting in them crawling your entire site, as you wish), or some of them may treat it as a fatal error and not crawl your site at all -- There's no telling which.

The second one will Disallow robots from fetching any URL on your site which starts with "/" -- In other words, it will Disallow *all* URLs on your site.

You have three choices:

1) Delete your robots.txt file, and put up with all of the 404-Not Found errors in your logs and the skewing of your "Site Statistics" reports because of these errors.

2) Upload a blank robots.txt file. This is perfectly-acceptable and allows all robots to crawl the site, while preventing the aforementioned 404 errors.

3) Use the correct syntax to explicitly allow all URLs to be fetched:
User-agent: *
Disallow:


Note that the Disallow argument is blank, and that there is a blank line at the end of this file.

Jim

Siteman




msg:3808697
 5:13 pm on Dec 16, 2008 (gmt 0)

Agree. But option 3 would be preferable.

.S.

brnm98105




msg:3808718
 5:45 pm on Dec 16, 2008 (gmt 0)

So like this :

User-agent: *
Disallow:

or

User-agent: *
Disallow: /

Habtom




msg:3808719
 5:51 pm on Dec 16, 2008 (gmt 0)

Like this :)

User-agent: *
Disallow:

brnm98105




msg:3808732
 6:07 pm on Dec 16, 2008 (gmt 0)

thanks all

g1smd




msg:3811530
 8:38 pm on Dec 19, 2008 (gmt 0)

Not the extra blank line after the last line of visible text.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved