homepage Welcome to WebmasterWorld Guest from 50.19.169.37
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt - Disallow: or Disallow: /
phred




msg:3900676
 10:34 pm on Apr 25, 2009 (gmt 0)

What's the correct universal go-away:

User-agent: *
Disallow:

or

User-agent: *
Disallow: /

Cheers,
Phred

 

g1smd




msg:3900677
 10:40 pm on Apr 25, 2009 (gmt 0)

The first example blocks nothing.

The second example blocks everything.

phred




msg:3900681
 10:49 pm on Apr 25, 2009 (gmt 0)

Thanks.

Phred

lixsmk1




msg:3957583
 8:38 pm on Jul 22, 2009 (gmt 0)

Hi g1smd what would this say about the a robot.txt file,is it good enough? when it contains the following syntax

User-Agent: *
Allow: /

and if we want to specify the sitemap location would it help spiderability and indexing. If yes then how would we include the sitemap if you could tell the proper syntax for it please,thanks

g1smd




msg:3957628
 9:40 pm on Jul 22, 2009 (gmt 0)

Most searchengines do not understand "Allow".

Use Disallow as in the post above.

lixsmk1




msg:3957948
 8:34 am on Jul 23, 2009 (gmt 0)

Hi g1smd,

Thanks for your reply mate.Why don't they understand it, would googlebot get it and what about using sitemap in robots.txt?

any reading resource to know more about it please.

phranque




msg:3958533
 12:41 am on Jul 24, 2009 (gmt 0)

1996 Robots Exclusion Standard [robotstxt.org]
Allowing Googlebot [google.com]
Yahoo! Slurp [help.yahoo.com]
Bing [bing.com]

lixsmk1




msg:3958748
 9:00 am on Jul 24, 2009 (gmt 0)

Cheers Phranque

toplisek




msg:3963071
 11:24 am on Jul 31, 2009 (gmt 0)

I have some errors in robots to show sitemap:
SITEMAP: http://www.example.com/sitemap.xml
How to do that validation will pass.
Error notice:
The following block of code contains some errors. You forgot to add a Disallow line in this block of code. You MUST insert at least one Disallow command. Please, remove all the reported errors and check again this robots.txt file.

[edited by: engine at 5:06 pm (utc) on Aug. 5, 2009]
[edit reason] please use example.com [/edit]

lixsmk1




msg:3966171
 2:58 pm on Aug 5, 2009 (gmt 0)

User-agent: *

Sitemap: http://www.example.com/sitemap.xml

is this a correct format ?

[edited by: engine at 5:07 pm (utc) on Aug. 5, 2009]
[edit reason] please use example.com [/edit]

phranque




msg:3966230
 4:03 pm on Aug 5, 2009 (gmt 0)

SITEMAP: http://www.example.com/sitemap.xml
How to do that validation will pass.

it might be case-sensitive so try "Sitemap:" instead.

is this a correct format ?

from [sitemaps.org...]
This directive is independent of the user-agent line, so it doesn't matter where you place it in your file.

so you probably don't need the User-agent line but it won't hurt.

g1smd




msg:3966433
 9:17 pm on Aug 5, 2009 (gmt 0)

If you add the 'User-agent' line, you must add the 'Disallow' line (even if the 'Disallow' line disallows nothing).

lixsmk1




msg:3967598
 9:03 am on Aug 7, 2009 (gmt 0)

cheers lads thanks alot for useful information. i'll revise it

toplisek




msg:3967602
 9:17 am on Aug 7, 2009 (gmt 0)

Yes, it is regardless of CASE SENSITIVE.
It is important to use User-agent' line and ALSO use of Disallow line.

lixsmk1




msg:3968624
 10:58 pm on Aug 8, 2009 (gmt 0)

ahaan i'll add in Disallow, cheers thanks

Michael william




msg:3979433
 12:26 pm on Aug 27, 2009 (gmt 0)

Hi I am Michael william

Please use the below syntax for correct code

User-agent: *
Allow: /
Disallow: /cgi-bin/

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved