homepage Welcome to WebmasterWorld Guest from 54.211.97.242
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt - Disallow: or Disallow: /
phred

5+ Year Member



 
Msg#: 3900674 posted 10:34 pm on Apr 25, 2009 (gmt 0)

What's the correct universal go-away:

User-agent: *
Disallow:

or

User-agent: *
Disallow: /

Cheers,
Phred

 

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3900674 posted 10:40 pm on Apr 25, 2009 (gmt 0)

The first example blocks nothing.

The second example blocks everything.

phred

5+ Year Member



 
Msg#: 3900674 posted 10:49 pm on Apr 25, 2009 (gmt 0)

Thanks.

Phred

lixsmk1

5+ Year Member



 
Msg#: 3900674 posted 8:38 pm on Jul 22, 2009 (gmt 0)

Hi g1smd what would this say about the a robot.txt file,is it good enough? when it contains the following syntax

User-Agent: *
Allow: /

and if we want to specify the sitemap location would it help spiderability and indexing. If yes then how would we include the sitemap if you could tell the proper syntax for it please,thanks

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3900674 posted 9:40 pm on Jul 22, 2009 (gmt 0)

Most searchengines do not understand "Allow".

Use Disallow as in the post above.

lixsmk1

5+ Year Member



 
Msg#: 3900674 posted 8:34 am on Jul 23, 2009 (gmt 0)

Hi g1smd,

Thanks for your reply mate.Why don't they understand it, would googlebot get it and what about using sitemap in robots.txt?

any reading resource to know more about it please.

phranque

WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3900674 posted 12:41 am on Jul 24, 2009 (gmt 0)

1996 Robots Exclusion Standard [robotstxt.org]
Allowing Googlebot [google.com]
Yahoo! Slurp [help.yahoo.com]
Bing [bing.com]

lixsmk1

5+ Year Member



 
Msg#: 3900674 posted 9:00 am on Jul 24, 2009 (gmt 0)

Cheers Phranque

toplisek

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3900674 posted 11:24 am on Jul 31, 2009 (gmt 0)

I have some errors in robots to show sitemap:
SITEMAP: http://www.example.com/sitemap.xml
How to do that validation will pass.
Error notice:
The following block of code contains some errors. You forgot to add a Disallow line in this block of code. You MUST insert at least one Disallow command. Please, remove all the reported errors and check again this robots.txt file.

[edited by: engine at 5:06 pm (utc) on Aug. 5, 2009]
[edit reason] please use example.com [/edit]

lixsmk1

5+ Year Member



 
Msg#: 3900674 posted 2:58 pm on Aug 5, 2009 (gmt 0)

User-agent: *

Sitemap: http://www.example.com/sitemap.xml

is this a correct format ?

[edited by: engine at 5:07 pm (utc) on Aug. 5, 2009]
[edit reason] please use example.com [/edit]

phranque

WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3900674 posted 4:03 pm on Aug 5, 2009 (gmt 0)

SITEMAP: http://www.example.com/sitemap.xml
How to do that validation will pass.

it might be case-sensitive so try "Sitemap:" instead.

is this a correct format ?

from [sitemaps.org...]
This directive is independent of the user-agent line, so it doesn't matter where you place it in your file.

so you probably don't need the User-agent line but it won't hurt.

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3900674 posted 9:17 pm on Aug 5, 2009 (gmt 0)

If you add the 'User-agent' line, you must add the 'Disallow' line (even if the 'Disallow' line disallows nothing).

lixsmk1

5+ Year Member



 
Msg#: 3900674 posted 9:03 am on Aug 7, 2009 (gmt 0)

cheers lads thanks alot for useful information. i'll revise it

toplisek

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3900674 posted 9:17 am on Aug 7, 2009 (gmt 0)

Yes, it is regardless of CASE SENSITIVE.
It is important to use User-agent' line and ALSO use of Disallow line.

lixsmk1

5+ Year Member



 
Msg#: 3900674 posted 10:58 pm on Aug 8, 2009 (gmt 0)

ahaan i'll add in Disallow, cheers thanks

Michael william



 
Msg#: 3900674 posted 12:26 pm on Aug 27, 2009 (gmt 0)

Hi I am Michael william

Please use the below syntax for correct code

User-agent: *
Allow: /
Disallow: /cgi-bin/

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved