Welcome to WebmasterWorld Guest from 54.147.28.255

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt - Disallow: or Disallow: /

     
10:34 pm on Apr 25, 2009 (gmt 0)

Junior Member

5+ Year Member

joined:May 11, 2008
posts:55
votes: 0


What's the correct universal go-away:

User-agent: *
Disallow:

or

User-agent: *
Disallow: /

Cheers,
Phred

10:40 pm on Apr 25, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


The first example blocks nothing.

The second example blocks everything.

10:49 pm on Apr 25, 2009 (gmt 0)

Junior Member

5+ Year Member

joined:May 11, 2008
posts:55
votes: 0


Thanks.

Phred

8:38 pm on July 22, 2009 (gmt 0)

New User

5+ Year Member

joined:Dec 12, 2008
posts:32
votes: 0


Hi g1smd what would this say about the a robot.txt file,is it good enough? when it contains the following syntax

User-Agent: *
Allow: /

and if we want to specify the sitemap location would it help spiderability and indexing. If yes then how would we include the sitemap if you could tell the proper syntax for it please,thanks

9:40 pm on July 22, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Most searchengines do not understand "Allow".

Use Disallow as in the post above.

8:34 am on July 23, 2009 (gmt 0)

New User

5+ Year Member

joined:Dec 12, 2008
posts:32
votes: 0


Hi g1smd,

Thanks for your reply mate.Why don't they understand it, would googlebot get it and what about using sitemap in robots.txt?

any reading resource to know more about it please.

12:41 am on July 24, 2009 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10550
votes: 10


1996 Robots Exclusion Standard [robotstxt.org]
Allowing Googlebot [google.com]
Yahoo! Slurp [help.yahoo.com]
Bing [bing.com]
9:00 am on July 24, 2009 (gmt 0)

New User

5+ Year Member

joined:Dec 12, 2008
posts:32
votes: 0


Cheers Phranque
11:24 am on July 31, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member

joined:Oct 25, 2006
posts: 1001
votes: 0


I have some errors in robots to show sitemap:
SITEMAP: http://www.example.com/sitemap.xml
How to do that validation will pass.
Error notice:
The following block of code contains some errors. You forgot to add a Disallow line in this block of code. You MUST insert at least one Disallow command. Please, remove all the reported errors and check again this robots.txt file.

[edited by: engine at 5:06 pm (utc) on Aug. 5, 2009]
[edit reason] please use example.com [/edit]

2:58 pm on Aug 5, 2009 (gmt 0)

New User

5+ Year Member

joined:Dec 12, 2008
posts:32
votes: 0


User-agent: *

Sitemap: http://www.example.com/sitemap.xml

is this a correct format ?

[edited by: engine at 5:07 pm (utc) on Aug. 5, 2009]
[edit reason] please use example.com [/edit]

4:03 pm on Aug 5, 2009 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10550
votes: 10


SITEMAP: http://www.example.com/sitemap.xml
How to do that validation will pass.

it might be case-sensitive so try "Sitemap:" instead.

is this a correct format ?

from [sitemaps.org...]
This directive is independent of the user-agent line, so it doesn't matter where you place it in your file.

so you probably don't need the User-agent line but it won't hurt.
9:17 pm on Aug 5, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


If you add the 'User-agent' line, you must add the 'Disallow' line (even if the 'Disallow' line disallows nothing).
9:03 am on Aug 7, 2009 (gmt 0)

New User

5+ Year Member

joined:Dec 12, 2008
posts:32
votes: 0


cheers lads thanks alot for useful information. i'll revise it
9:17 am on Aug 7, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member

joined:Oct 25, 2006
posts: 1001
votes: 0


Yes, it is regardless of CASE SENSITIVE.
It is important to use User-agent' line and ALSO use of Disallow line.
10:58 pm on Aug 8, 2009 (gmt 0)

New User

5+ Year Member

joined:Dec 12, 2008
posts:32
votes: 0


ahaan i'll add in Disallow, cheers thanks
12:26 pm on Aug 27, 2009 (gmt 0)

New User

joined:Aug 24, 2009
posts: 3
votes: 0


Hi I am Michael william

Please use the below syntax for correct code

User-agent: *
Allow: /
Disallow: /cgi-bin/

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members