Welcome to WebmasterWorld Guest from 54.162.167.40

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt - Disallow: or Disallow: /

     
10:34 pm on Apr 25, 2009 (gmt 0)

5+ Year Member



What's the correct universal go-away:

User-agent: *
Disallow:

or

User-agent: *
Disallow: /

Cheers,
Phred

10:40 pm on Apr 25, 2009 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



The first example blocks nothing.

The second example blocks everything.

10:49 pm on Apr 25, 2009 (gmt 0)

5+ Year Member



Thanks.

Phred

8:38 pm on Jul 22, 2009 (gmt 0)

5+ Year Member



Hi g1smd what would this say about the a robot.txt file,is it good enough? when it contains the following syntax

User-Agent: *
Allow: /

and if we want to specify the sitemap location would it help spiderability and indexing. If yes then how would we include the sitemap if you could tell the proper syntax for it please,thanks

9:40 pm on Jul 22, 2009 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Most searchengines do not understand "Allow".

Use Disallow as in the post above.

8:34 am on Jul 23, 2009 (gmt 0)

5+ Year Member



Hi g1smd,

Thanks for your reply mate.Why don't they understand it, would googlebot get it and what about using sitemap in robots.txt?

any reading resource to know more about it please.

12:41 am on Jul 24, 2009 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



1996 Robots Exclusion Standard [robotstxt.org]
Allowing Googlebot [google.com]
Yahoo! Slurp [help.yahoo.com]
Bing [bing.com]
9:00 am on Jul 24, 2009 (gmt 0)

5+ Year Member



Cheers Phranque
11:24 am on Jul 31, 2009 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



I have some errors in robots to show sitemap:
SITEMAP: http://www.example.com/sitemap.xml
How to do that validation will pass.
Error notice:
The following block of code contains some errors. You forgot to add a Disallow line in this block of code. You MUST insert at least one Disallow command. Please, remove all the reported errors and check again this robots.txt file.

[edited by: engine at 5:06 pm (utc) on Aug. 5, 2009]
[edit reason] please use example.com [/edit]

2:58 pm on Aug 5, 2009 (gmt 0)

5+ Year Member



User-agent: *

Sitemap: http://www.example.com/sitemap.xml

is this a correct format ?

[edited by: engine at 5:07 pm (utc) on Aug. 5, 2009]
[edit reason] please use example.com [/edit]

4:03 pm on Aug 5, 2009 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



SITEMAP: http://www.example.com/sitemap.xml
How to do that validation will pass.

it might be case-sensitive so try "Sitemap:" instead.

is this a correct format ?

from [sitemaps.org...]
This directive is independent of the user-agent line, so it doesn't matter where you place it in your file.

so you probably don't need the User-agent line but it won't hurt.
9:17 pm on Aug 5, 2009 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



If you add the 'User-agent' line, you must add the 'Disallow' line (even if the 'Disallow' line disallows nothing).
9:03 am on Aug 7, 2009 (gmt 0)

5+ Year Member



cheers lads thanks alot for useful information. i'll revise it
9:17 am on Aug 7, 2009 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



Yes, it is regardless of CASE SENSITIVE.
It is important to use User-agent' line and ALSO use of Disallow line.
10:58 pm on Aug 8, 2009 (gmt 0)

5+ Year Member



ahaan i'll add in Disallow, cheers thanks
12:26 pm on Aug 27, 2009 (gmt 0)



Hi I am Michael william

Please use the below syntax for correct code

User-agent: *
Allow: /
Disallow: /cgi-bin/

 

Featured Threads

Hot Threads This Week

Hot Threads This Month