g1smd
msg:3900677 | 10:40 pm on Apr 25, 2009 (gmt 0) |
The first example blocks nothing. The second example blocks everything.
|
phred
msg:3900681 | 10:49 pm on Apr 25, 2009 (gmt 0) |
Thanks. Phred
|
lixsmk1
msg:3957583 | 8:38 pm on Jul 22, 2009 (gmt 0) |
Hi g1smd what would this say about the a robot.txt file,is it good enough? when it contains the following syntax User-Agent: *
Allow: / and if we want to specify the sitemap location would it help spiderability and indexing. If yes then how would we include the sitemap if you could tell the proper syntax for it please,thanks
|
g1smd
msg:3957628 | 9:40 pm on Jul 22, 2009 (gmt 0) |
Most searchengines do not understand "Allow". Use Disallow as in the post above.
|
lixsmk1
msg:3957948 | 8:34 am on Jul 23, 2009 (gmt 0) |
Hi g1smd, Thanks for your reply mate.Why don't they understand it, would googlebot get it and what about using sitemap in robots.txt? any reading resource to know more about it please.
|
phranque
msg:3958533 | 12:41 am on Jul 24, 2009 (gmt 0) |
1996 Robots Exclusion Standard [robotstxt.org]
Allowing Googlebot [google.com]
Yahoo! Slurp [help.yahoo.com]
Bing [bing.com]
|
lixsmk1
msg:3958748 | 9:00 am on Jul 24, 2009 (gmt 0) |
Cheers Phranque
|
toplisek
msg:3963071 | 11:24 am on Jul 31, 2009 (gmt 0) |
I have some errors in robots to show sitemap:
SITEMAP: http://www.example.com/sitemap.xml
How to do that validation will pass.
Error notice:
The following block of code contains some errors. You forgot to add a Disallow line in this block of code. You MUST insert at least one Disallow command. Please, remove all the reported errors and check again this robots.txt file. [edited by: engine at 5:06 pm (utc) on Aug. 5, 2009]
[edit reason] please use example.com [/edit]
|
lixsmk1
msg:3966171 | 2:58 pm on Aug 5, 2009 (gmt 0) |
User-agent: * Sitemap: http://www.example.com/sitemap.xml is this a correct format ? [edited by: engine at 5:07 pm (utc) on Aug. 5, 2009]
[edit reason] please use example.com [/edit]
|
phranque
msg:3966230 | 4:03 pm on Aug 5, 2009 (gmt 0) |
SITEMAP: http://www.example.com/sitemap.xml
How to do that validation will pass. |
|
it might be case-sensitive so try "Sitemap:" instead. | is this a correct format ? |
|
from [sitemaps.org...]
| This directive is independent of the user-agent line, so it doesn't matter where you place it in your file. |
|
so you probably don't need the User-agent line but it won't hurt.
|
g1smd
msg:3966433 | 9:17 pm on Aug 5, 2009 (gmt 0) |
If you add the 'User-agent' line, you must add the 'Disallow' line (even if the 'Disallow' line disallows nothing).
|
lixsmk1
msg:3967598 | 9:03 am on Aug 7, 2009 (gmt 0) |
cheers lads thanks alot for useful information. i'll revise it
|
toplisek
msg:3967602 | 9:17 am on Aug 7, 2009 (gmt 0) |
Yes, it is regardless of CASE SENSITIVE.
It is important to use User-agent' line and ALSO use of Disallow line.
|
lixsmk1
msg:3968624 | 10:58 pm on Aug 8, 2009 (gmt 0) |
ahaan i'll add in Disallow, cheers thanks
|
Michael william
msg:3979433 | 12:26 pm on Aug 27, 2009 (gmt 0) |
Hi I am Michael william Please use the below syntax for correct code User-agent: *
Allow: /
Disallow: /cgi-bin/
|
|
