jdMorgan
msg:1527670 | 5:45 pm on Dec 16, 2004 (gmt 0) |
venuc, Welcome to WebmasterWorld! 4) None. You have misspelled "Disallow:" and omitted the colon on "User-agent:". ;) Robots use prefix-matching; Any URL which starts with the URL-path that you specify will not be fetched.
[b]U[/b]ser-agen[b]t:[/b] *
Di[b]s[/b]allow: /help/
This will request that robots not fetch any URLs starting with "/help/". See [robotstxt.org...] Jim
|
treeline
msg:1527671 | 9:03 pm on Dec 16, 2004 (gmt 0) |
Another useful tool, and links to understanding robots.txt is at: [searchengineworld.com...]
|
Hints
msg:1527672 | 2:58 am on Dec 17, 2004 (gmt 0) |
Remember that directory and filnames are case sensitive. Hints
|
venuc
msg:1527673 | 4:01 pm on Dec 17, 2004 (gmt 0) |
Dear dMorgan, You said that, Any URL which starts with the URL-path that you specify will not be fetched. for example, the robots file be, User-agent: *
Disallow: /help/ i am little bit confused with the word "starts with" [sample.com...] ( is it mean starts with /help/)? and i want to know whether URLs are case sensitive? thankz for ur reply
venu
|
teaperson
msg:1527674 | 5:52 pm on Dec 20, 2004 (gmt 0) |
The spider disregards the domain, so in your example [sample.com...] the spider would consider it to start with /help/. And as someone else said, it is case-sensitive.
|
venuc
msg:1527675 | 5:32 am on Dec 21, 2004 (gmt 0) |
Hi everybody, i really thankfull to all of you valuable reply.. thank you very much venu
|
|
