| 5:45 pm on Dec 16, 2004 (gmt 0)|
Welcome to WebmasterWorld!
4) None. You have misspelled "Disallow:" and omitted the colon on "User-agent:". ;)
Robots use prefix-matching; Any URL which starts with the URL-path that you specify will not be fetched.
This will request that robots not fetch any URLs starting with "/help/".
| 9:03 pm on Dec 16, 2004 (gmt 0)|
Another useful tool, and links to understanding robots.txt is at:
| 2:58 am on Dec 17, 2004 (gmt 0)|
Remember that directory and filnames are case sensitive.
| 4:01 pm on Dec 17, 2004 (gmt 0)|
You said that, Any URL which starts with the URL-path that you specify will not be fetched.
for example, the robots file be,
i am little bit confused with the word "starts with"
[sample.com...] ( is it mean starts with /help/)?
and i want to know whether URLs are case sensitive?
thankz for ur reply
| 5:52 pm on Dec 20, 2004 (gmt 0)|
The spider disregards the domain, so in your example [sample.com...] the spider would consider it to start with /help/.
And as someone else said, it is case-sensitive.
| 5:32 am on Dec 21, 2004 (gmt 0)|
i really thankfull to all of you valuable reply..
thank you very much