Forum Moderators: goodroi
for example:
In my robots.txt file statements are created like,
user-agent *
Dissallow : /help/
Now, for example my directors are like,
[sample.com...]
[sample.com...]
my queston is, which of the above link is restricted to the spiders.
1)the parent directory name /help/ will be restricted
2)or whatever it may be directory name(whether parent directory or sub directory) /help/ will be restricted
3) the directory codes/help/ also restricted?
Welcome to WebmasterWorld!
4) None. You have misspelled "Disallow:" and omitted the colon on "User-agent:". ;)
Robots use prefix-matching; Any URL which starts with the URL-path that you specify will not be fetched.
[b]U[/b]ser-agen[b]t:[/b] *
Di[b]s[/b]allow: /help/
See [robotstxt.org...]
Jim
You said that, Any URL which starts with the URL-path that you specify will not be fetched.
for example, the robots file be,
User-agent: *
Disallow: /help/
i am little bit confused with the word "starts with"
[sample.com...] ( is it mean starts with /help/)?
and i want to know whether URLs are case sensitive?
thankz for ur reply
venu
And as someone else said, it is case-sensitive.