Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: goodroi
In my robots.txt file statements are created like,
Dissallow : /help/
Now, for example my directors are like,
my queston is, which of the above link is restricted to the spiders.
1)the parent directory name /help/ will be restricted
2)or whatever it may be directory name(whether parent directory or sub directory) /help/ will be restricted
3) the directory codes/help/ also restricted?
Welcome to WebmasterWorld!
4) None. You have misspelled "Disallow:" and omitted the colon on "User-agent:". ;)
Robots use prefix-matching; Any URL which starts with the URL-path that you specify will not be fetched.
You said that, Any URL which starts with the URL-path that you specify will not be fetched.
for example, the robots file be,
i am little bit confused with the word "starts with"
[sample.com...] ( is it mean starts with /help/)?
and i want to know whether URLs are case sensitive?
thankz for ur reply