Welcome to WebmasterWorld Guest from 18.104.22.168 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
About robots.txt file Confusion in robots.txt file...i need hep.. venuc
i want to add robots.txt for my server. i have doubt in how spider interperate robots.txt content.
In my robots.txt file statements are created like,
Dissallow : /help/
Now, for example my directors are like,
...] sample.com [ ...] sample.com
my queston is, which of the above link is restricted to the spiders.
1)the parent directory name /help/ will be restricted
2)or whatever it may be directory name(whether parent directory or sub directory) /help/ will be restricted 3) the directory codes/help/ also restricted?
Welcome to WebmasterWorld!
4) None. You have misspelled "Disallow:" and omitted the colon on "User-agent:". ;)
Robots use prefix-matching; Any URL which
starts with the URL-path that you specify will not be fetched. [b]U[/b]ser-agen[b]t:[/b] * Di[b]s[/b]allow: /help/ This will request that robots not fetch any URLs starting with "/help/".
Another useful tool, and links to understanding robots.txt is at:
...] searchengineworld.com Hints
Remember that directory and filnames are case sensitive.
You said that, Any URL which starts with the URL-path that you specify will not be fetched.
for example, the robots file be,
i am little bit confused with the word "starts with"
...] ( is it mean starts with /help/)? sample.com
and i want to know whether URLs are case sensitive?
thankz for ur reply
The spider disregards the domain, so in your example [ ...] the spider would consider it to start with /help/. sample.com
And as someone else said, it is case-sensitive.
i really thankfull to all of you valuable reply..
thank you very much