homepage Welcome to WebmasterWorld Guest from 54.227.41.242
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
About robots.txt file
Confusion in robots.txt file...i need hep..
venuc




msg:1527669
 5:37 pm on Dec 16, 2004 (gmt 0)

i want to add robots.txt for my server.
i have doubt in how spider interperate robots.txt
content.

for example:
In my robots.txt file statements are created like,

user-agent *
Dissallow : /help/

Now, for example my directors are like,

[sample.com...]
[sample.com...]

my queston is, which of the above link is restricted to the spiders.

1)the parent directory name /help/ will be restricted
2)or whatever it may be directory name(whether parent directory or sub directory) /help/ will be restricted
3) the directory codes/help/ also restricted?

 

jdMorgan




msg:1527670
 5:45 pm on Dec 16, 2004 (gmt 0)

venuc,

Welcome to WebmasterWorld!

4) None. You have misspelled "Disallow:" and omitted the colon on "User-agent:". ;)

Robots use prefix-matching; Any URL which starts with the URL-path that you specify will not be fetched.

[b]U[/b]ser-agen[b]t:[/b] *
Di[b]s[/b]allow: /help/

This will request that robots not fetch any URLs starting with "/help/".

See [robotstxt.org...]

Jim

treeline




msg:1527671
 9:03 pm on Dec 16, 2004 (gmt 0)

Another useful tool, and links to understanding robots.txt is at:

[searchengineworld.com...]

Hints




msg:1527672
 2:58 am on Dec 17, 2004 (gmt 0)

Remember that directory and filnames are case sensitive.

Hints

venuc




msg:1527673
 4:01 pm on Dec 17, 2004 (gmt 0)

Dear dMorgan,

You said that, Any URL which starts with the URL-path that you specify will not be fetched.

for example, the robots file be,

User-agent: *
Disallow: /help/

i am little bit confused with the word "starts with"

[sample.com...] ( is it mean starts with /help/)?

and i want to know whether URLs are case sensitive?

thankz for ur reply
venu

teaperson




msg:1527674
 5:52 pm on Dec 20, 2004 (gmt 0)

The spider disregards the domain, so in your example [sample.com...] the spider would consider it to start with /help/.

And as someone else said, it is case-sensitive.

venuc




msg:1527675
 5:32 am on Dec 21, 2004 (gmt 0)

Hi everybody,

i really thankfull to all of you valuable reply..

thank you very much

venu

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved