homepage Welcome to WebmasterWorld Guest from 23.22.173.58
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Not sure how to handle space character
Jon_King

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 521 posted 2:44 pm on Dec 22, 2004 (gmt 0)

If disallowing a subdir with a space in the name which is correct:

Disallow:/new directory/

or

Disallow:/new%20directory/

 

jatar_k

WebmasterWorld Administrator jatar_k us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 521 posted 2:55 pm on Dec 22, 2004 (gmt 0)

spaces aren't really valid in directory names or urls so I am not sure there is a valid format for robots.txt

I looked around at a bunch of tutorials and saw no references

Jon_King

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 521 posted 3:12 pm on Dec 22, 2004 (gmt 0)

Yes, I wouldn't have used the space, but it's something tough to get changed in this particular organization.

I couldn't find a reference to it either.

encyclo

WebmasterWorld Senior Member encyclo us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 521 posted 3:27 pm on Dec 22, 2004 (gmt 0)

I would disallow both versions to cover both bases, but I would but the version with the space last in the robots.txt file in case it causes a parse error for some bots.

The trouble is, with no certainty of exactly how this situation is supposed to be handled, you can expect problems. Is it possible to add a robots meta tag to those pages also?

Jon_King

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 521 posted 1:48 am on Jan 6, 2005 (gmt 0)

A follow up. It seems the %20 does indeed hang/confuse some bots. Repeated requests for the robots.txt, and the pages in the disallowed directory are indexed.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved