Forum Moderators: goodroi

Message Too Old, No Replies

Filenames have spaces - How to reflect in robots.txt

Website has filenames that contain spaces. What to do?

         

SanDiego Art

11:40 pm on Mar 15, 2005 (gmt 0)

10+ Year Member



The filenames on my website have spaces in the filenames (built on MCMS). My question is this:

Do I include spaces in the filenames of the robots.txt file or do I represent spaces with the "+" symbol. I'm not sure if it matters in the robots.txt file but in Webtrends the "+" messed up the reporting.

Ex.

User-agent: *

Disallow: /Solutions/Solutions+by+Industry/FS/Credit+Test.htm

OR

Disallow: /Solutions/Solutions by Industry/FS/Credit Test.htm

Anyone have any experience with this? Thanks!

ThomasB

11:49 pm on Mar 15, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



SanDiego Art, first of all Welcome to WebmasterWorld!

The best way to avoid problems with search engines, other browsers and other tools you might use in the future is to use hyphens "-" instead of spaces. It's also much easier for robots.txt

SanDiego Art

11:54 pm on Mar 15, 2005 (gmt 0)

10+ Year Member



I know the filenames with spaces aren't necessarily the "best-practice" but it is a limitation of the content management system we are using in order to dynamically generate product index pages. Since there are no actual physical pages, they are built on top of templates. I'm in a new position at this company and the site structure pre-dates me...

So besides changing all my filenames to suit the robots.txt file, does anyone has experience or knowledge with FILENAMES WITH SPACES and robots.txt syntax.

Thanks

SanDiego Art

2:54 am on Mar 16, 2005 (gmt 0)

10+ Year Member



More information:

We also already use robots META tags, but we want use both the robots.txt and META tags in combination as some search engines are finding duplicate results (we have print version of some content and duplicate information in different channels for navigation reasons) and we don't want to get penalized.

Thanks