homepage Welcome to WebmasterWorld Guest from 54.197.183.230
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Facing Problem With Robots.txt
Facing Problem With Robots.txt
webexpert2u



 
Msg#: 4048842 posted 9:02 am on Dec 24, 2009 (gmt 0)

I want suggestion for the below robots.txt code:

===================================================
User-agent: *
Disallow: /business/ # business directory
Disallow: /admin/ ? ? # admin directory
Disallow: /search/ ? # search page
Disallow: /stats/ ? # stats directory
===================================================

The syntax above is being displayed in Google Webmasters Tool "Crawler Access" section. But when look at the robots.txt at the root of my website then the code is being displayed as below:

===================================================
User-agent: *
Disallow: /business/ # business directory
Disallow: /admin/ # admin directory
Disallow: /search/ # search page
Disallow: /stats/ # stats directory
===================================================

So I want to know that:

1) why both syntax are different as Webmaster Tool Accesses the same lines which are contained in robots.txt text file.

2) What is the significance and effect of ? character after /business/ ?, /admin/ ?

Waiting for replies...............

 

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4048842 posted 10:24 pm on Dec 24, 2009 (gmt 0)

Does the file include non-ASCII characters that do not show in your editor?

If there is a UTF-8 character that the web page (ie google WMT) cannot display in your browser then it will show a ? instead. This can happen with some editors - eg smart-quotes and long dashes do not always show on web pages unless they are spefically UTF-8 enabled.

View your robots file within the same web browser you view WMT (exmple . com/robots.txt).

Some editors allow you to switch on "code" characters, or you could read the file in a binary editor.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved