Forum Moderators: goodroi

Message Too Old, No Replies

Facing Problem With Robots.txt

Facing Problem With Robots.txt

         

webexpert2u

9:02 am on Dec 24, 2009 (gmt 0)

10+ Year Member



I want suggestion for the below robots.txt code:

===================================================
User-agent: *
Disallow: /business/ # business directory
Disallow: /admin/ ? ? # admin directory
Disallow: /search/ ? # search page
Disallow: /stats/ ? # stats directory
===================================================

The syntax above is being displayed in Google Webmasters Tool "Crawler Access" section. But when look at the robots.txt at the root of my website then the code is being displayed as below:

===================================================
User-agent: *
Disallow: /business/ # business directory
Disallow: /admin/ # admin directory
Disallow: /search/ # search page
Disallow: /stats/ # stats directory
===================================================

So I want to know that:

1) why both syntax are different as Webmaster Tool Accesses the same lines which are contained in robots.txt text file.

2) What is the significance and effect of ? character after /business/ ?, /admin/ ?

Waiting for replies...............

dstiles

10:24 pm on Dec 24, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Does the file include non-ASCII characters that do not show in your editor?

If there is a UTF-8 character that the web page (ie google WMT) cannot display in your browser then it will show a ? instead. This can happen with some editors - eg smart-quotes and long dashes do not always show on web pages unless they are spefically UTF-8 enabled.

View your robots file within the same web browser you view WMT (exmple . com/robots.txt).

Some editors allow you to switch on "code" characters, or you could read the file in a binary editor.