Welcome to WebmasterWorld Guest from 54.159.190.106

Forum Moderators: goodroi

Message Too Old, No Replies

Facing Problem With Robots.txt

Facing Problem With Robots.txt

   
9:02 am on Dec 24, 2009 (gmt 0)

5+ Year Member



I want suggestion for the below robots.txt code:

===================================================
User-agent: *
Disallow: /business/ # business directory
Disallow: /admin/ ? ? # admin directory
Disallow: /search/ ? # search page
Disallow: /stats/ ? # stats directory
===================================================

The syntax above is being displayed in Google Webmasters Tool "Crawler Access" section. But when look at the robots.txt at the root of my website then the code is being displayed as below:

===================================================
User-agent: *
Disallow: /business/ # business directory
Disallow: /admin/ # admin directory
Disallow: /search/ # search page
Disallow: /stats/ # stats directory
===================================================

So I want to know that:

1) why both syntax are different as Webmaster Tool Accesses the same lines which are contained in robots.txt text file.

2) What is the significance and effect of ? character after /business/ ?, /admin/ ?

Waiting for replies...............

10:24 pm on Dec 24, 2009 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Does the file include non-ASCII characters that do not show in your editor?

If there is a UTF-8 character that the web page (ie google WMT) cannot display in your browser then it will show a ? instead. This can happen with some editors - eg smart-quotes and long dashes do not always show on web pages unless they are spefically UTF-8 enabled.

View your robots file within the same web browser you view WMT (exmple . com/robots.txt).

Some editors allow you to switch on "code" characters, or you could read the file in a binary editor.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month