Welcome to WebmasterWorld Guest from 220.127.116.11 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
Facing Problem With Robots.txt Facing Problem With Robots.txt webexpert2u msg:4048844 9:02 am on Dec 24, 2009 (gmt 0) I want suggestion for the below robots.txt code:
User-agent: * Disallow: /business/ # business directory Disallow: /admin/ ? ? # admin directory Disallow: /search/ ? # search page Disallow: /stats/ ? # stats directory ===================================================
The syntax above is being displayed in Google Webmasters Tool "Crawler Access" section. But when look at the robots.txt at the root of my website then the code is being displayed as below:
User-agent: * Disallow: /business/ # business directory Disallow: /admin/ # admin directory Disallow: /search/ # search page Disallow: /stats/ # stats directory ===================================================
So I want to know that:
1) why both syntax are different as Webmaster Tool Accesses the same lines which are contained in robots.txt text file.
2) What is the significance and effect of ? character after /business/ ?, /admin/ ?
Waiting for replies...............
dstiles msg:4049054 10:24 pm on Dec 24, 2009 (gmt 0)
Does the file include non-ASCII characters that do not show in your editor?
If there is a UTF-8 character that the web page (ie google WMT) cannot display in your browser then it will show a ? instead. This can happen with some editors - eg smart-quotes and long dashes do not always show on web pages unless they are spefically UTF-8 enabled.
View your robots file within the same web browser you view WMT (exmple . com/robots.txt).
Some editors allow you to switch on "code" characters, or you could read the file in a binary editor.