- - Google
- -- Google SEO News and Discussion
- ---- Changed Robots.txt - now googlebots are blocked?
lucy24 - 11:20 pm on Mar 28, 2013 (gmt 0)
A compliant robot MUST honor the "disallow" directive.
A compliant robot may CHOOSE to honor non-standard directives such as "allow".
The word "compliant" means everything.
There are no numbered versions of the robots.txt standard.
|The /robots.txt standard is not actively developed. |
|Also, you may not have blank lines in a record, as they are used to delimit multiple records. |
Thread source: http://www.webmasterworld.com/google/4559189.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com