joined:May 21, 2004
Noticed this the other day... for the longest time my robots.txt followed the format of :
which is consistent with what Google shows you in their webmaster tools excluding the sitemap field.
Now, on robotstxt.org I noticed they state that there is no 'allow' field and the only requirement is atleast one 'disallow'. So in other words the above example would be changed to :
What is everyones thoughts on this? In theory it sounds like they accomplish the same exact thing, however, is the 'Allow: /' necessary... or on the flipside... would it hurt to have it?
Would like to hear opinions on this.