0verdose - 11:57 am on Aug 22, 2013 (gmt 0)
the disallow bit wasn't so important, I just left it in to remind myself how to disallow certain parts of the site... I can remove it.
So if just reads:
and its named robots.txt placed at www.domain.com/robots.txt it will be found regardless? There's no other factors?
I'm sure I checked it and it works, will investigate further and find more details about where I saw it was never crawled.
Thanks so much for your response..