Page is a not externally linkable
reseller - 8:37 am on Dec 20, 2006 (gmt 0)
A designer or a program has left the following robots.txt behind: Thats not some theoritical example, or a jock. Its exactly what happened in real life! Maybe it was done in good faith while developing the site, and the designer/programmer just forgot to edit the robots.txt after finishing the developement work to read, for example: User-agent: * But he/she did write the "# Be nice.". Go figure ;-)
When you notice a site has started behaving strange on the serps and the number of indexed pages have started to decline, you may wish to start your SEO work by checking out an obvious factor in SEO; robots.txt.
# Robots.txt
# Be nice.
#
User-agent: *
Disallow: /
# Robots.txt
Disallow: