Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Robots.txt - Site Disallowed Mistake?


tedster - 1:21 am on Nov 28, 2010 (gmt 0)


The major search engines will index the site eventually now that the incorrect robots.txt file is removed. I'd suggest you serve an actual file, rather than just deleting the previous one. You can simply use this syntax:

User-agent: *
Disallow:

That disallows nothing, and it explicitly allows the crawlers in. It might speed things up a few days.


Thread source:: http://www.webmasterworld.com/robots_txt/4235802.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com