renewed site not being indexed - old pages in serp cache

how to fix robots.txt mistake

[Ring]

A new client came to us with the following problem:
Their renewed site (summer 2005), with added cms system, is still not being indexed in the SE's. However, their old site is still showing up in de SERP's
(cached version).

When investigating this problem we discovered that the webdeveloper had left following robots.txt file in the site root:
User-agent: *
Disallow: /

The renewed site also features a new url, but no decent 301 was used apparently ...

D'oh!

We have already removed this malicious robots.txt file and resubmitted the renewed site, but the SERP's are still not showing the new pages, only the old cached ones.

Could there be another problem? Does anyone know how we can get the renewed site indexed and the old pages in the SE's cache removed from their index?

Thanks very much in advance for your help!

[edited by: ThomasB at 10:52 am (utc) on Feb. 15, 2006]
[edit reason] URL removed. See TOS [/edit]

[Dijkgraaf]

How long has it been since you removed the global disallow from your robots.txt?
It can take a while before your site is re-crawled, and sometimes even longer before that crawl is included in the search engines results.

Getting the old results varies from search engine to search engine. You will have to visit their guides/FAQs to find out.

[Ring]

Hi Dijkgraaf,

Just removed the malignant robots.txt last week.
I then added a corrected robots.txt and also a Google sitemap.xml
Both have been crawled already. So I guess it's wait and see now.
How long do you think it will take before the pages show up in the serps? Are we talking days, weeks or months?

cheers,
Ring

[Dijkgraaf]

Hopefully it will be in the order of weeks.
However sometimes it can be months, especially if the search engine is in the midst of doing an upgrade to their indexing system.
Let us know how it goes.