robots.txt mess-up

[strawbleu]

Hi,

I'm doing some consultancy at a small web design firm and we've just spotted a major flaw; they had a no spider robots.txt in the root of their shared CMS directory - effectively banning search engines from the site.

We've now removed this; does anyone know if there is a way to 'nudge' Google or Yahoo into checking the robot.txt again so we can then submit a sitemap and try and get these sites 'visible' to the engines again?

Thanks for any help

Jon

[daveozzz]

You could try submitting your sitemap through Google Webmaster Tools. Not sure however if that will bring the spider your way any quicker.
You can do the same on Yahoo Search Site Explorer.

[goodroi]

The search engines always request the robots.txt when they visit a site. The best way to get the search engines to visit your site is to get mroe links to your site.

On a very well linked site that had the same problem as yours, Google took less than 24 hours to start indexing it. If you dont have a lot of links then the search engines may take several days/weeks before they revisit and notice the change to robots.txt.

[g1smd]

Once they pick up the change, pages from the site will usually appear in the index within a week or so.

In fact, I have seen changes to robots.txt reflected in the index within 24 hours on some occasions.