Welcome to WebmasterWorld Guest from 54.167.157.247

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt mess-up

   
11:20 am on May 9, 2007 (gmt 0)

10+ Year Member



Hi,

I'm doing some consultancy at a small web design firm and we've just spotted a major flaw; they had a no spider robots.txt in the root of their shared CMS directory - effectively banning search engines from the site.

We've now removed this; does anyone know if there is a way to 'nudge' Google or Yahoo into checking the robot.txt again so we can then submit a sitemap and try and get these sites 'visible' to the engines again?

Thanks for any help

Jon

11:31 am on May 9, 2007 (gmt 0)

5+ Year Member



You could try submitting your sitemap through Google Webmaster Tools. Not sure however if that will bring the spider your way any quicker.
You can do the same on Yahoo Search Site Explorer.
12:39 pm on May 9, 2007 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



The search engines always request the robots.txt when they visit a site. The best way to get the search engines to visit your site is to get mroe links to your site.

On a very well linked site that had the same problem as yours, Google took less than 24 hours to start indexing it. If you dont have a lot of links then the search engines may take several days/weeks before they revisit and notice the change to robots.txt.

9:12 pm on May 18, 2007 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Once they pick up the change, pages from the site will usually appear in the index within a week or so.

In fact, I have seen changes to robots.txt reflected in the index within 24 hours on some occasions.