homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

robots.txt mess-up

 11:20 am on May 9, 2007 (gmt 0)


I'm doing some consultancy at a small web design firm and we've just spotted a major flaw; they had a no spider robots.txt in the root of their shared CMS directory - effectively banning search engines from the site.

We've now removed this; does anyone know if there is a way to 'nudge' Google or Yahoo into checking the robot.txt again so we can then submit a sitemap and try and get these sites 'visible' to the engines again?

Thanks for any help




 11:31 am on May 9, 2007 (gmt 0)

You could try submitting your sitemap through Google Webmaster Tools. Not sure however if that will bring the spider your way any quicker.
You can do the same on Yahoo Search Site Explorer.


 12:39 pm on May 9, 2007 (gmt 0)

The search engines always request the robots.txt when they visit a site. The best way to get the search engines to visit your site is to get mroe links to your site.

On a very well linked site that had the same problem as yours, Google took less than 24 hours to start indexing it. If you dont have a lot of links then the search engines may take several days/weeks before they revisit and notice the change to robots.txt.


 9:12 pm on May 18, 2007 (gmt 0)

Once they pick up the change, pages from the site will usually appear in the index within a week or so.

In fact, I have seen changes to robots.txt reflected in the index within 24 hours on some occasions.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved