Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt mess-up

11:20 am on May 9, 2007 (gmt 0)

New User

10+ Year Member

joined:Apr 24, 2004
votes: 0


I'm doing some consultancy at a small web design firm and we've just spotted a major flaw; they had a no spider robots.txt in the root of their shared CMS directory - effectively banning search engines from the site.

We've now removed this; does anyone know if there is a way to 'nudge' Google or Yahoo into checking the robot.txt again so we can then submit a sitemap and try and get these sites 'visible' to the engines again?

Thanks for any help


11:31 am on May 9, 2007 (gmt 0)

New User

5+ Year Member

joined:Apr 27, 2007
votes: 0

You could try submitting your sitemap through Google Webmaster Tools. Not sure however if that will bring the spider your way any quicker.
You can do the same on Yahoo Search Site Explorer.
12:39 pm on May 9, 2007 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
votes: 153

The search engines always request the robots.txt when they visit a site. The best way to get the search engines to visit your site is to get mroe links to your site.

On a very well linked site that had the same problem as yours, Google took less than 24 hours to start indexing it. If you dont have a lot of links then the search engines may take several days/weeks before they revisit and notice the change to robots.txt.

9:12 pm on May 18, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
votes: 0

Once they pick up the change, pages from the site will usually appear in the index within a week or so.

In fact, I have seen changes to robots.txt reflected in the index within 24 hours on some occasions.


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members