homepage Welcome to WebmasterWorld Guest from 54.227.20.250
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt mess-up
strawbleu

10+ Year Member



 
Msg#: 3334730 posted 11:20 am on May 9, 2007 (gmt 0)

Hi,

I'm doing some consultancy at a small web design firm and we've just spotted a major flaw; they had a no spider robots.txt in the root of their shared CMS directory - effectively banning search engines from the site.

We've now removed this; does anyone know if there is a way to 'nudge' Google or Yahoo into checking the robot.txt again so we can then submit a sitemap and try and get these sites 'visible' to the engines again?

Thanks for any help

Jon

 

daveozzz

5+ Year Member



 
Msg#: 3334730 posted 11:31 am on May 9, 2007 (gmt 0)

You could try submitting your sitemap through Google Webmaster Tools. Not sure however if that will bring the spider your way any quicker.
You can do the same on Yahoo Search Site Explorer.

goodroi

WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3334730 posted 12:39 pm on May 9, 2007 (gmt 0)

The search engines always request the robots.txt when they visit a site. The best way to get the search engines to visit your site is to get mroe links to your site.

On a very well linked site that had the same problem as yours, Google took less than 24 hours to start indexing it. If you dont have a lot of links then the search engines may take several days/weeks before they revisit and notice the change to robots.txt.

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3334730 posted 9:12 pm on May 18, 2007 (gmt 0)

Once they pick up the change, pages from the site will usually appear in the index within a week or so.

In fact, I have seen changes to robots.txt reflected in the index within 24 hours on some occasions.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved