homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Robots.txt - Site Disallowed Mistake?

 11:57 pm on Nov 27, 2010 (gmt 0)

Hi Everyone,

my programmer uploaded a robots.txt file that disallowed my entire website about 4 days ago. i just noticed it. my site wasn't getting crawled and i thought this was quite strange. of course, i deleted this file...but i hope i didn't find out too late.

is this going to hurt my site in any way? obviously my site is not in the serps, but i am hoping they come back now since I deleted this errounous file.

suffice to say, i'm real pissed at my programmer right now.



 1:21 am on Nov 28, 2010 (gmt 0)

The major search engines will index the site eventually now that the incorrect robots.txt file is removed. I'd suggest you serve an actual file, rather than just deleting the previous one. You can simply use this syntax:

User-agent: *

That disallows nothing, and it explicitly allows the crawlers in. It might speed things up a few days.


 1:28 am on Feb 3, 2011 (gmt 0)

I took your advice and used that file. my site is showing in the google results now..but when I log into google webmaster tools.....i still see quite a bit of my urls restricted, including my home page (Says "URL Restricted by Robots.txt", detected November 20, 2010).
i'm assuming since they detected this around the time of the problem, google just stopped coming back to crawl my site? would do I do from here to get google to realize i am no longer restricting my home page from them?


 6:16 am on Mar 1, 2011 (gmt 0)

If there are links to these pages, google would eventually show up again.


 10:10 am on Mar 1, 2011 (gmt 0)

WebmasterTools might take many weeks to update.

It doesn't accurately reflect what is in their search index, WMT often lagging by days to weeks.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved