Welcome to WebmasterWorld Guest from 54.205.209.95

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt - Site Disallowed Mistake?

   
11:57 pm on Nov 27, 2010 (gmt 0)

5+ Year Member



Hi Everyone,

my programmer uploaded a robots.txt file that disallowed my entire website about 4 days ago. i just noticed it. my site wasn't getting crawled and i thought this was quite strange. of course, i deleted this file...but i hope i didn't find out too late.

is this going to hurt my site in any way? obviously my site is not in the serps, but i am hoping they come back now since I deleted this errounous file.

suffice to say, i'm real pissed at my programmer right now.
1:21 am on Nov 28, 2010 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



The major search engines will index the site eventually now that the incorrect robots.txt file is removed. I'd suggest you serve an actual file, rather than just deleting the previous one. You can simply use this syntax:

User-agent: *
Disallow:

That disallows nothing, and it explicitly allows the crawlers in. It might speed things up a few days.
1:28 am on Feb 3, 2011 (gmt 0)

5+ Year Member



Tedster,
I took your advice and used that file. my site is showing in the google results now..but when I log into google webmaster tools.....i still see quite a bit of my urls restricted, including my home page (Says "URL Restricted by Robots.txt", detected November 20, 2010).
i'm assuming since they detected this around the time of the problem, google just stopped coming back to crawl my site? would do I do from here to get google to realize i am no longer restricting my home page from them?
Thanks,
6:16 am on Mar 1, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If there are links to these pages, google would eventually show up again.
10:10 am on Mar 1, 2011 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



WebmasterTools might take many weeks to update.

It doesn't accurately reflect what is in their search index, WMT often lagging by days to weeks.