homepage Welcome to WebmasterWorld Guest from 54.205.207.201
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt - Site Disallowed Mistake?
mardomania

5+ Year Member



 
Msg#: 4235802 posted 11:57 pm on Nov 27, 2010 (gmt 0)

Hi Everyone,

my programmer uploaded a robots.txt file that disallowed my entire website about 4 days ago. i just noticed it. my site wasn't getting crawled and i thought this was quite strange. of course, i deleted this file...but i hope i didn't find out too late.

is this going to hurt my site in any way? obviously my site is not in the serps, but i am hoping they come back now since I deleted this errounous file.

suffice to say, i'm real pissed at my programmer right now.

 

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4235802 posted 1:21 am on Nov 28, 2010 (gmt 0)

The major search engines will index the site eventually now that the incorrect robots.txt file is removed. I'd suggest you serve an actual file, rather than just deleting the previous one. You can simply use this syntax:

User-agent: *
Disallow:

That disallows nothing, and it explicitly allows the crawlers in. It might speed things up a few days.

mardomania

5+ Year Member



 
Msg#: 4235802 posted 1:28 am on Feb 3, 2011 (gmt 0)

Tedster,
I took your advice and used that file. my site is showing in the google results now..but when I log into google webmaster tools.....i still see quite a bit of my urls restricted, including my home page (Says "URL Restricted by Robots.txt", detected November 20, 2010).
i'm assuming since they detected this around the time of the problem, google just stopped coming back to crawl my site? would do I do from here to get google to realize i am no longer restricting my home page from them?
Thanks,

JohnRoy

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4235802 posted 6:16 am on Mar 1, 2011 (gmt 0)

If there are links to these pages, google would eventually show up again.

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4235802 posted 10:10 am on Mar 1, 2011 (gmt 0)

WebmasterTools might take many weeks to update.

It doesn't accurately reflect what is in their search index, WMT often lagging by days to weeks.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved