homepage Welcome to WebmasterWorld Guest from 54.197.147.90
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt - Site Disallowed Mistake?
mardomania




msg:4235774
 11:57 pm on Nov 27, 2010 (gmt 0)

Hi Everyone,

my programmer uploaded a robots.txt file that disallowed my entire website about 4 days ago. i just noticed it. my site wasn't getting crawled and i thought this was quite strange. of course, i deleted this file...but i hope i didn't find out too late.

is this going to hurt my site in any way? obviously my site is not in the serps, but i am hoping they come back now since I deleted this errounous file.

suffice to say, i'm real pissed at my programmer right now.

 

tedster




msg:4235803
 1:21 am on Nov 28, 2010 (gmt 0)

The major search engines will index the site eventually now that the incorrect robots.txt file is removed. I'd suggest you serve an actual file, rather than just deleting the previous one. You can simply use this syntax:

User-agent: *
Disallow:

That disallows nothing, and it explicitly allows the crawlers in. It might speed things up a few days.

mardomania




msg:4262005
 1:28 am on Feb 3, 2011 (gmt 0)

Tedster,
I took your advice and used that file. my site is showing in the google results now..but when I log into google webmaster tools.....i still see quite a bit of my urls restricted, including my home page (Says "URL Restricted by Robots.txt", detected November 20, 2010).
i'm assuming since they detected this around the time of the problem, google just stopped coming back to crawl my site? would do I do from here to get google to realize i am no longer restricting my home page from them?
Thanks,

JohnRoy




msg:4274438
 6:16 am on Mar 1, 2011 (gmt 0)

If there are links to these pages, google would eventually show up again.

g1smd




msg:4274495
 10:10 am on Mar 1, 2011 (gmt 0)

WebmasterTools might take many weeks to update.

It doesn't accurately reflect what is in their search index, WMT often lagging by days to weeks.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved