homepage Welcome to WebmasterWorld Guest from 54.161.200.144
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
My website ranking dropped!
after i create a wrong robotx.txt...
virtualreality




msg:3683615
 1:09 am on Jun 26, 2008 (gmt 0)

My website is a content website. It was doing great up to Thursday when by mistake i I did disallow / instead of disallow. So the next few days i noticed even bigger drop in SE.

I fixed this issue in Sunday but since then my organic ranking has not improved at all. On the contrary it is very bad. I have RSS, XML sitemap, txt sitemap and html sitemap and i add content all the time. So i wonder what must be wrong? Any ideas would be greatly appreciates!

Also if in my meta I use <meta name="robots" content="index,follow"> do I still need to use a robots.txt file:

User-agent: *
Disallow:

Thanks
Sisie

 

jeffposaka




msg:3683637
 2:10 am on Jun 26, 2008 (gmt 0)

There are no files or folders that you need to block? It is hard to believe.

I think meta tag is not needed across the site if you are on top of the robots.txt file.

jdMorgan




msg:3683643
 2:29 am on Jun 26, 2008 (gmt 0)

If you wish to allow robots to access everything, then you don't need the <meta name="robots" tag at all, and your robots.txt file can be blank. I recommend that even if it is blank, you should still have the file there, since this will prevent many 404 errors in your logs from robots trying to fetch it if it's missing.

It may take several more days or even a couple of weeks for your site to recover mostly, and longer to recover fully. Always review your work carefully and test it. There are several files in which you cannot afford to make mistakes, and robots.txt is one of them...

Jim

Senthil




msg:3684935
 7:30 am on Jun 27, 2008 (gmt 0)

I had the same experience but I got reindexed within few days...

Better update/create your Google Sitemap/XML file and resubmit it to Google Webmasters Tools...

catman




msg:3709383
 4:42 pm on Jul 28, 2008 (gmt 0)

Google stopped indexing my site on July 11, and I found a robots.txt file that I did not place on the site that appeared to be blocking all search engines. We removed the file altogether. Is it possible that my host added this file, and will Google now begin indexing the site with no robots.txt in place? The site is: <snip>

Thanks for your help.

[edited by: goodroi at 3:32 am (utc) on July 29, 2008]
[edit reason] Please no specific URLs [/edit]

JamieBrown




msg:3709397
 4:50 pm on Jul 28, 2008 (gmt 0)

I had the same experience but I got reindexed within few days...

I've seen it take a few weeks. I think it depends on how popular your site was to start with.

I would recommend:
1) Resubmit your sitemap
2) Do a bit of good old fashioned link building

What is your webmaster tools analysis looking like right now? Does it have the latest robots.txt or is it showing the old version?

catman




msg:3709417
 5:06 pm on Jul 28, 2008 (gmt 0)

Thanks for your help. The analysis reads 404 not found. Does this mean I need to create a robots.txt page and leave it blank, or will the search engines index the page anyway? I've never submitted a sitemap and have always pulled up very highly on search engines, I think largely because I have an extremely long list of quality links.

Thanks again, your help is most appreciated

JamieBrown




msg:3709457
 5:33 pm on Jul 28, 2008 (gmt 0)

If the robots.txt page in Webmaster Tools says 404 not found then it can't find the robots.txt file on your site. Have you deleted it?

If so, that's a good thing (for the moment) - Google should start indexing all of the URLs on your domain.

No robots.txt generally means "index everything" (as long as you don't have <meta name="robots" content="noindex,nofollow"> on your pages).

catman




msg:3709871
 1:07 am on Jul 29, 2008 (gmt 0)

Thanks again for your help. Yes, we deleted the robots.txt file. I'll most likely get some help in creating a new one once Google starts indexing my pages.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved