Welcome to WebmasterWorld Guest from 23.20.184.141

Forum Moderators: goodroi

Message Too Old, No Replies

My website ranking dropped!

after i create a wrong robotx.txt...

     
1:09 am on Jun 26, 2008 (gmt 0)

Full Member

5+ Year Member

joined:June 26, 2008
posts: 248
votes: 5


My website is a content website. It was doing great up to Thursday when by mistake i I did disallow / instead of disallow. So the next few days i noticed even bigger drop in SE.

I fixed this issue in Sunday but since then my organic ranking has not improved at all. On the contrary it is very bad. I have RSS, XML sitemap, txt sitemap and html sitemap and i add content all the time. So i wonder what must be wrong? Any ideas would be greatly appreciates!

Also if in my meta I use <meta name="robots" content="index,follow"> do I still need to use a robots.txt file:

User-agent: *
Disallow:

Thanks
Sisie

2:10 am on June 26, 2008 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 15, 2006
posts: 103
votes: 0


There are no files or folders that you need to block? It is hard to believe.

I think meta tag is not needed across the site if you are on top of the robots.txt file.

2:29 am on June 26, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


If you wish to allow robots to access everything, then you don't need the <meta name="robots" tag at all, and your robots.txt file can be blank. I recommend that even if it is blank, you should still have the file there, since this will prevent many 404 errors in your logs from robots trying to fetch it if it's missing.

It may take several more days or even a couple of weeks for your site to recover mostly, and longer to recover fully. Always review your work carefully and test it. There are several files in which you cannot afford to make mistakes, and robots.txt is one of them...

Jim

7:30 am on June 27, 2008 (gmt 0)

New User

5+ Year Member

joined:June 24, 2008
posts:6
votes: 0


I had the same experience but I got reindexed within few days...

Better update/create your Google Sitemap/XML file and resubmit it to Google Webmasters Tools...

4:42 pm on July 28, 2008 (gmt 0)

New User

5+ Year Member

joined:July 28, 2008
posts: 3
votes: 0


Google stopped indexing my site on July 11, and I found a robots.txt file that I did not place on the site that appeared to be blocking all search engines. We removed the file altogether. Is it possible that my host added this file, and will Google now begin indexing the site with no robots.txt in place? The site is: <snip>

Thanks for your help.

[edited by: goodroi at 3:32 am (utc) on July 29, 2008]
[edit reason] Please no specific URLs [/edit]

4:50 pm on July 28, 2008 (gmt 0)

Junior Member

10+ Year Member

joined:June 28, 2005
posts: 80
votes: 0


I had the same experience but I got reindexed within few days...

I've seen it take a few weeks. I think it depends on how popular your site was to start with.

I would recommend:
1) Resubmit your sitemap
2) Do a bit of good old fashioned link building

What is your webmaster tools analysis looking like right now? Does it have the latest robots.txt or is it showing the old version?

5:06 pm on July 28, 2008 (gmt 0)

New User

5+ Year Member

joined:July 28, 2008
posts: 3
votes: 0


Thanks for your help. The analysis reads 404 not found. Does this mean I need to create a robots.txt page and leave it blank, or will the search engines index the page anyway? I've never submitted a sitemap and have always pulled up very highly on search engines, I think largely because I have an extremely long list of quality links.

Thanks again, your help is most appreciated

5:33 pm on July 28, 2008 (gmt 0)

Junior Member

10+ Year Member

joined:June 28, 2005
posts: 80
votes: 0


If the robots.txt page in Webmaster Tools says 404 not found then it can't find the robots.txt file on your site. Have you deleted it?

If so, that's a good thing (for the moment) - Google should start indexing all of the URLs on your domain.

No robots.txt generally means "index everything" (as long as you don't have <meta name="robots" content="noindex,nofollow"> on your pages).

1:07 am on July 29, 2008 (gmt 0)

New User

5+ Year Member

joined:July 28, 2008
posts: 3
votes: 0


Thanks again for your help. Yes, we deleted the robots.txt file. I'll most likely get some help in creating a new one once Google starts indexing my pages.