Added robots.txt - 3 days later rankings gone!

[Block19Row13]

i added robots.txt because there was a page in my site i didnt want to be spidered.

i check my rankings a few days later and all my page 1's are languishing in pages 8 - 10

ive done without robots.txt for best part of a year and got great search engine listings.

now all the hard work has gone

[engine]

It may have been coincidence, however, what have you got in your robots.txt?

[Block19Row13]

i havent done anything else though that would have caused such a catastrphic result.

my robots says this

=============
User-agent: *
Disallow: sitemap.htm
=============

sitemap.htm is a nasty dynamic page that is created every time my site is published (i use e-commerce software), its about 300k, so i built my own.

maybe the search engine liked it, maybe thats why im being punished, for not wanting them to see it.

[engine]

Well, if it's the site in your profile, I can't see the file. If I can't see it, neither can the search engines. What I see is your home page.

Try using the the Brett's robots.txt validator [searchengineworld.com...]

[Block19Row13]

i got rid of it this morning thats why you can see it.

just dont what to do about it.

[Block19Row13]

sorry, thats why you cant see it

[engine]

Difficult to tell without seeing what's in the file you refer to. 300k is a little big. 100k or below is about the right size.

Your file called sitemap.htm I assume is a site map. A sitemap is usually a good way for search engine spiders to find their way around your site. By banning the robots reading it you're stopping them from traversing the site easily.

If you changed nothing else and you had good site performance previously it is unlikly to be anything else related to robots.txt, other than what I said above.

I assume you're talking about Google and have studied this. [webmasterworld.com]

[Block19Row13]

the dynamic site map is far too big.

so i built a light version - i know site maps are usefull, thats why i built one under 100k with less than 100links.

its so frustrating, every site seems to have survied!

only other thing i can think of is last week our service providers experienced a large period of outage and maybe google came along when we were down.

this, i pray is the answer!

[Lord Majestic]

User-agent: *
Disallow: sitemap.htm

This disallow directive will not work with robots.txt compliant crawlers because you missed path to the filename (/ if its the root). A pattern matching crawler like Googlebot may still work, but in principle all others will ignore your directive.

[Block19Row13]

well its removed now.

just to let you know, a different data centre is displaying g*****gle.co.uk / uk results and we are back.

i think it may have somethign to do with the downtime and a certain data centre trid to crawl.

had a torrid time trying to do so and sent us down.

[Matt Probert]

only other thing i can think of is last week our service providers experienced a large period of outage and maybe google came along when we were down.

That's a more likely explanation.

Matt