Google dropped me

how much does robots.txt have to do with it?

[webwoman]

I recently changed hosting services and in the change I failed to upload a robots.txt file. This was in early May, I discovered it in June and corrected it right away (late June) My referrals from Google have been diminishing for a month, and just at the start of July my site was completely gone from Google. We had been on page 1,2 or 3 for most of our keywords for 2 years. When I type in the actual name of our company, it fails to give me our site, but does give all the various links to us. It appears we are completely off the engine...

My question is this: Could lack of a robots.txt file cause complete obliteration from Google? If not, what are some other things that could cause this?

[ScottM]

My understanding is that the robots.txt only EXCLUDES 'bots.

I don't have a single robots.txt on any of my domains. All do well in Google.

I'd look to see if the nameservers have updated...which is beyond my ability to answer.

Others want to chime in?

[pageoneresults]

> I recently changed hosting services and in the change I failed to upload a robots.txt file.

All depends on what the robots.txt file was doing.

[egomaniac]

The mere presence or lack or presence of a robots.txt file will not cause you to be banned or penalized in Google.

Is it possible that by eliminating the robots.txt file, you exposed pages of duplicate or near duplicate content that you had previously excluded?

To see which of your pages are in Google, use this search:

inurl:mydomain.com

[mack]

last time googlebot visited my site it didnt even try and "get" robots.txt so i dont think that would be the only reason. I would say it is more likely to be the server change it'self that had caused the problems.. thre have been some pretty recent posts about this sort of thing.

[jdMorgan]

Also, are the pages on your old hosting service still accessible? If you kept
the same domain name and changed DNS to point to your new server, then this
isn't a problem. But if you also changed domain names and the old pages are
still on-line, you may have hit a duplicate content filter.

Also, someone here has previously reported that Google handles their own DNS,
and that sometimes their DNS gets a bit stale. They may have tried to reach
your site at the old IP address and had a problem since it's gone missing.

In that case there's nothing for it but to wait for the next update.

HTH,
Jim

[pageoneresults]

webwoman, if its the site in your profile, there are some hosting issues.

I just did a ping and traceroute. Here are the results. This could be temporary, or it could be something that is ongoing. If it is ongoing, then this would be the first culprit at the top of my list!

On TraceRoute, I have a maximum of 32 hops set as the default. Hops 11 thru 22 timed out. I stopped it there.

On Ping all timed out. I'm able to view your site but it is very slow to load and I'm on a T1.

[msr986]

See also:

[webmasterworld.com...]

concerning robots.txt and custom 404 pages.

[webwoman]

Thank you all for the various responses.

To Pageone, thanks for the results of your TraceRoute. I contacted SBC (the new hosting service) and they told me that they do not allow pinging and have traceroute disabled. When I tried it myself I noticed that it died somewhere in san jose...

But, the other thing that I discovered is that some fly by night "financial portal" site has mirrored my site and submitted it to Google. I have traced them to their ISP and sent an abuse complaint, also I have emailed help@google to explain the situation and let them know the mirrored site has nothing to do with our company.

I am not so hopeful on hearing back from help@google - perhaps someone knows of an actual person at google I could contact? I do expect the ISP to co-operate and if not, I will file a DMCA complaint.

Sheeeesh - this is a huge hassle. I wouldn't care so much except that Google is (was) my top referral engine....

Any other suggestion would be very welcome and thanks again for the input.