Another URL Removal cautionary tale

[tartle]

My entire hobby website has been wiped from google. No search results, no cache, no PR and no google directory listing. I used the URL Removal tool, but still have no idea why it wiped out my entire site.

I updated the robots.txt to get rid of some dead links. I am as paranoid as the next person, so I tested the robots.txt with googles analysis tool before submitting it to the URL Removal tool.

According to the Google URL Removal tool, I requested the removal of my entire site. Which I didn't. Then exactly one minute and one second later the log shows the robots.txt file that I did submit. For some reason it lists the contents of the file twice.

I tried to reproduce this with the same robots.txt and a couple of throw away sites. Both times around, Google behaved as I originally had expected it to. The robots.txt was listed once when submitted, and the root domain was not listed for removal. In fact, when I deliberately tried to remove the root domain as a dead URL, google correctly indicated that the page was still there.

I have read about others using this tool without problems, but from now on will be using a 410 in the .htaccess and using a little more patience.

As there is nothing I can do for six months, I guess this is the perfect time to finally change my domain name.

[tedster]

This is the third such report I've heard in the last two weeks. I've never ever used the Google URL removal tool, and now I have more paranoia about it than ever.

Sorry for your problem, but thanks for letting us know.

[activeco]

I used the removal tool too and got the confirmation that all "index.php" pages were de-indexed.
I made REAL static pages instead and index.php had been returning 404 for a long time, while Google still shown hundreds of index.php? pages as suplemental results.
Changed it to 410 and after a few weeks the pages were gone (accidentaly? at the same time as most supplemental pages by others), leaving only single, home page in the index.
Gotcha, I thought, until I noticed a lot of 404's in my logs.
Guess what: Like mad, Google has been still trying to crawl all index.php pages!
I don't think they are ALL linked from somewhere.

[g1smd]

There is no problem with Google asking for those pages. Their bot needs to confirm that the pages really are still "gone".

There would be a big problem if theythen showed those URLs in the index. From what you say they do not do that. And that is good.

[texasville]

>>>As there is nothing I can do for six months, I guess this is the perfect time to finally change my domain name.<<<
Couldn't you do a reinclusion request and submit a sitemap? Seems it might do an updated version of your site that way.

[activeco]

There is no problem with Google asking for those pages. Their bot needs to confirm that the pages really are still "gone".

g1smd: The url was confirmed as removed in March 2006.

[g1smd]

If you sold your domain, and the new owner put a page up at www.domain.com/that.page.html and it was never crawled or indexed, the new owner would be puzzled.

So, Google has to check the site from time to time to see that the page is still gone at that URL, or whether some new content has appeared there.

If it worked any other way, then once you let a page go 404 and it was deindexed, then you would never be able to put a page online at that URL ever again.

[kamran mohammed]

Thanks to tartle for sharing his experience with us on URL removal tool

I have used Google removal tool earlier and within 3-4 days Google has removed the pages which was listed in robots.txt file.

But now at the moment I am more skeptical about the URL removal tool as Google itself have mojor issues and bugs with it's latest update.

What I feel to do is now for a week is -

- Stop Link campaign
- Stop using URL removal Tool
- Deactivate Sitemaps Tool - if anyone is using it, it'e better to stop that for sometime.
- Stop all the major activites for Google & On Google.

Maybe it's better to take off to some cold place for a vacation as Matt is also on Vacation when webmasters need him - he is all enjoyin.

What does others has to say about it?

KaMran

[activeco]

If it worked any other way, then once you let a page go 404 and it was deindexed, then you would never be able to put a page online at that URL ever again.

Yes, but I don't think it is reasonable to crawl de-indexed pages who don't get any new links (and even the old links do not exist anymore).
It would be the same as crawling any random fictional page on a domain.

[activeco]

Addition:

Just found a visitor in stats who came via Google search.
I checked the phrase and my "de-indexed" url shown up as supplemental result!
When doing site:mysite no pages other than homepage are returned.

[cbartow]

The tool definitly doesn't do anything good. I've had a site gone for over 6 months and the entire web site is still in the index. I used the removal tool to remove each page over 2 weeks. It eventually was down to like 5-10 pages.

Now the entire site is back into the index.

[activeco]

I've had a site gone for over 6 months and the entire web site is still in the index.

If it is just returned to the index, then it is in compliance to what Google claims:

Pages removed using our automatic URL removal system are excluded from our index for at least 6 months regardless of whether they become available to our crawler during that time.

[google.com...]

It could be related to new and existing links, although I am inclined to think that tolbar users' bookmarks could do some messing here.

[twebdonny]

Our self imposed 180 days of delirium will be expiring next month...can't hardly wait to see the outcome

[trinorthlighting]

Submit a reinclusion request via sitemaps...