How long does google take to remove 404 and 301 pages?

[idonen]

Back in the days before I found this site, I had a site which used 302 redirects from .asp pages with parameters in the querystring to url-rewritten pages, as well as the same pages with invalid parameters (because the items specified by the params no longer exist) back to the home page.

About a month ago, I took that version of the site down and made live a new version which is all url-rewritten, uses 301 redirects where appropriate to go from old .asp pages to the new versions, and uses 404 pages where appropriate when something doesnt exist any more.

Googlebot visits often, and hits these 301 and 404 pages, but the pages remain in the serps. Any idea how long it takes for 404s and 301s to disappear from the serps?

[g1smd]

It takes about 24 hours if you use the Google urlcontroller to remove them.

You have to register first.

[Span]

For pages that don't exist anymore you need to serve a '410 Gone' instead of a 404. Google understands a 410 and stops asking for them.

[annej]

I just forward people to the new page but I put a noindex meta tag up. That seems to get the page out of the serps in a couple of days. If a site doesn't get deep crawled very often I would be longer.

[iProgram]

hehe, I 301 redirected my old site to a new one at May 2004 and delete the whole old site last year. And now the old site (which does not exist at all) still has a PR5 and a lot of pages indexed by google. I can even see some of these pages in serp when I search something.

[g1smd]

Use the urlcontroller to get rid of them from the index again.

[johnpinochet]

iProgram,

You say the old site still has a PR of 5. What about the new site? Also is the new content the same or similar enough to the old content such that if both old and new are still in the SERPs you might receive a duplicate content penalty?

[idonen]

I'm trying out the 410 Gone.... I'll report back if/when it works...

[iProgram]

You say the old site still has a PR of 5. What about the new site?

PR old + PR new = PR 10 :-)

Also is the new content the same or similar enough to the old content such that if both old and new are still in the SERPs you might receive a duplicate content penalty?

Same content. That's why I used 301 redirection.

The problem is, I followed all steps of "Google web master info" to more domain old to domain new, and not work. (My new site is indexed too, but old site is still there)

[g1smd]

Use the urlcontroller to get rid of them from the index.

[catchmeifucan]

g1smd,

What is a urlcontroller?

[jakegotmail]

I am very curious also to find out what it is ...

[g1smd]

I used a well known search engine to look for it: [google.com ]

[GuinnessGuy]

G1,

Just curious, but shouldn't he take the 301 away first before doing that? Is it possible that the removal bot will be 301 re-directed to his new domain page and delete those urls instead?

Also, a friend of mine took down his entire site by using the urlcontroller. He accidentally entered:

h*tp://www.hisdomain.com/

There is an email address on the control panel page which google says to use if you have a problem, but it just auto-responded that no human reads emails from that address(nice feature, Google). So is he screwed now for 90 days or so? He did the addurl just today.

GuinnessGuy

[g1smd]

Google's communications are messed up. They tell you to send email to a certain address, then you get an autoresponder from that one saying the address is no longer in use. Yes there is a lot of that at the moment.

He can contact them via [google.com ] and I can confirm that that was working OK as of just a few days ago.

I just hope that you get a reply from someone who actually knows what they are talking about. Four of my friends asked the exact same question (about their own site each time) and got four different answers: ranging from "Yes", to "probably", to "probably not", to "definately No".

[GuinnessGuy]

G1,

just hope that you get a reply from someone who actually knows what they are talking about. Four of my friends asked the exact same question (about their own site each time) and got four different answers: ranging from "Yes", to "probably", to "probably not", to "definately No".

Sorry, but I'm not sure what you are referring to in the above statement. Is this in reference to the 301 question or the 90 days until re-inclusion question from my previous post?

GuinnessGuy

[g1smd]

No. I was saying that four people asked the same (unspecified here) question and they received four different answers.

The question was about indexing, but was not about 301s, redirects, or hijacking. It was a different question; but they received four completely different answers (and all on the same day too).

[idonen]

Also, google's own instructions specifically state that when you remove a page (which ordinarily would cause a 404) that the page will be removed from the serps and that when you 301 a page, google will automatically follow and update the index appropriately, but numerous people have reported problems with this.

[idonen]

Update: I updated my robots.txt to disallow the path which all my old outdated pages used, and then punched in the address of the robots.txt file into the urlcontroller, and the old pages are now no longer in the serps, and it took less than 24 hours. Not bad...

[idonen]

Quick question...

What's the robots.txt format for disallowing a file regardless of parameters?

I have a bunch of no-longer-existing pages of the form:
/showcat.asp?a=1&b=2

Can I just do:

Disallow: /showcat.asp

and that'll do the trick?

[g1smd]

It should do. It will disallow anything that starts with those characters.

Disallow: /foo

will disallow /foo and /foobar and /foo.htm and /foo/file.htm etc.