Out the other side? 6 months after using the removal tool

[Nick0r]

Has anyone successfully pasted the 6 month mark with the URL remove tool and then gone on to get those pages re-indexed? The 6 month mark has just passed me, but I don't see any increased activity by google, or any new results in a site: search yet.

[lammert]

After the six months (which is in my experience sometimes a few days longer) the pages will come back again at approximately the same position in the SERPs as when they were deleted.

You can find more info about the URL removal tool in this previous thread [webmasterworld.com].

[g1smd]

For pages that I no longer wanted to be indexed I used the removal tool to remove them, mostly using a robots.txt file to confirm the removal, but some were done for pages that simply returned 404 status.

Some came back after 3 months, others after 6 months. Even if the page was 404, or the domain no longer existed, Google added them all back into the results as if nothing had happened.

[Nick0r]

Hmmm - I removed a dynamic file, e.g thread.php that shows threads from a forum. I wonder if it'll go back to indexing it again, including variables like thread.php?t=124125

[Nick0r]

Gotta bump this thread because I never got a definitive answer. I've given it more time.

Quite a while ago (about 6 and a half months ago), I used the remove URL tool on my robots.txt. It had showthread.php in it which was to block out all of my old forum threads, as I had changed to a rewrite system that used blah-blah-blah.html as the URLs. My aim was to prevent a duplicate content penalty. I then reverted to a showthread.php system and scrapped my rewrites. Now I'm in a situation where google won't go back and re-index any new threads, or even older threads that were around in the time I used the rewrite tool.

A site: search shows a huge number of pages at the top... but when you actually visit each page of the site: results, there are probably only 20-30 URLs that are there (all non-showthread.php).

I'd really like peoples advice on what to do here.... I don't really want to change showthread.php to thread.php but I will if that is a last resort. I can't imagine how much traffic I'm losing from all the tens of thousands of threads on my forum, that could be ranking for tail terms.

[Nick0r]

Bump, anyone?

[CainIV]

I think possibly your asnwer has been answered. Pages which were removed have been reindexed.

[Nick0r]

That's not the case for me though, and I've waited over 6 months.

[lammert]

Did you change your robots.txt or robots meta tags in such a way that Googlebot has access again? If Google can't find the pages or is not allowed to crawl or index them, they might not reappear after the six month period or only for a short time or as supplemental results.

Another problem could be the Bigdaddy rollout. Many threads here report a lower number of pages in the Bigdaddy index for specific types of sites.

[Nick0r]

I removed all mention of those files from robots.txt a long time ago, and they are these pages are linked to from my homepage amongst others.

[lammert]

In that case they should come back in the index after six months, either as normal listings or as supplemental. Did you check the status in the URL remove console of the URLs? After the six months period, the status changes from "complete" to "expired".

[Nick0r]

Weird,

That doesn't seem to have happened - maybe I'm one month off.

Date of removal: 2005-08-04 10:49:08 GMT

Date today: 2006-02-20

Hmmm....

[g1smd]

What is the status for each URL that is listed?

[steve40]

Nick

I have a site just come back into the index removed early august not sure of the date but only about 10% of the site back so far with new green showing on pages that are back in mainly on top level pages only appears only pages pr2 or pr3 and above have so far been reincluded not positive but thats what i suspect

so dont think gbot has not yet gone deep enough suspect with luck if they are coming back will be in next 2 weeks

steve

[Nick0r]

g1smd,

Just checked, the status of the showthread pages are 'completed' but for some reason some pages i removed at the same time are now set to 'expired', those pages never existed though - weird!

[g1smd]

The "expired" pages will re-appear as "Supplemental Results"... a direct effect of the "our index is bigger than your index" methodology that they follow, rather than one that has a coherant database representing what really exists on the web.

[Nick0r]

Hmmmm... but why would pages that _didn't/don't_ exist expire, yet pages that do exist are still 'complete'?

Also ones those pages eventually (fingers crossed) reach expired status, will there ever be a chance of them returning to the normal, main index and ranking again?
- As I said before these pages SHOULD rank, and it was probably a mistake using the remove tool on them, if there is no chance of them ranking again I will have to change the filenames and 301 them, I'm missing out on a lot of trafifc.

Unsure of what to do, looking for more advice.

[Nick0r]

OK I really am unsure of what to do now. It's been 8 months and these pages have been spidered for over a month now (googlebot is actively crawling them), but still no sign in SERPs and the status in the URL remove tool is refusing to go to 'Expired'.

I'm seriously considering just renaming these pages, but because there is 100,000 of them, I'm worried that i'll just be in a worse position then when I first started with google.

[texasville]

Nickor,
I would wait on doing anything until big daddy has come to fruition. Seems a lot of things are happening and reindexing might be one of them. Renaming now might prove to be bad as the old pages may still come back and then you would be hit with dup content.