Welcome to WebmasterWorld Guest from 34.236.216.93

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

URL Console removal expired

After six months, removed pages marked as expired

     
8:45 pm on May 12, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 15, 2005
posts:380
votes: 0


Some members quite recently removed their sites struggling to remove 302 hijacks with URL Console.

Myself, I've lost a root URL in one site while trying to remove www and no-www duplicate. But it was almost six months ago. It will be six months next week.

And I can expect the site back a week later - because I just noticed some earlier removed URLs marked as expired.

More precisely, all URLs removed six months and one week ago are marked as expired, while all other are still marked as complete as they used to.

I write this, because I want to share with members waiting for their accidentally removed sites to be back, that six months period appears to work, in the contrary to "90 days" we believed before.

5:35 pm on May 13, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 15, 2005
posts:380
votes: 0


And today all removed URLs older than six months and five days are expired - so it isn't always six months and a week, but six months and several days.
7:24 pm on May 13, 2005 (gmt 0)

New User

10+ Year Member

joined:Jan 28, 2005
posts:14
votes: 0


hi Wizard

I removed a site from google back in Novemember and am still waiting for its return.

Are you saying that after 6 months it could return? because I emailed google back in Feburary and they told me it would be 3 months and then failed to reply to any of my follow up messages when the site remained out of their index after the three month period passed.

Also what do you mean by marked as expired?

Any advice would really be appreciated here, I am going nuts about this.

8:11 pm on May 13, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 15, 2005
posts:380
votes: 0


I used URL Console for the first time on 5.11.2004, removing several outdated links. Then, I destroyed my main page on 19.11.2004 (the problem was related to www and non-www handling by URL Console, discussed in other threads before).

After 90 days I emailed Google, and they told me about 90 days, so I emailed again saying that 90 days had passed, they sent another automatic response, but a few weeks later WebmasterWorld members noticed that information in URL Console started to say 'six moths' instead of 90 days.

Yesterday, I noticed my URLs removed on 5.11.2004 marked as expired instead of complete - I mean, in URL Console you can select 'show all requests' and see all pages you ever removed, with status either pending, complete or expired.

I don't care to check if these already expired removal URL entries can be reincluded in the index, because they are meant to stay out of the index, but after about two weeks my main URL will likely get expired status and then I'll know if it's reincluded or not - it's spidered at least daily all these six months and some PR checkers show it's PR, and it raised after last toolbar update, so I expect it to be back soon.

I think your page will also get back, but when - it depends on what exact day you removed it.

9:07 pm on May 14, 2005 (gmt 0)

New User

10+ Year Member

joined:Sept 9, 2003
posts:16
votes: 0


Hello,

Yesterday I've remove, with the remove url console , about 4000 pages without title and descriptions on google serp.
The pages are topic*.html (*=1,2,3,4,5,6,7...)

Today, the url console says "COMPLETED".

Now I want to force google to re-indexing those pages and I've remove from my robots.txt " DISALLOW /topic "

What could I do now? Could I rename the old pages from "topic" to "topic-" to invited Google to index as a new pages or leave the pages with their old names?

Thank you

(sorry for my english)

5:33 pm on May 15, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 15, 2005
posts:380
votes: 0


You cannot re-index these pages under old URLs before six months for removal date. You can re-index them under any other URLs, or, strictly saying, under any URLs Google considers different.

Adding a hyphen will work, however I'm sure you can figure out something better to change in it - you could add a keyword eventually.

7:29 pm on May 15, 2005 (gmt 0)

New User

10+ Year Member

joined:Sept 9, 2003
posts:16
votes: 0


Thank you so much wizard, sorry for little OT...

Bye Tintin

6:19 am on May 17, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 15, 2005
posts:380
votes: 0


More and more URLs goes to expired status, generally all that are older than six months and six days.

For one day, a few removed pages, which don't exists for over six months (404), appeared in "site:" results with snippets, but now they disappeared.

But there are two URLs that still show complete even if they have been removed on 9.11.2004.

5:49 pm on May 24, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 15, 2005
posts:380
votes: 0


My site is back!

Accidentally removed on 19.11.2004, when I attempted to remove www.site.xx and lost both www and non-www, now has expired in URL Console and shows in site: search with snippet and fresh tag. Cache and tPR still can't be seen, but it's likely to be datacenters issue. R...brick tool shows the same PR it gained in tPR update last month (it was removed, but still PR tools showed PR and PR was passed!).

I'm so happy, thank you Google!

P.S. My other removed sites are also back and already with PR, so it's certain it's going to be back.

6:32 pm on May 24, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 8, 2004
posts:176
votes: 0


Thanks for the update, Wizard. Does your newly listed site rank the way it used to?
5:52 am on May 25, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 15, 2005
posts:380
votes: 0


Today it has PR back and from the start it ranked well.
11:11 pm on May 29, 2005 (gmt 0)

New User

10+ Year Member

joined:Jan 28, 2005
posts:14
votes: 0


My site has now expired and has reappeared in the index. (it was just over six months)

Hurray.

Well spotted wizard.

12:24 am on May 30, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 1, 2004
posts:1987
votes: 0


I'm counting the days till August 15th.

Maybe then my homepage will be back and I'll have recovered from an overdose of Bourbon update too!

8:15 pm on May 30, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 10, 2003
posts:929
votes: 13


For future reference, if your home page is removed from the index by accident, couldn't you just rename it (or copy it) to say index2.htm and then 301 redirect index.htm to index2.htm?
Won't Googlebot then find index2.htm via previous external links to index.htm (and perhaps some new internal links from your other indexed pages), index it, and transfer all the PR because of the 301?
8:03 am on May 31, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 15, 2005
posts:380
votes: 0


Mike: of course it will work, but I had removed main index, "mysite.#*$!.yy/", so I couldn't do it this way - if I removed / to /sth.html it would work, but I obviously wanted / listed!

There is another problem with expiring URL removal - Google system doesn't check if expired pages still exist, and now I have many 404 pages listed and I have to remove them in URL Console again. I don't like doing it every six months, so I hope Google Team will do something with this. They should just crawl expired page and don't reinclude it if it returns 404!

10:05 am on May 31, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 1, 2004
posts:1987
votes: 0


I had removed:

mydomain.com/
mydomain.com/index.htm

And created a new homepage mydomain.com/index.html

This didn't work even though the index.html has a PR and shows the normal backlinks from the toolbar. There is no cache and is not searchable in the G index.

The end of Site: command list always show #*$! of xxx+1 pages and I believe the +1 page is the homepage which they won't show.

Apparently no one can damage your website except yourself with G's help, lack of support and arbitrary ill-defined rules and instructions. Thanks again G.

5:07 pm on May 31, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 15, 2005
posts:380
votes: 0


Google treats domain.com/, domain.com/index.htm and domain.com/index.html as the same URL, that's why you didn't succeed.

There used to be a bug, that you could remove competitor's site removing their site.com/index.html IF they had no index.html (but index.php or default.asp instead). One WW member exploited it to remove microsoft.com and adobe.com, and then Google repaired the bug in a few days, but it still treats these three URLs as one so if you remove any of them, you're out of all.

6:56 am on June 1, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 8, 2001
posts:2882
votes: 0


It's probably 180 days instead of six months. I think that a few people are working on going through our webmaster pages to find any stale info (e.g. "90 days" when it should say 180 days) and bring things more up to date.
3:07 pm on June 1, 2005 (gmt 0)

New User

10+ Year Member

joined:May 31, 2004
posts:39
votes: 0


Hi GoogleGuy,

May I ask is it possible to ask Google for help to reduce this 180 days period? Since I accidently removed all pages of a site (I was trying to remove all pages without www only).

Thanks.

3:42 pm on June 1, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 1, 2004
posts:1987
votes: 0


Wizard,

If those three are treated the same then why does my domain.com/index.html have PR and backlinks (no cache) whereas domain.com/ has neither?

GoogleGuy,

Sure would be nice to be able to get out of this 180 day hold. I appears to be killing my entire site in terms of search rankings.

5:59 pm on June 1, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 10, 2003
posts:929
votes: 13


This is weird. I just submitted a page which was 302 redirecting to me about 4 days ago. It went thru "pending" status to "complete", stayed there all of about 36 hours and now says "expired" already!
What's with THIS?! I sure hope I won't have to be doing it every few days with the new G algo! Others I did at the same time still say complete though, at least for the moment.
6:44 pm on June 1, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 15, 2005
posts:380
votes: 0


If those three are treated the same then why does my domain.com/index.html have PR and backlinks (no cache) whereas domain.com/ has neither?

It's perhaps removal from the index doesn't mean removing from links database used in PR calculations. I have told it before in other thread, but I'll repeat it:

Page removed from Google index with URL console can still have PR, and pass it to pages it links to!

You may find arguments against it, but it was the case with my page! First, it shown no PR. After a few months page.#*$!.yy/index.html shown PR in some PR checking tools. Then, after some time page.#*$!.yy started showing PR while /index.html didn't show it anymore. When the last toolbar update took place, this PR has raised, and when the page finally came back to index, it started to show this PR value in the toolbar.

Removal with URL console doesn't actually means removing it from the index, it's just it's not shown in SERPS, but can be still crawled, links can be followed, and it can have PR and be cached (even if this cache is not shown anywhere).

And these pages which I deliberately removed six months ago, returning 404 ever since then, and reappearing right now in SERPS with old snippets - are additional proof that removed pages are just not shown but forgotten.

8:16 pm on June 1, 2005 (gmt 0)

New User

10+ Year Member

joined:Aug 26, 2004
posts:9
votes: 0


Hi Wizard, Great thread. I have a question for you. Six months ago the removal pages didn't say anything about the six month temporary removal period, correct? Did it say 90 days back then?

Almost 6 months ago, I removed 2 sites that were under development and got crawled and indexed as URL only listings (I was bad and didn't have a robots.txt file, since there weren't any links to the sites - I suspect the toolbar phoning home, but that's different matter). I don't remember seeing any time limit. One of the sites went live in 2/05 and I watched googlebot come and crawl again and again. I kept wondering why I wasn't getting listed. Then I remembered the URL removal and when I went to check, I saw the 6 months and freaked a bit. I swear it wasn't there before, but I might have forgotten it if it just said 90 days.

My 6 months is up on 6/4. ;-)

I have learned how to get some traffic without Google though and that's a valuable lesson to learn.

martin

8:32 am on June 2, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 8, 2001
posts:2882
votes: 0


bd4rr, it requires brain surgery on a database to do that. The last time I said I'd try to help with it, it turned out that the person had submitted 450 (!) different requests all over random parts of his site. They'd made complicated Swiss cheese out of the site with the url removal tool. So I'm pretty wary to promise anything. :( If it's a single site with one removal, you can try writing to user support and asking them to pass the request on to GoogleGuy.
10:02 am on June 2, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 15, 2005
posts:380
votes: 0


Six months ago the removal pages didn't say anything about the six month temporary removal period, correct? Did it say 90 days back then?

Yes, when I removed it in November 2004, it had been written 90 days.

In February, after 90 days, I e-mailed Google twice describing the situation: first - that it's over 90 days and site is not back, second - that I removed www version and lost both www and no-www.

Google gave me two automated responses, as they usually do, but a few weeks later they changed 90 days to six months on URL Console page, and they changed status information so it started to show both www and no-www pending if you submit one of them to removal, so obviously they took some more care about these issues.

There is no doubt we would like better solutions from Google, including possibility to cancel removal of each removed URL in Console, but they must have their reasons to not implement it right now - in the beginning of this year Google fought hard with spam in the results, for example, and they have to follow their priorites.

I wish I had been able to reinclude my site earlier, but I dare to think my emails at least called engineers
attention that some things could be improved in URL Console eventually, now my site is back and I just keep trying to make other aware of problems with careless using of URL removal tool.

In my experience of web developer it would be easier to implement new features in URL Console than bother with individual reinclusion requests, but my experience has nothing to do with Google immense scale.

7:47 pm on June 2, 2005 (gmt 0)

New User

10+ Year Member

joined:May 31, 2004
posts:39
votes: 0


If it's a single site with one removal, you can try writing to user support and asking them to pass the request on to GoogleGuy.

Hi GoogleGuy,

Thanks a lot. Just the reply itself is very encouraging. For my case, it is a single site with one removal. (I have used the removal tool many times, but all others worked as I expected).

I did asked for help from this page:
[google.com...]
but only get an automated response.

I will try again as you instructed.

2:22 am on June 15, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:June 13, 2005
posts:479
votes: 0


Can I please confirm my understanding of this thread?

I've uploaded my robots.txt file to google's automatic url removal proceedure (is this what's referred to as Google Console?). All that dross which google should have but have never removed is now removed from my google searches.

Am I to understand correctly that they haven't been removed from their index at all but have only been removed from the SERPS? This is only temporary and all that dross will return to the SERPS in 6 months 6 days time?

Jeez. How do we ever get all this old dross out of their index. The pages don't even exist any more. Do we have to clutter-up our robots.txt and .htaccess file with this stuff forever and resubmit robots.txt to google every six months. That's ridiculous.

2:36 am on June 15, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:June 13, 2005
posts:479
votes: 0


One other question. All that dross that's now deleted from the SERPS. How did/does that affect any duplicate content filters google might be applying to my pages?
2:40 am on June 15, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:June 13, 2005
posts:479
votes: 0


One more last question. I didn't realise that the robots.txt which I submitted should only have included the files I wanted removed and not all the standard folders I don't want google so search, like /images/. Will Google now ignore more robots.txt file in 6 months 6 days time and index all those folders they shouldn't be searching?
2:44 am on June 15, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 31, 2005
posts:1108
votes: 0


Here is something people can try for such dross.
Get your web server to respond with a 410 (Gone) for any requests for that page.

If you don't want some contents spidered or a link to it in a search engine, make sure that is requires the user to login to get to the content, if it is publicly accessible, sooner or later there is the chance that it will come accross a link to it otherwise.

This 34 message thread spans 2 pages: 34