homepage Welcome to WebmasterWorld Guest from 54.205.254.108
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

This 95 message thread spans 4 pages: 95 ( [1] 2 3 4 > >     
Site disappears - always seems connected to Sitemap
BuckerBucker




msg:3239852
 7:59 pm on Feb 1, 2007 (gmt 0)

I found it!

After months of up and down with Google, I saw it with my own eyes!

Three hours ago Google downloaded my sitemap.

Three hours ago I went into oblivion, and disappeared.

Google Sitemap controls report my sitemap is OK with only one error for January 23, which I corrected the same day.

According to all of my notes, every major drop has resulted right after Google accessed my sitemap, and today, I saw it with my own eyes.

My most serious drop-off with Google, that lasted almost two months, came immediately after I had designated domain within the sitemap control panel.

Maybe this drop won't last as long, or maybe longer. But I am convinced that sitemap retrieval by Google is implementing the beginning of the fall.

I also know that I have a solid site, no spam and no black hat techniques. Google is losing out bigtime on authority sites, as the roller coaster continues, and good webmasters go away.

One other thing...my page ranks reported in my sitemap control panel have never recovered from the first major fall, and show that most of my site has not been ranked.
In fact, 98% of my site shows google page rankings, and has for over three years.

I still think something is broken at google.

[edited by: tedster at 8:10 pm (utc) on Feb. 1, 2007]
[edit reason] create separate thread [/edit]

 

caryl




msg:3239918
 8:31 pm on Feb 1, 2007 (gmt 0)

I still think something is broken at google.

I AGREE!

But I don't think it is 'directly related to sitemaps.

I DO BELIEVE they are having (whether they know it or not) PROBLEMS with spiders. Particularly the spiders that are used to populate their PRIMARY INDEX.

My 10 sites were ALSO hit sometime between 10:AM and 11:AM EST this morning! Google traffic has all but STOPPED to ALL of them.

These sites had been 'in oblivion' since last summer - but made a comeback on January 14th (started on Jan 8th).

Just previous to that 'comeback' ALL sites were heavily spidered on Jan. 6th. I record spider activity every day for all of the sites and they NEVER all get deep spider activity on the same day!.

YESTERDAY - ALL WERE HEAVILY spidered again - and now they are ALL 'back to oblivian'!

Caryl

BigDave




msg:3239959
 8:48 pm on Feb 1, 2007 (gmt 0)

According to your logic, I should never get any traffic to my sites with sitemaps, as google downloads them several times a day.

They might be related to the reason your site disappears, but I don't think it is a direct cause and effect. If downloading the sitemap leads to problems, then I would suspect that there is something about your site that Google tends to find when you update your sitemap.

AndyA




msg:3239974
 9:01 pm on Feb 1, 2007 (gmt 0)

The Google Webmaster Tools always seem to be off.

For instance, the search query results will show a term ranked at #1, or #4, or #20, but when you click on the link, it's at the bottom of the results. (950 Penalty).

Plus, it takes forever for Google to drop URLs when you disallow them through robots.txt. For instance, unknown to me, my forum was generating new URLs faster than the speed of light. In order to contain the damage, I disallowed /forum in robots.txt to just get those pages out of the index. It's been working, but after 60-90 days, there are still many listed, although they are supplemental.

Meanwhile, other pages on my site haven't been crawled in months. And they've had several updates since the last crawl.

Googlebot keeps going back to check and recheck and recheck and recheck pages that have been 301'd for years. Hey Google, I have updated pages and you're not getting them. How about cutting back on those 301's and hitting good URLs a bit more often, OK? Because I'll bet if a URL has been 301 for over a year, it's gonna stay that way. Check it twice a year, if you must, but it doesn't need to be hit weekly.

And, when a webmaster goes to the trouble to do a sitemap for you, that tells you which pages are on their site. If the sitemap lists 100,000 pages, and your index has 250,000, something's wrong. Same goes if you only show 500 for that domain. This would seem to be a really good way to discount scraper and directory sites. If they aren't listed on the sitemap, they AREN'T part of the domain, and shouldn't be counted as part of it, even if a 302 redirects to it.

I'm very frustrated with Google, as I have one site that was hit in November/December 2004 that has never recovered. And I see a lot of sites listed above mine for search terms that don't have 5% of the unique content my site has. And no one can find my pages, as they rank below all the garbage Google has listed.

As far as Webmaster Tools is concerned, my site is in the index, and then it isn't. Then it is again. Sometimes I have lots of pages on a site: search, then a minute later, I don't. I think something is broken, but does Google know about it? And are they working on it, or do they just not care anymore?

Serpent




msg:3240021
 9:35 pm on Feb 1, 2007 (gmt 0)

We just saw the exact same thing. 2 hours ago our sitemap was downloaded and immediately our site dropped out for most of our keywords. We have been cycling in and out since mid december.

We really don't do anything spammy and have been baffled as to why our site with tons of unique and relevant content keeps getting boned.

We deleted our site maps and our whole google webmaster account right away. I would encourage everyone to do the same.

In general Google seems unable to hit the broadside of a barn lately when determining who is the original source of unique content.

We have one article for example that got picked up by digg. Our articles is nowhere to be found if you search for it by title in google even if you include our domain name. You get first of all the reference in Digg, then Buzzle which picked up the article. You then get a number of MFA sites that have nothing but ads and a link to our page. Even though all of these pages link to us, and we were the source of the content, our article is nowhere. I guess it's because we were stupid enough to link to it in our google site map.

webslinger




msg:3240063
 10:09 pm on Feb 1, 2007 (gmt 0)

I've been so spooked by what I see Google doing lately with some great sites that I'm afraid to tie my site to any service provided by Google. I won't use services like sitemaps or adsense any longer.

mvander




msg:3240097
 10:33 pm on Feb 1, 2007 (gmt 0)

sometime between 10:AM and 11:AM EST this morning! Google traffic has all but STOPPED to ALL of them.

I am seeing the exact same thing. My site came on nicely in January after dropping a lot in December. Now this morning it just slowed down to an almost complete drop in traffic from Google. And yes, just late yesterday I had updated my sitemap after not having touched it for at least a month.

I am going to delete/remove my sitemaps right now.

walkman




msg:3240996
 6:25 pm on Feb 2, 2007 (gmt 0)

Serpent,
maybe you got linked from all these sites and goog reacted badly to it? Not saying that's it but I have pages that have more than average when it comes to links, and they don't rank at all.

The sitemap thing could be coincidental. Unless Google downloaded the pages why would they make the site disappear?

tedster




msg:3241015
 6:41 pm on Feb 2, 2007 (gmt 0)

I lean toward coincidence here - 2 hours just seems like too short a time for an effect to show up in the index.

gehrlekrona




msg:3241020
 6:45 pm on Feb 2, 2007 (gmt 0)

@caryl,

Interesting.... your site came back January 14th wne a lot of sites went to oblivion! Did your site by any chance get dumped on June 27th?

A lot of people have recorded problems after using sitemaps so I am pretty sure they have a problem but won't admit it.
I have since long stopped using sitemaps since it always seems to have problems and I don't care what they say when they tell us to use it since if they can't find our pages anyway, then we have problems with our link structure on the pages. I don't think Amazon.com or Microsoft.com submit any sitemaps, do they? And if they do, how big would their sitemaps be? 500 MB or more...
The ONLY thing I use the Webmaster Control panel for is to email Google and ask them to "re-iclude" my site when they have one of their majow srewups and the site disappears from the radar. Not sure if it helps but that is the only thing it is good for!

caryl




msg:3241067
 7:10 pm on Feb 2, 2007 (gmt 0)

@caryl,

Interesting.... your site came back January 14th wne a lot of sites went to oblivion! Did your site by any chance get dumped on June 27th?

gehrlekrona,

YES! My site WAS DUMPED ON JUNE 27th (that was the second time - it was originally hit on April 27th)

My sites have been in the 'pits of hell' since then - until traffic returned Jan 14th.

NOW - Feb 1st - Dumped again :(

Caryl

PS - I have never used sitemaps BUT - ALL of my sites (that were hit) were heavily spidered on Jan 31st!

Bewenched




msg:3241195
 8:53 pm on Feb 2, 2007 (gmt 0)

Yup .. same here ... check our rankings and the site: results during the day on the 31st and all was fine .. not great... but fine. Today traffic has been pathetic so I checked sitemaps and all the sitemaps were retrieved the evening of the 31st and now are almost completely supplimental... what the *&#$ dang it!

I am so freaking close to deleting the sitemaps it's not even funny.... we never had problems with our site/traffic etc until we decided to give google full access to our site... since then it has been non stop.

BigDave




msg:3241197
 8:54 pm on Feb 2, 2007 (gmt 0)

While there might be problems for some sites with sitemaps, I suspect that it is an convenient excuse for most people. Much easier than looking for what the real problem might be.

A lot of people have recorded problems after using sitemaps so I am pretty sure they have a problem but won't admit it.

When I lived in California, the nearby general aviation airport. When commercial jet service to LAX was started in 1986 with the very quiet BAe jets, noise complaints shot through the roof. Everyone thought they were hearing the commercial jets. The more people complained, the more other people started hearing the jets. But what they were really hearing were the private Learjets that had been flying out of the airport for years.

You may have a set of "facts" that lead you to conclude that sitemaps are your problem. You may be able to convince other people that it is the problem. You can easily fill threads with people will also become convinced that that sitemaps knoced their sites from the SERPs. But What happened to all the sites that disappeared from the SERPs before sitemaps? Or the sites that still disappear, but they never implemented sitemaps?

Instead of considering sitemaps to be the first possibility, you should consider it to be the last. Unless you just like to play the victim, then go ahead and grasp the easy answer.

Like I said, sitemaps might have some problems with some sites. But it works just fine on a lot more. I would be willing to bet that the vast majority of claims against sitemaps is simple coincidence.

BigDave




msg:3241253
 9:29 pm on Feb 2, 2007 (gmt 0)

I checked sitemaps and all the sitemaps were retrieved the evening of the 31st and now are almost completely supplimental... what the *&#$ dang it!

How often are your sitemaps downloaded. I just checked the logs for my newest site (1 month) and they list the sitemap downloaded Feb 1, Jan 30, 29, 28, 27, 26, 25, 24, 23. I just checked a site that is several years old and rarely updated, and the sitemap has been downloaded every day this past week.

Do you see the possible problem with blaming sitemaps when something has gone wrong, just because Google downloaded it within the last 24 hours?

netmeg




msg:3241259
 9:43 pm on Feb 2, 2007 (gmt 0)

I have said it before - I have over 60 sites (close to 70 now) in Sitemaps (Webmaster Central) and I have never ONCE seen a problem I could by any stretch attribute to that site's being there. I certainly have had (and still have) sites with issues, but the timing and the variety of issues don't point to anything having to do with Sitemaps. I just don't see it, and it's been months if not years (when did GWC start up, anyway? I don't even remember but I started adding sites as soon as it was open to me)

BuckerBucker




msg:3241271
 9:55 pm on Feb 2, 2007 (gmt 0)

Would the webmasters that have noticed significant coincidence to sitemaps and the disappearance of their sites compare notes with me just in case...

I did two major changes in June 2006 on my sitemaps.

The first was changing the google urlset to the all inclusive url set that yahoo and msn and google agreed on.

The second major change was that I indicated my domain as www to be my primary address, by clicking on that choice in my control panel.

Immediately after these two changes, my site tanked.

My sitemap is "ok" and no major problems.

I had a great half month in January when everything returned as normal from last year before the drop.

Then yesterday, the sitemap was downloaded, and I immediately disappeared off the radar of google.

I have over 1800 pages indexed in Google, and almost all of them have a pagerank.

My control panel states otherwise, and lists most of them as being unrated.

I have good solid links, and use absolutely no blackhat or spam techniques that I am aware of.

Does anyone have similar experience in a major drop connected to sitemaps?

gehrlekrona




msg:3241275
 10:00 pm on Feb 2, 2007 (gmt 0)

I don't think that downloading a sitemap one will destroy your site the next day! But I DO think that using sitemaps sometimes causes problems and I have seen it myself, or at least I THINK I have seen it myself :)

Like I said before, I am not using sitemaps anymore since I am thinking that Google should be able to crawl your site anyway. I don't see the advantage of having a sitemap.
If you MANUALLY add pages to your sitemap, then you might add a page that you don't have a link to on your pages, and if you don't have a link to it, then Google won't find it. IF you have links to all your pages, what reason would there be for a sitemap?
The program I have used to create sitemaps all crawls your web site and create the sitemap. If that software can do it, why can't Google?
If ANYONE can give me a good reason and some real results they have got from the sitemaps then I might consider changing my mind, but until then, no sitemaps for me.

BigDave




msg:3241278
 10:09 pm on Feb 2, 2007 (gmt 0)

Then yesterday, the sitemap was downloaded, and I immediately disappeared off the radar of google.

How often was your sitemap downloaded during that month and a half and it didn't tank? My guess is right around 45 times.

walkman




msg:3241286
 10:18 pm on Feb 2, 2007 (gmt 0)

>> Then yesterday, the sitemap was downloaded, and I immediately disappeared off the radar of google.

Let's assume that your pages are pure spam, I mean there's no doubt about them in G's eyes, and as soon as Googlebot sees them, it stops everything else to make your site disappear. How did they know about your pages simply by downloading the sitemap?

Let's look at this more rationally: what color was the shirt you were wearing the day the site disappeared?

gehrlekrona




msg:3241300
 10:36 pm on Feb 2, 2007 (gmt 0)

Here's something to read about Sitemaps. Has a part in there from GoogleGuy:
[netconcepts.com...]

Oh, and I don't think it has something to do with the shirt you wore on the day of the disappeareance. It's more like "Did you step on the man-hole lid a month ago?" :)

BuckerBucker




msg:3241339
 11:45 pm on Feb 2, 2007 (gmt 0)

That's it! The shirt is history! Didn't fit me well anyway. :)

In all seriousness...it fills like that my shirt color is one of the google algorithms, which is really a shame. Good Webmasters shouldn't have to rely on good luck to succeed.

You are right...many times my sitemaps were accessed in the middle of January and I did not take a fall.

But at the end of January, right after a sitemap was accessed, my site disappeared.

Can it be one errant spider, with one wrong algorithm that is capturing a sitemap and spitting out the site? Can it be one data center that is screwed up enough, that the sitemap is triggering an event?

Most importantly, is there anything that I can do as a webmaster to matter to google? I hate being riff raff that falls to their algorithms that is targeting the bad guys.

Their aggressive changes throw away good sites, and peoples livelihoods are being ruined. Just because in the big scheme of things through google eyes, there is no evidence of problems, does not make it real.

It's a pity that I have to search for that needle in the haystack, and shameful that I have to change the color of my shirts, to give me a little confidence in the system. In the end, I'm afraid I'll be going shirtless, just like the good webmasters that lost their shirts during the Christmas crash of 2006.

[edited by: BuckerBucker at 12:20 am (utc) on Feb. 3, 2007]

andrewshim




msg:3241371
 12:21 am on Feb 3, 2007 (gmt 0)

I don't have that problem with Google sitemaps. In fact, with every update of my sitemap, ALL my pages (approx 500) are indexed. I try not to fiddle around with what's not broken, so I only update my sitemap when reeeeeally necessary.

I DO have EXACTLY that SAME problem with Yahoo's sitemap though, but that's not relevant here...

BigDave




msg:3241421
 1:26 am on Feb 3, 2007 (gmt 0)

Can it be one errant spider, with one wrong algorithm that is capturing a sitemap and spitting out the site? Can it be one data center that is screwed up enough, that the sitemap is triggering an event?

Can it be that it has nothing-what-so-ever to do with sitemaps, but you can't get that idea out of your head?

walkman




msg:3241523
 4:06 am on Feb 3, 2007 (gmt 0)

"Correlation and Causation
We must be very careful in interpreting correlation coefficients. Just because two variables are highly correlated does not mean that one causes the other. In statistical terms, we say that correlation does not imply causation. There are many good examples of correlation which are nonsensical when interpreted in terms of causation.

Ice cream sales and the number of shark attacks on swimmers are correlated.
Skirt lengths and stock prices are highly correlated (as stock prices go up, skirt lengths get shorter)..."
[stat.tamu.edu...]

webslinger




msg:3241524
 4:08 am on Feb 3, 2007 (gmt 0)

"Can it be that it has nothing-what-so-ever to do with sitemaps, but you can't get that idea out of your head?"

If you look at Google SERPS and you see so many garbage sites without sitemaps left in them and great sites with sitemaps are the ones getting booted. And when Google downloads the XML your site disappears from it's normal listings you kind of notice it.

If someone is seen leaving the scene of a crime right after the crime has taken place you better believe the police are gonna be curious.

I don't see how you could rationalize mere coincidence with that.

BigDave




msg:3241538
 4:37 am on Feb 3, 2007 (gmt 0)

If you look at Google SERPS and you see so many garbage sites without sitemaps left in them and great sites with sitemaps are the ones getting booted.

How, pray tell, can you tell if a site is using sitemaps? Do you have access to Google's database of filenames?

Oh well, I give up. Your wonderful sites with sitemaps were booted while my lousy sites are continuing to get traffic increases without them. Except that my sites have sitemaps.

Believe what you want. Blame Google if you want. It will save you the work of figuring out what the real problem is.

kevsh




msg:3241560
 5:57 am on Feb 3, 2007 (gmt 0)

I'm not going to jump in and say Sitemaps is my problem, I agree it's a stretch unless you know specifically (which none of us do).

In my case, however, I certainly believe it may be aggravating an existing problem.

Due to a limitation in the backend software, I had many products listed in multiple categories. When I originally built my Sitemap, some of the products had duplicate URLs, one for each category. I cleaned most of this up and used only 1 entry per product - but still had the duplicates on the site. Over time products were added, dropped, even moved.

Up until about a month ago most were indexed. Then, very quickly, half of my pages were dropped out of the index and 95% of the rest are in supplemental hell.

Doing a site search these days using text strings, I see a pattern. While most products have one URL listed, a few show one listing for each category - sometimes one is in the supplementals, sometimes neither.

But almost without fail (95%+), the URL listed in the index is the one NOT listed in my sitemap. In the case of 2 listings, the one in the main index is also the one NOT in the sitemap. What the...

Google should have at least in a reasonable % of situations, picked up the one listed in Sitemaps while crawling. Each listing is as easy to find as the other via the site's links.

So, in the case of products in 2+ categories, Google seemingly virtually ignored every product URL in Sitemaps and grabbed the non-listed one. How or why I simply have no idea...

(As an aside, and to answer proactively some of the obvious questions that may come my way, I am slowly but surely deleting secondary category listings for all products, redirecting, and everything else possible to avoid duplicate content penalties which is likely the partner in crime in my scenario. It's a slow process for many reasons but as I can see probably vital.)

activeco




msg:3241686
 12:27 pm on Feb 3, 2007 (gmt 0)

The second major change was that I indicated my domain as www to be my primary address, by clicking on that choice in my control panel.
...
Then yesterday, the sitemap was downloaded, and I immediately disappeared off the radar of google.

This could be indeed a problem to some sites.
Be sure that your chosen primary root is the one Google already prefers in their results and that it is properly set on your server.
Be sure that your sitemap is made according to this above. Canonical issues are very easy to screw the things up.

glengara




msg:3241687
 12:30 pm on Feb 3, 2007 (gmt 0)

I'd share Big Ds suspicions of there being no direct connection, my own hunch is you need to be a lighter shade of grey to make use of Webmasters Tools, particularly in the linkage area.

Serpent




msg:3243205
 4:36 pm on Feb 5, 2007 (gmt 0)

It's all fine and good to dismiss the site map thing as coincidental, but within 48 hours of getting rid of our sitemap, we were back in again in similar positions to where we were before the drop. In the previous drops over the last two months, it was always at least 10 days before there was any change.

If we drop again in a short period of time, I'll agree it's just a coincidence, but until then I'm staying away from google site maps.

This 95 message thread spans 4 pages: 95 ( [1] 2 3 4 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved