Forum Moderators: Robert Charlton & goodroi
I was wondering if anyone could pass any advice. Here's the situation.
We use a .net portal for our main site. The portal manages logins o the site. The site is constantly changing as we have a portal and blogs etc. We pass an xml sitemap feed to google webmaster tools so that all pages are indexed. We have had good results for some time now in the google serps but pay for adwords exposure too to to ensure we get maximum relevant traffic.
Recently we began new development on both the integration of a new, second website which we're building and also SSO (single sign on) integration with the Google API so that when our users log in, they are also automatically logged in to our second site AND to google apps (we have a ISP google apps account and offer users branded gmail). After much testing we found that the only we we could suggesfully do this was by doing a quick redirect to all three sites briefly while logging in - this was to collect all 3 cookies as apparently you can't have a cookie which works on more than 1 domain.
Anyway, we wrote a http handler to ensure that only well known browsers saw this redirect and the bots didn't (as they don't need to as they never log in). Anyway, it looks like google have seen it somehoe (they must use crawlers which look like IE7 or similar to check for black hat I guess) and the majority of our natural traffic has gone. The annoyance is we didn't notice as it happened exactly on the start of the holidays here so we put it down to that.
A quick study of the crawl stats and google analytics however shows that it's definately due to google seeing the redirects and therefore dropping the pages from the index.
Yesterday, just over a month of this happening, we saw the bug and removed the http handler and the redirects. We've forced a re-submit of the xml feed through webmaster tools and it's registered the xml without issues (previously the redirects were highlighted here but we just didn't see them!).
Anyway, my questions is...
1) has anyone else been dumb enough to make such a mistake and did you get your serps results and traffic back?
2) although we have all of our now good again URL's submitted through sitemap, the number of these indexed or even crawled is still zero (0)! How long do you think it will take before we're crawled and then back in the results pages?
The last cache of our site is from 7 days ago. I'm not sure how but some pages still show in the top 3 positions in the serps for major (but highly competitive keywords).
For info, our main page is a PR5 and we're relatively well linked in to for our size. Have we been partically penalised or does google think we're just stupid temporarily and things should pop back.
Google registered the errors due to a '303 temporary redirect'.
Any feedback, advice or help would be much appreciated.
Lyndon (Trueways)
[edited by: Receptional_Andy at 6:40 pm (utc) on Aug. 13, 2008]
it looks like google have seen it somehoe (they must use crawlers which look like IE7 or similar to check for black hat I guess)
While they do perform such checks, I don't believe that would effect the crawl stats you see within webmaster tools - it sounds like your user-agent detection may not have been working properly. It's always worth checking such implementations by spoofing a googlebot user-agent, in addition to checking out web server log files to see what's actually happening when spiders visit.
In terms of what happens now, Google is usually fairly lenient when it comes to HTTP errors, since these can occur for such a wide variety of innocent reasons. I'm no fan of sitemaps, and I don't believe the use of one will significantly speed up the time for Google to recrawl the site and reinstate your results, which is likely what needs to happen for your pages to return.
The other consideration is that if you believe Google may have mis-interpreted your activity and penalised your site for contravening guidelines, you should submit a reconsideration request via Google Webmaster Tools, explaining the problem as carefully as you can and what you've done to fix it.
