can ISAPI rewrite incur PR penalty?

[justme]

Hi. Our website took a big PR hit yesterday. It went from PR6 to PR4. The website has been a PR6 for several years. I checked other sites similar to ours that I have been comparing our site to over the years and did not see their PR drop. We sell links on our main page, but always use the "rel=no follow" tag. The only change we made about a month ago is install ISAPI_rewrite so that IIS always includes "www." to our domain name in the URL.

Could the installation of the ISAPI_rewrite module incur a PR penalty? I've some experience with SEO/SERP but definately not a master. If this is not the cause of the penalty, how do I go about finding if a penalty was applied on out site?

Thanks in advace...

[phranque]

welcome to WebmasterWorld [webmasterworld.com], justme!
have you checked GWT to see if there are reported problems there?

[justme]

Thanks for the welcome and reply.

I did review the GWT diagnostics for our site. I saw no malware notice but a number of crawl errors.--e.g. 40x and 200 errors. I do not believe these would explain our drop in PR.

I did not see anything that would indicate a PR penalty. However, I am not a frequent user of GWT so I am not an expert user by any means. What should I focus on if I were looking for a PR penalty?

How would one go about to determine if a PR penalty has been applied by Google?

[phranque]

which domain had the PR6 before you implemented the redirects?

[justme]

The original domain was example.com and we implemented the ISAPI_rewrite module to
rewrite the domain with "www." so that www.example.com is always displayed on any
browser. This was done about a month ago.

Is the ISAPI_rewrite considered a redirect?

[edited by: phranque at 4:14 am (utc) on Jul 11, 2010]
[edit reason] exemplified domains [/edit]

[pageoneresults]

Could the installation of the ISAPI_rewrite module incur a PR penalty?

By itself, no. Installed and configured incorrectly, yes.

Unfortunately we'll never find out as someone who understands ISAPI_Rewrite will need to fully investigate your rewrite rules to make sure you've done everything properly. I've seen many folks screw up the base configuration and tank their sites. You may have introduced a redirect chain somewhere in the process. You may not be returning the proper server responses. All sorts of things can go wrong once you install and configure.

You also need to keep in mind that many of the default server responses are now handled by the ISAPI module. If you did not configure your rules to take into account all of that, you've got challenges.

A quick check and I'd say you have some redirect issues. You should check your server responses using Googlebot. I'm seeing a Content Location Header that doesn't jive with the destination URI. Don't ask me the details on that, your programmer should fully understand what is happening.

Your 404 handler is also jacked. That's probably where a bulk of your challenges are.

I did review the GWT diagnostics for our site. I saw no malware notice but a number of crawl errors.--e.g. 40x and 200 errors. I do not believe these would explain our drop in PR.

Want to bet? Due to your 404 mishandling, you've created a PR black hole. Any non-www 404 on your site is going through your 404 handler which is issuing a 301 which is returning a destination URI of the original request with a 200 status. So, your 404 pages (non-www) are actually being permanently redirected to themselves - that young man, is a PR black hole.

You also have canonical issues, lot's of them. You're using cased URIs and I can browse to both the upper and lower case versions. If I were a savvy lowlife competitor, I'd index your entire site, then create some really nasty URI paths and push those into the indices for crawling. Within a month or two, your site will take a major dive.

Added: You also have a META Refresh of 10 on your 404 page. Wow, talk about a chain of redirects. I'd say you've totally screwed the indexing for now. You've got some work cut out for you and a few months of recalculation before things normalize.

In GWT, is Google reporting Soft 404s?

Added: Your regular 404 page returns a 200. I'm sitting here hacking your URIs and as I remove each character from the string, every single destination is returning a 200. That is what you call a PR black hole.

Your programmer should fully understand what is happening.

I'd be drafting up a PINK SLIP for that particular person right about now. What a mess they've gotten you into and I AM NOT exaggerating. It's going to take you a bit of time to clean and recover.

How would one go about to determine if a PR penalty has been applied by Google?

Usually a quick glance at the little green bar is a good indicator. A drop from PR6 to PR4 is definitely a clue. ;)

[justme]

Thanks for the detailed insight, pageoneresults. I've been spending the last week reviewing our error handler. Based on your comments, the company has given me an evening job in addition to my current day job. ;)

In GWT, is Google reporting Soft 404s?

There were no soft errors reported in GWT.

You also have canonical issues, lot's of them. You're using cased URIs and I can browse to both the upper and lower case versions. If I were a savvy lowlife competitor, I'd index your entire site, then create some really nasty URI paths and push those into the indices for crawling. Within a month or two, your site will take a major dive.

I'm not sure I understand your statement above. Can you give me an example of how a competitor can create a nasty URI and push them to be indexed for crawling that would penalize our PR?

This caught my attention because one of the other items I'm looking into is the report from our DNS host that we are close to exceeding our DNS query quota for the month. This has been going on for each month this year and most of last year. However, our traffic from our analytics for the same period does not justify the excessive DNS queries reported for each month. I'm wondering if the indexing you describe above could be the cause for the large DNS queries reported each month.

Can you point to literature or web pages that explain in detail the workings behind your comment above? I need to get a better understanding of this and if it is the cause for the excessive DNS queries reported. How do I go about tracing the queries to determine if we are being victimized by this nasty URI/push/index for crawling PR penalty.

[pageoneresults]

I'm not sure I understand your statement above. Can you give me an example of how a competitor can create a nasty URI and push them to be indexed for crawling that would penalize our PR?

I won't lay out the framework for disrupting a website's crawling patterns. Whenever you have the same content that can be browsed to using multiple URIs e.g. a cased version vs a non cased version, there is potential for disruption in crawling. /Sub/FileName and /sub/filename are two different addresses. If you're serving the same content under both, there COULD be challenges.

This caught my attention because one of the other items I'm looking into is the report from our DNS host that we are close to exceeding our DNS query quota for the month.

I wouldn't be too sure that the two are related. We're going outside my scope of experience now in discussing DNS query quota. I know just enough to get me into trouble. I too have received notifications on this over the years from my server admins and it has always been traced back to low TTLs which are easily corrected.

This has been going on for each month this year and most of last year. However, our traffic from our analytics for the same period does not justify the excessive DNS queries reported for each month. I'm wondering if the indexing you describe above could be the cause for the large DNS queries reported each month.

Due to the large number of issues that could influence the DNS Query Quota, I wouldn't be so quick to tie the two together. I'd suggest that you get someone in there to review what you have set up and determine the root of the issue(s). There are just way too many "what ifs" involved here and this topic could run for weeks guessing.

I need to get a better understanding of this and if it is the cause for the excessive DNS queries reported. How do I go about tracing the queries to determine if we are being victimized by this nasty URI/push/index for crawling PR penalty.

Seek assistance from someone who can review your server setup to make sure all is in order. I'd be concerned about the excessive DNS query thing, especially after 18 months of warnings.

[justme]

I too have received notifications on this over the years from my server admins and it has always been traced back to low TTLs which are easily corrected.

This is the first thing I checked for. Unless an 1800s TTL is now considered low, I don't believe this is the cause in our case.

I won't lay out the framework for disrupting a website's crawling patterns.

Can you point me to website resources where I can learn more about this? Or keywords I can use to search for articles and websites that could help me better understand this?


Thanks for your time and the guidance. As you stated, I'm gonna have to roll up my sleeves now and start grinding on our error handling methods.

[pageoneresults]

Unless an 1800s TTL is now considered low.

I'd say that was very low and would cause an excessive number of DNS queries. It is all relative to the DNS record type.

[phranque]

it also depends on what the TTL value is timing (refresh/retry/expire/etc.)
the recommended TTL values for DNS SOA records are specified in IETF's RFC 1912 [apps.ietf.org].

[justme]

the recommended TTL values for DNS SOA records are specified in IETF's RFC 1912 [apps.ietf.org].

Thanks for this. The information in the RFC is definitely useful--especially since the
1800s TTL I stated above is the default and recommended setting from our DNS hosting service.
It is the time-to-live for the resource records.