Reasons why spiders may not be visiting a site

[hannamyluv]

I was asked to do a bit of light consulting for another website. I don't have too much access to info on the site, and their web person is more of a marketing person so they aren't very search engine or technically savvey.

Here is the issue. They are just not getting much in the way of organic traffic. I took a look at their site and the G toolbar gives it a PR0, which is kind of odd since they have been around for 6 years and have dozens and dozens of real links that they garnered not for PR but for actual traffic (imagine that) from info sites that relate to the product they sell. Google shows none of these links as backlinks. I can't imagine it's a penalty, since they just decided last week to take a look at their organic traffic and have done no SEO in the past.

They have two domains for the same site, but I don't have definate info on whether the non-advertised site is a redirect or a parked domain. It apprears to be a redirect though.

I told them to check their robots.txt. I also asked them to find out if the secondary domain is redirected or parked, as those can both cause issues.

What other reasons might be causing search engines (not just google) to avoid their site?

[glengara]

"they have been around for 6 years... and have done no SEO in the past"

That sort of raised the eyebrow, have you looked at it through the Wayback Machine?

[hannamyluv]

It's really not all that odd. They are a catalog company. Most catalog companies are just now figuring out that the web can be a source of customers not a place for current customers to place an order. Many are just now hearing and understanding the phrase "Search Engine Optimization" (which is what happened here and why I was asked to talk to them)

[rogerd]

Weird script links is another possibility, i.e., spiders may not be able to navigate the site. For starters, spider the site yourself with Xenu or similar. Xenu will demonstrate the spiderability as well as disclose other weird behavior that may not be obvious, like internal redirects.

Do the backlinks that don't show up have PR of 4 or greater? Even if not, though, a PR0 on the home page is odd.

Did you compare the www and non-www versions of the site? I've seen a fair amount of weirdness caused by separate indexing in the last year or so. Check links and PR for both versions.

Check the server headers on the redirect and primary sites. It could be that Google thinks the redirect site is the "real" site instead of the one the client is trying to promote.

[King of Bling]

Another possibility... does the site validate W3C and CSS? Things as simple as an incorrect DOC-TYPE can cause headaches.