Reasons why Google might ignore a site?

[Mike521]

Hello all, I'm new to these forums

Just looking for information on why Google might decide to ignore a site for several months. I'll give some info --

Currently google only looks at the index page and goes NO further. index page has text links, image links, etc. It's PR is 5/10, and most internal pages have a PR of 4/10. However a saturation check comes up showing only the homepage listed in Google. Furthermore I've checked the logs, and googlebot has only ever requested robots.txt (there is none) and the homepage. There are no strange or robot meta tags, only keyword, description, and abstract, which are all probably ignored by google anyway.

I've emailed google and asked if the site was banned, and they replied that it was not banned, and to just wait for the googlebot to crawl the site.

Nevertheless, since roughly june of this year, the site has never been deeply crawled by google.

Other engines seem to crawl it ok, but not spectacular. Inktomi is the best so far, with 40 pages crawled. Alta Vista and Alltheweb only index the pages I submit, but go no further. Ask Jeeves (Teoma) seems to crawl all pages.

Any other info... the site is roughly half html and half cfm pages.. it has a site map in text, linked immediately in text from the index page..

anyway any information and suggestions would be greatly appreciated. This has been stumping me for a while now. Thanks in advance

[icebane]

I'm currently experiencing the same thing... however it has only been 3 weeks. Only the front page is indexed.
The site will definitely have a PR of 6 given the other sites I own that link to it.

[SlyOldDog]

You say internal pages are PR4 but then you say they are not listed in Google. Which is it to be?

Is your site dynamic? That can cause major headaches.

[Netizen]

ou say internal pages are PR4 but then you say they are not listed in Google. Which is it to be?

The toolbar estimates the PR of pages not in the index - typically PR of one less than the page.

As for the original problem - are the links of the front page static or dynamic? If they are dynamic do they have id=blah in the links, or do you use sessionids?

[Essex_boy]

I had this problem with my site, submitt ALL your pages to google for indexing that was the only way I found.

[SlyOldDog]

>>The toolbar estimates the PR of pages not in the index - typically PR of one less than the page

I haven't seen this since 2002 except for dynamic sites.

[superscript]

Essex_boy

Submit all the pages - that's a blo*dy good idea - certainly worth a try. I'm having this index-only crawl problem at the moment. Index is PR 7, rest of site PR6. It's been up for two years and previously deep crawled, but nothing for months. I was having non-www, www duplicate probs., so I searched for and corrected all the non-www links on the web I could find, and also set up a proper 301. What's happening is the pages listed as non-www are gradually being dropped, but without a proper crawl they're not being reinstated as www. I'm now down to only 12 pages in the Google index. It's been one hell of an end to the year :(

p.s. all static, nothing fancy, standard links, intuitive navigation.

p.p.s.

The toolbar estimates the PR of pages not in the index - typically PR of one less than the page

Concur - missing pages still have an extimated PR along these lines.

[superscript]

I've just manually submitted a small selection of 'pivotal' sub-pages to try to get the bot to remember there's more to my site than the index page - I'll get back with the results - if any.

[Mike521]

Thanks for all the suggestions, much appreciated :)

about the links, all links are static, just regular old html, nothing fancy at all

about the PR issue, yep I guess the internal pages are just an estimate, because they're not listed when I do a search for them in google, yet they all have a PR of 4 or so

perhaps I'll submit the site map and see if that gets me anywhere..

essex boy, how long was your site in that state for, before you decided to submit each page individually? what have your results been since then?

I had been under the impression that submitting a whole slew of pages under one domain could be viewed as spam?

[superscript]

I had been under the impression that submitting a whole slew of pages under one domain could be viewed as spam?

Yeah, this worried me a bit. But it's likely to be an automated process, and while the guidance says it's only necessary to submit the index (the spider will do the rest ;) - there's nothing in the guidance that actually prohibits the submission of sub-pages / multiple pages as far as I can see. Anyway, to be on the safe side I just submitted a selection of half a dozen select pages that are well linked to the rest of the site. Added: Don't forget its a manual submission tool provided by Google - they frown on automated submission software, but it's highly improbable you could be penalised for using something they provide.

[nileshkurhade]

Does the site have a sitemap.? Maybe this will also help.

[SlyOldDog]

I don't really get it. How can the pagerank estimation still be working?

Unless someone is using a different toolbar from me, this gives PR0.

[yahoo.com...]

I see PR0 when I type any unindexed page on any existing URL, where it has a special page for 404s or not.

What you are seeing must be something else.

[superscript]

Hi SlyOldDog, I just tried one of my pages which is definitely not in the index and it shows PR6. In the past I've noticed odd things about PR display, for example landing on one of my pages with something like /?src=fav appended would show zero PR, but navigating to that same page from within the site would suddenly show it up.

Does this page

[yahoo.com...]

actually exist? If not, surely this is an entirely different case.

[SlyOldDog]

No, that Yahoo page doesn't exist. That's the point. It has not been crawled yet but still shows PR0 - not an estimation based on PR10 from the home page.

I've noticed that if you go to a page with a real variable name which has PR, like:

www.blah.com/index.php?id=25

if that page has pagerank, you can type in an imaginary value, and Google will still give a PR score. eg.

www.blah.com/index.php?id=anynumber

But if you type:

www.blah.com/index.php/imaginarypage

you will get PR0

[superscript]

I thought we were discussing PR showing on pages that actually exist, within an indexed site, but that are not currently indexed in Google. It's of little surprise that a non-existent page has no PR.

Interesting though - if the PR of a page can be estimated from the general PR of a site, how does Google 'know' whether that page does actually exist? Unless there's some remanant of its prior existence in the DB somewhere.

[SlyOldDog]

In the past, Google would show a grey bar (unindexed) for unindexed or unranked sites. PR0 was stritcly for indexed pages.

[Netizen]

I just added a new page on my site and it shows zero PR. This is using toolbar 2.0.102-big/en (GGLD). Got to keep up :-)

[superscript]

It's been my understanding that the distinction between grey/white toolbars hasn't been meaningful for months.

[Essex_boy]

Ok, my site is HTML nothing fancy the site was set up earlier this year around March and only the index page was listed although it was updated by Google it didnt go any deeper.

I submitted individual pages via googles own submit page - never use a commercial submitter, I did this on the 8th of December and by the 15th they were listed. Breath takingly quick.

Google does NOT punish the multiple submission of a single URL so I see no reason why you would be punished for submiting several URLS from the same site.

I had no problem. All pages rank well athough im yet to receive any page rank for them.

[superscript]

Essex_Boy

Confirmed: Google states: "We do not penalize sites for 'over-submitting'"

Whilst at the same time hinting 'but don't bother with submitting' ;)

[Essex_boy]

well yes that was probably true pre MAY update since then Google hasnt been anything other than unpredictable. Has it?

Submit away and see what happens.

[div01]

I got a couple of sites in the same predicament. I guess I'll be hand submitting all the pages this afternoon.

[Essex_boy]

Yeah its great fun really! Put a CD on and submit away!

[superscript]

LOL! Pity you can't burn it to CD and send it to the Plex

[Mike521]

LOL

thanks again guys, glad that I'm not the only one experiencing this problem, and glad to see that google was relatively fast for you with indexing your hand submitted pages

my site does have a site map, so I'll submit that, and a handful of other important ones, and see if the spider makes up its mind to dig a little further this time. If not, then I'll be hand submitting every page in the site lol

slydog, I have no idea why the pages inside the site are showing up with a PR but not in the index, but nevertheless, that is what is happening. The idea about it being an estimate sounds good to me. The pages are definitely not indexed though, google confirmed this when I emailed them and asked them if they had a problem with the site.

[Mike521]

well, I tried submitting pages within the site and still they are being ignored by google. I sent them an email today to try to get to the bottom of this :/

[DJFlite]

Every page on my site gets indexed almost every week, but as of two weeks ago I stopped being listed on google completely. Even if I type in www.mydomain.net it doesnt come up. Whats the deal with that?