Forum Moderators: open
Normally I would not be concerned about this, but it has been going on since before December now. The PR has jumped from grey, to white, to PR3 and now PR4, so there are no problems with the links in.
I have today put in a deep link to see if it makes a difference, and if that page gets picked up. Anyone a suggestion please?
I still largely think it is Googlebot normal behaviour, but normal behaviour for me would mean 200 pages spidered by now at least!
If the URLs are quite complex (eg. /index.pl?1=foo&2=bar&3=yin&4=yang ) then Google isn't likely to crawl so deep. If the URLs have a CGI parameter called id then I don't think they'll be followed.
After that, maybe buy some links on every page of a big PR7/8 for a month or so for $1-200 - if that doesn't work, you've a server/code problem.
Just had 8k+ pages of a 10 day old site crawled in last 24 hrs. Gbots been really busy in last few weeks so you should be getting some joy...
Could that be an issue do you think? My thought was that if it is showing PR on the index page, then it should be OK. Again, if googlebot is seeing two pages, then should it not continue through the other.
There are more than 20.000 sites indexed by Google. I'd say PR is nice to have and helps indexing sites, but it's not necessary.
Nowadays, people are seeing pages behave as if links are counted much sooner than the backlink/PR updates.
Plus, if you read the original google whitepaper -- that is basically what it says. That the probability a bot is going to follow a link on a page is proportional to the page's PR.
/Wayne
Just made another check and it has 3 backlinks from guestbooks, but PR is definetly 0. (white bar)
On a monthly basis, Google went from homepage only, to ~8K pages, to ~30K pages, to ~50K pages, to ~100K pages, to ~250K pages, but has been hovering around ~300 - 330K pages in the Google index for the last two months. This last month Google's spidering of the site has really slowed down and the total pages it's going to grab appears to have leveled off.
One of the things I did back when it was at about 30K pages in Google was to create a set of 12 site map pages linked from the bottom of the home page and then directly to pages in the "middle" of the site structure so that pages at the bottom would be closer link-distance from the home page. That seems to have helped.
Anyone else been in this situation? I'd really like to break out of the plateau with this site, but all I can think to do is set up an additional group of site-mappish pages after researching what sections of the site are the ones mostly not in Google.
For new sites, I've been trying to set them up and structure them so that their natural largest size will be no more than 150-200K pages, just to avoid this sort of problem in the future, but I'd still like to figure out how to get the rest of this site indexed completely.
Nowadays, people are seeing pages behave as if links are counted much sooner than the backlink/PR updates.
Just made another check and it has 3 backlinks from guestbooks, but PR is definetly 0. (white bar)
Anyone else been in this situation? I'd really like to break out of the plateau with this site
Global Wayne. Sorry cannot run with that idea.
Not seen any perceptable difference in any of the examples I have been involved with.
Interesting idea about the site maps. I have the /site_map.php pages at the bottom of the page too, as the menu is all in a menu.js file, so I know this will not be spidered. Thing is Googlebot has not been near them. Still left with either:
Dozy bot.
Penalised URL (anyone seen this?)
server problem ( 1 other page picked up so can it be?)
None seem likely. What am I missing?
Well Thomas, that comment certainly defies conventional thinking.
If you can get 10k pages indexed from a couple of guestbooks giving PR<1 then something interesting is happening.
Perhaps this is the reason for your problems?
Your Gbot's behaviour sounds like what I had on my site.
Before doing anything radical there, I'd stongly recommend getting some very high PR links for 1-2 months.
Then you'll know for definite if its a PR issue or not.
J
I just changed all links to point to the site I wanted crawled and after what seemed like forever, it did finally and the pages ranked well from last May until last Sunday!
:(
You could always get a brand new domain & start from scratch?
Ho Ho Ho... the chap who owns it paid money for it. Not sure he would be delighted. He also bought his own server, so is stuck with an odd ball one, it was cheaper :(.
By spiral structure, do you mean sort-of offset partially meshed? Could you give me (or point me to) a good description of a spiral structure?
I'm not going to go back and mess that much with the above site's architecture, but I have a site still in the planning stages that has a really flat structure I could turn into a spiral if I could come up with a good rule of thumb for creating the spiral. I'm not adverse to trying out a new structure for a site to see what the results are. :)
Googlbot still looking for the old IP Address [webmasterworld.com]
a snippet from the thread:
Someone left me sticky mail suggesting that the original server be resurrected (with its original DNS). I took their advice, and observed the traffic. Google begin spidering it like crazy within minutes, even though the server had been gone for a year. It would seem that Google thought that the old and new servers were the same; the presence of the new server kept Google coming back, but it would continue to try to hit the old, and failing that, would never add pages to its index. ie. it would look to see if the new site was there by hitting the root index page, and then try to access pages on the old server.
