Forum Moderators: open
With the new crawling behaviours of googlebot, there seems to now be the opportunity, with a high PR, to get constant big crawls everyday scooping up all your nice new files. For my PR4 site, the crawling is quite good, google bot seems quite active, but still some trouble in getting down deep into the site. (i've got no worries about link structure).
With a higher PR I would assume and other stories have suggested that googlebot is more thorough. I also have a PR3 site, with fewer external links to the homepage, (neither site has external links to other pages). Googlebot's behaviour with this pr3 site is markedly different. There are usually a small number of hits a day and in the last three weeks there has been one day when there was a deep crawl where many pages were indexed. It seems to me that if I want the website to get hit more then i need higher PR (or more links to my homepage if you like).
Is this a similar experience to others?
Certainly from my last attempts two months from site up too a complete listing seems to be a reality, which is quicker than what it used to be from what i can remember pre-esmarelda. That being said, getting the front page and its sub pages in - can be quick, get the necessary links, related topic and that will happen the deeper pages will come. In saying that - the site i have in mind at the moment had a full index for about 1 day in G after a week of going online - as you can imagine i thought i had found a new means of getting site's fully index and on the SERPs (hooray) - but the deep pages dropped back out and came back after a good 6-7 weeks and now they are permenant.
The moral of this tale is that the crawls / update cycle that G is doing is varying in depth, how its going deep one day then listing and dropping the next is anyone's guess.
You can get pages in extremely quickly if you link to them from frequently updated Web pages (i.e. if you have a weblog and link to a new page of yours). The listings you receive, however, are less than great. And even though Google will then crawl that new page every day, if it's a new site, it probably won't bother to crawl the rest of the site for a little while.
That said, I'm in charge of SEO on a new site. Ten days ago I gave it its first backlink just to get the ball rolling. The next day, the page that was directly linked to was in Google, and had a PR0 (white bar). No other pages from the site turned up, so I got a direct link to an internal page. That then turned up on Google, but nothing else.
About a week after starting, 'most' of the pages of the site appeared on Google, all with PR0, although no pages more than 1 deep entered the index.
It's ten days later, and nothing has changed. The pages have extremely poor positions in Google, but as Caine noted, it takes time for pages to be indexed 'properly' and to get the PR they deserve. In the previous 10 days, I've managed to obtain scores of good (legit) links from pages with decent PR, so I'm hoping that when Google finally does its 'proper update', I'll start to see some #1 positions there ;-)
That said, I once had a new site get a PR5 and have backlinks indexed after about four weeks, so your mileage may vary. That site, however, was a weblog which became linked quickly by third parties.
But then you're a blawg.
SN
I've also got a thread over in another section about
If-Modified-Since: [webmasterworld.com...]
Since you're facing the same challenges I've posted this with my Q's, specifically regarding Google.
Can anyone with some experience in this arena tell me definitively:
Googlebot does (?) can (?) will always (?) simply send an
"If-Modified-Since" header to my site's server, about a specific page.
If the content has not changed since the Last-Modified date/time that Googlbot recorded/cached for the page, the server sends a 304 header-only back to Googlebot, and Googlebot simply tells the accordant document "barrel" to keep the cached version?
If the content _has changed, the server instead sends the "new" page, with a 200 header?
I've probably over simplified the above synopsis, but do I have the general process correct?
I ask because I'm noticing Google's *still not finding my new content...
Log files tell me Googlebot calls for our default.htm
about once a day, but nothing else, regardless of new content added (in the root directory, for "ease of discovery"). So, in light of what I've read in these forums, I updated this page and now have a Last Modified of 10/1/03. I also created links to the new files on this page.
I'm wondering if I now the server shows a "new" Last-Modified date, will this prompt Googlebot (if I have the process right) to actually re-crawl the page, and find the new files via the links i put there?
