Googlebot actually preferring dynamic URLs now?

Or just my imagination?

[savvy1]

I really need to spend some more time analyzing the logs better, but, a very cursory inspection of the logs from the last week or so seems to show googlebot (Both the bots from 216.239.46/24 and 64.68.82/24) showing a preference to dynamic URL's on one site I'm looking at. The front page has some dynamic links and some static links, and the dynamics seem to be spidered first, more often, and deeper. Also, I've seen on several occasions the spiders grab the same dynamic URL more than once in the same crawl session (within less than an hour or so), which seems a bit odd.

GG stated that google had made some changes WRT dynamic URL's, but, it seems odd that they would be given an actual preference over static URL's. Again, this was a very cursory inspection and it could have just been a coincidence, that the crawler happened to pick one of the dynamic URL's to spider first, and, I'm just reading way too much into it. Just wondering if anyone has seen similar behaviour.

[thejenn]

Could it be that your static links have been picked up several times already from past updates and that Googlebot is looking for "fresh" content?

that might explain a cursory spidering of static pages and a frentic dig through the dynamic pages.

[Grumpus]

I have a 99% dynamic site. I, too, have seen it hit pages more than once in relatively short succession.

As I've mentioned elsewhere, I believe Google took a MAJOR leap in its dynamic crawling abilities last month. For the first time ever, more than just my homepage has a PR that isn't "Guessed" by the toolbar and identical to other pages with the same name.

i.e. In the past mypage.asp?ID=1 had the same guessed pagerank as mypage.asp?ID=120987. Now, each "ID" has it's own pagerank (so long as it's been in the index for several months - first month entries still seem guessed).

The thing that I'm noticing on the "multiple hits in short periods of time" pages is that they tend to be new to the google index or are pages being hit by the freshbot. I believe, though I can't be certain, that google is making an attempt to find algos to determine if pages actually have NEW information on them when it crawls or whether certain elements of the page are generated randomly and, therefore, shouldn't be counted as updated pages. (I have no idea if the algos are being build on a site by site, page by page, or web-wide level, but I believe it's a sound theory).

After the freshbot has looked at a specific page for a few days, it seems to hit it less frequently and if it's updated, it gets "freshed".

In essence, I believe those hits you're seeing are helping google hone in on its new ability to index dynamic pages and properly assign page rank.

G.

[amoore]

I believe Google took a MAJOR leap in its dynamic crawling abilities last month.

Either that or I finally made some of my pages with query strings attactive enough to crawl. Google hit a portion of a site of mine with a few hundred pages with query strings for the first time this month. It's good to see regardless of the reason. I just hope they get good treatment in the index now that they're presumably there.

[korkus2000]

IMO They have changed the way they spider dynamic pages. My site is about 50% dynamic and is really been hit hard by googlebot this month.

[allanp73]

One of my sites use Filemaker pro. Can the dynamic pages get spidered amd indexed? I have never seen any of these pages get listed on Google. It would be nice because 95% of more 10,000 page site is generated through dynamic pages. Would it make sense to create static pages for all my products in order to get more pages on Google?

[korkus2000]

What is the extension of your pages? Are their session variables in the query strings?

[allanp73]

The search page take the form of

actionlasso....&-op=bw&Active=Yes&-skipRecords=6&-maxRecords=6&-sortField=Product%20Name&-sortOrder=ascend&-search

Where the search criteria are displayed on a dynamically generated page.

[Grumpus]

You're not likely to get search results crawled for the simple fact that you need to type something into a box and hit a button to get those results. The googlebot is a very bad typer and can't be bothered with such things.

Now, if you create a link to a specific set of search results, that's another story...

You need to have some way for the googlebot to crawl the site starting from the front page and working its way through. Pretty much, it is only going to find things with links to them. No forms, no javascripts, no flash or java menus, any of that. It needs a real <a href> tag.

amoore - making better URL's for your dynamic pages is a good thing, but I can say that I changed nothing in the code for my main site last month. (Was working on the new "shop" side of the site in a sub-domain, so I didn't touch the main code).

G.