Fast spider smarter than Googlebot?

better handling of long dynamic URLs

[Finder]

After a recent discussion in the Google forum on how to better entice the Googlebot, I realized that my dynamic URLs were too long for Googlebot to handle. So, I changed them. The very next day, the Fast bot stops by and asks for all the old links that I just changed!

I removed the subsection part of my URLs:

index.html?section=foo&subsec=fee&specific=a+b+c+d

Becomes:

index.html?section=foo&specific=a+b+c+d

My site has never been fully indexed by the Googlebot. Now it's been extensively indexed by the Fast bot, but most of the pages are in the form of "Error: requested document not found."

I just can't win sometimes. :)

[xbase234]

I have no problems with Googlebot using 2-3 variables. Maybe it's because of the 4 variables?

[HenryUK]

I run a database-driven site with tens of thousands of individual records, with a churn of about 10% per month (ie a couple of thousand records drop out, and about the same number are created)

The site has dynamically created pages containing links to all those individual pages. Instead of using the query string these links are in the form www.mydomain.com/BlahXXXXX.htm where XXXXX is a numeric unique record identifier. These are handled by the error page to redirect to the appropriate record.

I have just had all these pages successfully indexed by Fast (they do all have genuine individual content!).

A handful of these pages have recently shown up in Google as temporary, dated items (freshbot; pages now dropped again). Now waiting for the google update to see if it works there too.

In a week or so I will be in a position to make a comparison!

[HenryUK]

Yes, worked on Google too.

:)

[Rumbas]

Good for you HenryUK. You are saying that both FAST and GG had no problem spidering all your pages?

So have the server come down from all the traffic? ;)