Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Fast spider smarter than Googlebot?

better handling of long dynamic URLs

1:44 am on Nov 13, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 18, 2002
votes: 0

After a recent discussion in the Google forum on how to better entice the Googlebot, I realized that my dynamic URLs were too long for Googlebot to handle. So, I changed them. The very next day, the Fast bot stops by and asks for all the old links that I just changed!

I removed the subsection part of my URLs:




My site has never been fully indexed by the Googlebot. Now it's been extensively indexed by the Fast bot, but most of the pages are in the form of "Error: requested document not found."

I just can't win sometimes. :)

12:02 am on Nov 19, 2002 (gmt 0)

Full Member

10+ Year Member

joined:Aug 26, 2002
votes: 0

I have no problems with Googlebot using 2-3 variables. Maybe it's because of the 4 variables?
3:52 pm on Nov 27, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 23, 2002
votes: 0

I run a database-driven site with tens of thousands of individual records, with a churn of about 10% per month (ie a couple of thousand records drop out, and about the same number are created)

The site has dynamically created pages containing links to all those individual pages. Instead of using the query string these links are in the form www.mydomain.com/BlahXXXXX.htm where XXXXX is a numeric unique record identifier. These are handled by the error page to redirect to the appropriate record.

I have just had all these pages successfully indexed by Fast (they do all have genuine individual content!).

A handful of these pages have recently shown up in Google as temporary, dated items (freshbot; pages now dropped again). Now waiting for the google update to see if it works there too.

In a week or so I will be in a position to make a comparison!

10:06 am on Nov 29, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 23, 2002
votes: 0

Yes, worked on Google too.


12:08 pm on Nov 29, 2002 (gmt 0)

Moderator from DK 

WebmasterWorld Administrator 10+ Year Member

joined:Oct 23, 2000
votes: 1

Good for you HenryUK. You are saying that both FAST and GG had no problem spidering all your pages?

So have the server come down from all the traffic? ;)