Welcome to WebmasterWorld Guest from 54.159.26.69

Forum Moderators: open

Message Too Old, No Replies

Fast spider smarter than Googlebot?

better handling of long dynamic URLs

     

Finder

1:44 am on Nov 13, 2002 (gmt 0)

10+ Year Member



After a recent discussion in the Google forum on how to better entice the Googlebot, I realized that my dynamic URLs were too long for Googlebot to handle. So, I changed them. The very next day, the Fast bot stops by and asks for all the old links that I just changed!

I removed the subsection part of my URLs:

index.html?section=foo&subsec=fee&specific=a+b+c+d

Becomes:

index.html?section=foo&specific=a+b+c+d

My site has never been fully indexed by the Googlebot. Now it's been extensively indexed by the Fast bot, but most of the pages are in the form of "Error: requested document not found."

I just can't win sometimes. :)

xbase234

12:02 am on Nov 19, 2002 (gmt 0)

10+ Year Member



I have no problems with Googlebot using 2-3 variables. Maybe it's because of the 4 variables?

HenryUK

3:52 pm on Nov 27, 2002 (gmt 0)

10+ Year Member



I run a database-driven site with tens of thousands of individual records, with a churn of about 10% per month (ie a couple of thousand records drop out, and about the same number are created)

The site has dynamically created pages containing links to all those individual pages. Instead of using the query string these links are in the form www.mydomain.com/BlahXXXXX.htm where XXXXX is a numeric unique record identifier. These are handled by the error page to redirect to the appropriate record.

I have just had all these pages successfully indexed by Fast (they do all have genuine individual content!).

A handful of these pages have recently shown up in Google as temporary, dated items (freshbot; pages now dropped again). Now waiting for the google update to see if it works there too.

In a week or so I will be in a position to make a comparison!

HenryUK

10:06 am on Nov 29, 2002 (gmt 0)

10+ Year Member



Yes, worked on Google too.

:)

Rumbas

12:08 pm on Nov 29, 2002 (gmt 0)

WebmasterWorld Administrator 10+ Year Member



Good for you HenryUK. You are saying that both FAST and GG had no problem spidering all your pages?

So have the server come down from all the traffic? ;)

 

Featured Threads

Hot Threads This Week

Hot Threads This Month