homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

Fast spider smarter than Googlebot?
better handling of long dynamic URLs

10+ Year Member

Msg#: 709 posted 1:44 am on Nov 13, 2002 (gmt 0)

After a recent discussion in the Google forum on how to better entice the Googlebot, I realized that my dynamic URLs were too long for Googlebot to handle. So, I changed them. The very next day, the Fast bot stops by and asks for all the old links that I just changed!

I removed the subsection part of my URLs:


My site has never been fully indexed by the Googlebot. Now it's been extensively indexed by the Fast bot, but most of the pages are in the form of "Error: requested document not found."

I just can't win sometimes. :)



10+ Year Member

Msg#: 709 posted 12:02 am on Nov 19, 2002 (gmt 0)

I have no problems with Googlebot using 2-3 variables. Maybe it's because of the 4 variables?


10+ Year Member

Msg#: 709 posted 3:52 pm on Nov 27, 2002 (gmt 0)

I run a database-driven site with tens of thousands of individual records, with a churn of about 10% per month (ie a couple of thousand records drop out, and about the same number are created)

The site has dynamically created pages containing links to all those individual pages. Instead of using the query string these links are in the form www.mydomain.com/BlahXXXXX.htm where XXXXX is a numeric unique record identifier. These are handled by the error page to redirect to the appropriate record.

I have just had all these pages successfully indexed by Fast (they do all have genuine individual content!).

A handful of these pages have recently shown up in Google as temporary, dated items (freshbot; pages now dropped again). Now waiting for the google update to see if it works there too.

In a week or so I will be in a position to make a comparison!


10+ Year Member

Msg#: 709 posted 10:06 am on Nov 29, 2002 (gmt 0)

Yes, worked on Google too.



WebmasterWorld Administrator 10+ Year Member

Msg#: 709 posted 12:08 pm on Nov 29, 2002 (gmt 0)

Good for you HenryUK. You are saying that both FAST and GG had no problem spidering all your pages?

So have the server come down from all the traffic? ;)

Global Options:
 top home search open messages active posts  

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved