Spider Behaviour

Googlebot clueless?

[futureX]

I dont know if i've posted this in the right place, but I just need to ask a few questions about googlebot behaviour... And also compare it to other spider's behviour, I may have mentioned this before, infact i'm sure I have. But the plot has thickened;

About 2 months ago I made my 5,000 page forums avalible to Google and all the other SEs, all the bots have come to visit the new pages and some have indexed them, some havent:

Inktomi visits my site a few times everyday, and every 2 weeks or so sends around a thousand spiders a day for a few days, which brings my inktomi hits up to around 5k Hits a month, but there is only one of those pages indexed. The rest are not being added to the index, they are being viewed, and the spider keeps coming back for more, but none of them get into the index. If it is following links surely it can index the pages?

Next up Is google, Visited the site and did nothing for one month, next month it came and it indexed 60 pages (after around 1k hits), they got into the index and are sitting there now, i thought, ok maybe it will crawl more next month (as it seemed to have crawled in a logial manner). However this month it has hit me on the 27th (300 hits) and has not come back since, will the bot return and do some more, or is something up?

Alltheweb seems to have come and only hit on the first level of links, and only indexed the category pages. Wondering if ATW will return too? Jeeves has come and hit around 1000 times, but no pages yet... Others dont seem to have done much.

[futureX]

uh, thanks...

[trillianjedi]

Hi futureX - sorry, I missed this thread last time around, so it must have scrolled off the screen pretty fast...

I can only really comment on two of the bots, but maybe with a little activity in this thread others can come and jump in with some answers too.

AskJeeves are renowned for their slowness in getting data into the index. My best guess is they update about once every three months, unless you PFI.

AJ came and grabbed about 170 of our forum pages (only a small proportion but it's a start) two months ago and so far not a sign of them in the index.

The google deepbot is known to increase the depth that it spiders each time it visits, if it likes you. You may have to wait for the next deepcrawl to get the bulk of your pages in there and then a few subsequent updates should see you get more and more in google.

Freshbot will probably not shove all of them in the index (based on my experience anyway).

TJ

[futureX]

thanks, I think its just been the freshbot thats come to the site, so I guess i'll just have to wait.