Forum Moderators: open
Now i have launched my site and opened up a part of the site which was previously unnavalible to the se's what kind of different behahiour should i see?
I noticed that the other day i had around 1 hit from inktomi every two minutes or so, then today inktomi were basically slamming my site around 3 times a minute, this has been going on for a good 12 hours and as you can tell thats a lot of hits from the SE :o
Also another quick question... Google seems to have passed totlly by the parts of the sites (subdomains) that have all the updated pages, whilst other part of the site have been updated in google, and i know the spoder has been there :)
Thanks for the help
Google's regular deep-crawler is used to build their monthly index. It can take deep-crawler 30 to 60 days to add your pages to this "permanent" index. If you have access to your logs, it comes from IP addresses beginning with 216.
Google's "freshbot" visits high-PageRank and frequently-updated pages more often, with the frequency of visits depending on how often you update your page and its PageRank (PR). However, freshbot additions to Google's index are temporary, and may in fact disappear and reappear over the course of a month. Freshbot's IP addresses start with 64.
Since you state that your pages were added to Google's index within a few hours, it is certain that they were added by the freshbot.
Until your pages have been on-line for 30 to 60 days, all bets are off as to whether they will appear in the Google search results at any given time. If some of your pages don't appear, that means their PR is too low for freshbot. If some of your currently-listed pages disappear, don't worry until after the next monthly update.
The other search engines have update schedules which range from a few days to several months.
HTH,
Jim
