| 10:30 pm on Aug 22, 2003 (gmt 0)|
I'm in a similar situation. Google grabs the main front page links (product categories), a few of the subdirectory links (specific product categories) beneath those...but never beyond that to our actual product pages. PR is only at 2, and we've only been online now for about 3 months...but Google hits us 4-5 times/week and never goes further than explained.
| 10:44 pm on Aug 22, 2003 (gmt 0)|
I don't think google is deepcrawling as much this month... very wierd as last month they were deepcrawling everyday.
| 11:20 pm on Aug 22, 2003 (gmt 0)|
Same problem here. New pages not getting spidered. For the last 2 weeks, freshbot is only spidering pages already in the index, not new pages.
| 11:50 pm on Aug 22, 2003 (gmt 0)|
Yup just checked some of the new sites I put out there and lots of links are not getting spidered. Wonder if this means google is slowing down their continuious update or maybe they are moving back towards the dance... Oh well it was good while it lasted :/
| 12:28 am on Aug 23, 2003 (gmt 0)|
Isn't that what site maps are for? The poor spider probably doesn't know which way to go. Hopefully Google hasn't come up with an internal cross linking penalty.
|1000 or so links to internal pages from each of the 25 pages. |
| 2:01 am on Aug 23, 2003 (gmt 0)|
just to let you know my sites have site maps.. I think google will do a deep crawl real soon, else their index might shrink which i don't think they want that to happen, they want to include every tidbit of information possible even if they don't rank it high i think they would rather have it than not, correct me if i'm wrong GoogleGuy if you're around.
| 10:05 am on Aug 23, 2003 (gmt 0)|
Google acted the same way like at your sites. But right now it started to grab all of them ...
22442 /grillrezepte/detailrezept.php crawler10.googlebot.com 18.104.22.168 Googlebot/2.1 (+http://www.googlebot.com/bot.html) 2003-08-23 12:03:49 .COM googlebot.com? Googlebot 0 0 0 0 0
22441 /grillrezepte/detailrezept.php crawler10.googlebot.com 22.214.171.124 Googlebot/2.1 (+http://www.googlebot.com/bot.html) 2003-08-23 12:03:45 .COM googlebot.com? Googlebot 0 0 0 0 0
22440 /grillrezepte/detailrezept.php crawler14.googlebot.com 126.96.36.199 Googlebot/2.1 (+http://www.googlebot.com/bot.html) 2003-08-23 12:03:43 .COM googlebot.com? Googlebot 0 0