|Will DeepCrawl hit twice in one month?|
Last month it hit around the 3rd and then 12th...
So I screwed up, and thought I had made my homepage links spiderfriendly, but apparently did not. First thing I fixed was to remove the relative path nature of them. All my links used to be like this:
I fixed that, removing the dot, but all the links still appended a session variable to the URLs (if cookies are unavailable) such as
I had thought that Googlebot could handle one (but only one) variable after the url, but maybe mine was too long...
anyway, I've modified my session code so that it won't present this variable to bots, based on the HTTP_USER_AGENT string.
My question is, do I have to wait another 30days to see the DeepCrawl bot again? Most of my sites saw DeepCrawl twice, around Jan 3 and again around Jan 12.
Is it possible that it will come back soon and notice that I've fixed the links? I'd really like to see this bot finally crawl my whole site and not get stuck on the frontpage!
I guess I neglected to mention it explicitly, but the DeepCrawl bot:
crawl4.googlebot.com - - [06/Feb/2003:02:56:38 -0500] "GET .....
has been to the sites I'm talking about yesterday night and this AM. Each site, it took robots, / (redirected), and the index.php?sess_id=... file, but got no further.
In december, I saw deepbot (216.x.x.x)appear twice
(or once, with a one day gap in the middle).
In January, i got hit nonstop for 9 days.
Prior to december, if memory serves me correctly,
deepbot only came once in the month.
freshbot (64.x.x.x) hit my site almost every day.
When repeat visits occurred (twice in dec, many times in jan), did it ever get your main index page more than once?
The bots do not like session variables, so yes, that would have been your problem. I have noticed that the Googlebot can handle more than one variable in a url though, possibly depending on the site's PR.
I'm not sure about the deep crawl bot visiting more than once a month, but I have found that pages added later in the month can still get in the index - last month I added pages around the 12th or so and they still made it into the January update, so I reckon you still stand a chance of getting listed.