Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: open
I fixed that, removing the dot, but all the links still appended a session variable to the URLs (if cookies are unavailable) such as
I had thought that Googlebot could handle one (but only one) variable after the url, but maybe mine was too long...
anyway, I've modified my session code so that it won't present this variable to bots, based on the HTTP_USER_AGENT string.
My question is, do I have to wait another 30days to see the DeepCrawl bot again? Most of my sites saw DeepCrawl twice, around Jan 3 and again around Jan 12.
Is it possible that it will come back soon and notice that I've fixed the links? I'd really like to see this bot finally crawl my whole site and not get stuck on the frontpage!
crawl4.googlebot.com - - [06/Feb/2003:02:56:38 -0500] "GET .....
has been to the sites I'm talking about yesterday night and this AM. Each site, it took robots, / (redirected), and the index.php?sess_id=... file, but got no further.
Prior to december, if memory serves me correctly,
deepbot only came once in the month.
freshbot (64.x.x.x) hit my site almost every day.
The bots do not like session variables, so yes, that would have been your problem. I have noticed that the Googlebot can handle more than one variable in a url though, possibly depending on the site's PR.
I'm not sure about the deep crawl bot visiting more than once a month, but I have found that pages added later in the month can still get in the index - last month I added pages around the 12th or so and they still made it into the January update, so I reckon you still stand a chance of getting listed.