Welcome to WebmasterWorld Guest from 54.221.119.45

Forum Moderators: open

Message Too Old, No Replies

Will DeepCrawl hit twice in one month?

Last month it hit around the 3rd and then 12th...

     

WileE

11:24 pm on Feb 6, 2003 (gmt 0)

10+ Year Member



So I screwed up, and thought I had made my homepage links spiderfriendly, but apparently did not. First thing I fixed was to remove the relative path nature of them. All my links used to be like this:
[somesite.abc...]
or
[somesite.abc...]

I fixed that, removing the dot, but all the links still appended a session variable to the URLs (if cookies are unavailable) such as

[somesite.abc...]

I had thought that Googlebot could handle one (but only one) variable after the url, but maybe mine was too long...

anyway, I've modified my session code so that it won't present this variable to bots, based on the HTTP_USER_AGENT string.

My question is, do I have to wait another 30days to see the DeepCrawl bot again? Most of my sites saw DeepCrawl twice, around Jan 3 and again around Jan 12.

Is it possible that it will come back soon and notice that I've fixed the links? I'd really like to see this bot finally crawl my whole site and not get stuck on the frontpage!

WileE

11:37 pm on Feb 6, 2003 (gmt 0)

10+ Year Member



I guess I neglected to mention it explicitly, but the DeepCrawl bot:

crawl4.googlebot.com - - [06/Feb/2003:02:56:38 -0500] "GET .....

(for example)

has been to the sites I'm talking about yesterday night and this AM. Each site, it took robots, / (redirected), and the index.php?sess_id=... file, but got no further.

hskfun

1:06 am on Feb 7, 2003 (gmt 0)

10+ Year Member



In december, I saw deepbot (216.x.x.x)appear twice
(or once, with a one day gap in the middle).
In January, i got hit nonstop for 9 days.

Prior to december, if memory serves me correctly,
deepbot only came once in the month.

freshbot (64.x.x.x) hit my site almost every day.

WileE

1:16 am on Feb 7, 2003 (gmt 0)

10+ Year Member



hskfun -

When repeat visits occurred (twice in dec, many times in jan), did it ever get your main index page more than once?

daamsie

2:25 am on Feb 7, 2003 (gmt 0)



Hi WileE

The bots do not like session variables, so yes, that would have been your problem. I have noticed that the Googlebot can handle more than one variable in a url though, possibly depending on the site's PR.

I'm not sure about the deep crawl bot visiting more than once a month, but I have found that pages added later in the month can still get in the index - last month I added pages around the 12th or so and they still made it into the January update, so I reckon you still stand a chance of getting listed.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month