homepage Welcome to WebmasterWorld Guest from 50.17.7.84
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
Will DeepCrawl hit twice in one month?
Last month it hit around the 3rd and then 12th...
WileE




msg:49635
 11:24 pm on Feb 6, 2003 (gmt 0)

So I screwed up, and thought I had made my homepage links spiderfriendly, but apparently did not. First thing I fixed was to remove the relative path nature of them. All my links used to be like this:
[somesite.abc...]
or
[somesite.abc...]

I fixed that, removing the dot, but all the links still appended a session variable to the URLs (if cookies are unavailable) such as

[somesite.abc...]

I had thought that Googlebot could handle one (but only one) variable after the url, but maybe mine was too long...

anyway, I've modified my session code so that it won't present this variable to bots, based on the HTTP_USER_AGENT string.

My question is, do I have to wait another 30days to see the DeepCrawl bot again? Most of my sites saw DeepCrawl twice, around Jan 3 and again around Jan 12.

Is it possible that it will come back soon and notice that I've fixed the links? I'd really like to see this bot finally crawl my whole site and not get stuck on the frontpage!

 

WileE




msg:49636
 11:37 pm on Feb 6, 2003 (gmt 0)

I guess I neglected to mention it explicitly, but the DeepCrawl bot:

crawl4.googlebot.com - - [06/Feb/2003:02:56:38 -0500] "GET .....

(for example)

has been to the sites I'm talking about yesterday night and this AM. Each site, it took robots, / (redirected), and the index.php?sess_id=... file, but got no further.

hskfun




msg:49637
 1:06 am on Feb 7, 2003 (gmt 0)

In december, I saw deepbot (216.x.x.x)appear twice
(or once, with a one day gap in the middle).
In January, i got hit nonstop for 9 days.

Prior to december, if memory serves me correctly,
deepbot only came once in the month.

freshbot (64.x.x.x) hit my site almost every day.

WileE




msg:49638
 1:16 am on Feb 7, 2003 (gmt 0)

hskfun -

When repeat visits occurred (twice in dec, many times in jan), did it ever get your main index page more than once?

daamsie




msg:49639
 2:25 am on Feb 7, 2003 (gmt 0)

Hi WileE

The bots do not like session variables, so yes, that would have been your problem. I have noticed that the Googlebot can handle more than one variable in a url though, possibly depending on the site's PR.

I'm not sure about the deep crawl bot visiting more than once a month, but I have found that pages added later in the month can still get in the index - last month I added pages around the 12th or so and they still made it into the January update, so I reckon you still stand a chance of getting listed.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved