Forum Moderators: open

Message Too Old, No Replies

Googlebot - Deepcrawl period

         

Albaba

2:27 pm on Feb 7, 2003 (gmt 0)

10+ Year Member



Just curious
How long time the deep crawling? is it up to 10 days or ...?

thanks

lazerzubb

2:43 pm on Feb 7, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Normally for around 5-8 days i would say, then sometimes it comes back in the middle and pick up a few pages.
(P.s, we call it Googlebot)

taxpod

3:07 pm on Feb 7, 2003 (gmt 0)

10+ Year Member



I'm curious how much time Gbot typically spends on everyone's sites. I usually see her for 72 hours, then nothing. She usually hits my site then gradually speeds up until she is at her fastest speed on my site for about 48 hours - usually a page every 4 seconds. Next she begins to slow and then she is gone, never to be seen until the next deepcrawl.

xlcus

3:18 pm on Feb 7, 2003 (gmt 0)

10+ Year Member



How long time the deep crawling?

From last months logs:

First sign of DeepCrawl bot...

  • 04/01/2002 04:17:23

    Last sign of DeepCrawl bot...

  • 12/01/2002 06:04:08
  • Grumpus

    3:21 pm on Feb 7, 2003 (gmt 0)

    WebmasterWorld Senior Member 10+ Year Member



    That sounds about right. For my site, from start to finish is usually 7-10 days. My site is large, though. For smaller sites, I'd assume it is only there as long as it takes to crawl the entire site or the maximum # of pages it's PR sets.

    G.

    Albaba

    3:22 pm on Feb 7, 2003 (gmt 0)

    10+ Year Member



    well 8 days :-)...i think it enough to crawl all my 150k pages :-)

    thanks for the info guys

    xlcus

    3:34 pm on Feb 7, 2003 (gmt 0)

    10+ Year Member



    i think it enough to crawl all my 150k pages

    You have a site with 150000 pages? That sounds huge! What sort of site is it?

    Albaba

    3:39 pm on Feb 7, 2003 (gmt 0)

    10+ Year Member



    yep xlcus
    i generated site with php/mysql (huge database)...
    i check log file, now she eating my pages :)...very agresive :)

    ExtremeExports

    4:21 pm on Feb 7, 2003 (gmt 0)

    10+ Year Member



    >>I'd assume it is only there as long as it takes to crawl the entire site or the maximum # of pages it's PR sets.<<

    I don't understand that, what do you mean by Gbot will only crawlt the max # of pages it's pr sets. I've added 100 pages before this deepcrawl my pr is a 5. I have a total of 200 pages. Will they all be crawled or does pr 5 only allow less?

    ExtremeExports

    4:22 pm on Feb 7, 2003 (gmt 0)

    10+ Year Member



    >>I'd assume it is only there as long as it takes to crawl the entire site or the maximum # of pages it's PR sets.<<

    I don't understand that, what do you mean by Gbot will only crawlt the max # of pages it's pr sets. I've added 100 pages before this deepcrawl my pr is a 5. I have a total of 200 pages. Will they all be crawled or does pr 5 only allow less?

    SubZeroGTS

    4:54 pm on Feb 7, 2003 (gmt 0)

    10+ Year Member



    i have a PR5 site with 13000+ pages indexed last month

    mfishy

    5:24 pm on Feb 7, 2003 (gmt 0)

    WebmasterWorld Senior Member 10+ Year Member



    When GG deepcrawls a page a few times let's say in the period of a week, does it index the page it sees on the last crawl if you make changes?

    wired4fun

    4:49 am on Feb 9, 2003 (gmt 0)

    10+ Year Member



    Near as i can tell, it does a recrawl from
    your home page, or any inbound links. If
    googlebot can't find a link to a page it crawled
    last cycle, it wont crawl it again.

    We did a wholesale change to our urls, and
    within 2 mos all the old urls were out of
    the index, except those that had inbound links.
    (we added some apache rewrite rules to keep
    all of the old links functional)

    Albaba

    10:24 am on Feb 9, 2003 (gmt 0)

    10+ Year Member



    wow good googlebot :)
    She still crawl my site...i hope her eat all my pages ...