Forum Moderators: open

Message Too Old, No Replies

When was the deep crawl

especially on Oct 3

         

Powdork

8:05 am on Oct 12, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The UUNET backbone was experiencing difficulties on Oct 3 from about 7 am edt to 4 pm edt. Unfortunately, myclientsite.com was down during most of this period. Does anyone have a record of Googlebot (not 64.68...freshbot, but the deep crawler 216.239...) visiting their site's during this period or was the deep crawl over before this.
myclientsite.com's stat package has a copyright date of 2000. It shows no referrers for Oct 1-3 although it does show visitors. It even shows vistors for times when I know the site was down. It also list the only two search engines as Yahoo! and alta vista. It also list Google referrals as referrals from a domain, not a search engine. They had it way wrong back then because Google was already a search engine and Yahoo! was still a directory.

shelleycat

8:56 am on Oct 12, 2002 (gmt 0)

10+ Year Member



My site was deep crawled on the 4th October. I'm not sure what time zone my log files are in (hmm, must check that) but the 4th here is still the 3rd in many places. But there seems to be a wide window in which googlebot crawls, other people were probably crawled much earlier and some many still be getting crawled now. Fingers crosses she hasn't found you yet.

Grumpus

12:34 pm on Oct 12, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Google seems to still be crawling now (at least I was getting some MainBot hits last night, haven't checked this morning).

Google is certainly aware of the outage and if they couldn't get through, they'll likely keep trying. As I said, I'm still seeing evidence of the main crawler touching down on my site (and I haven't seen the "end of crawl frenzy" yet). There's still a good chance she'll pop in.

G.

bnhall

1:53 pm on Oct 12, 2002 (gmt 0)

10+ Year Member



We got deep crawled on the 3rd.

Chico_Loco

3:07 pm on Oct 12, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I got deep crawled on the 2nd, with requests for the robots.txt and homepage ongoing daily..

Powdork

5:44 pm on Oct 12, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Never thought to look so recently. She grabbed robots.txt and the homepage on the eleventh. Hopefully she was checking things out for a visit later today. I'll go turn down a bed, whatever that means.
Thanks

allanp73

5:06 pm on Oct 13, 2002 (gmt 0)

10+ Year Member



I was deep crawled on the 4th and 8th.

Powdork

7:07 pm on Oct 13, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The references to 216.239 that were in my stat program for the eleventh are now gone! I'm afraid the buggy program was misleading me the entire time and I probably have nothing to worry about. Funny, when it did show up it listed it as biznewsonline.
Thanks for your all your help.

lazyz

9:59 pm on Oct 14, 2002 (gmt 0)

10+ Year Member



Until two days ago, I had several domains getting crawled everday since the last update, deep crawled, many pages each time. I've only seen googlebot grab my index page over the last couple of days.

martin

10:13 pm on Oct 14, 2002 (gmt 0)

10+ Year Member



From:

216.239.46.148 - - [03/Oct/2002:11:41:19 -0500]

to:

216.239.46.100 - - [03/Oct/2002:19:08:28 -0500]

19 non-fresh pages.