Google Deep Crawling My site Today.

Forum Moderators: open

Message Too Old, No Replies

Google Deep Crawling My site Today.

Chief

3:30 pm on May 1, 2003 (gmt 0)

Crawling again. About 20+ bots on my site. Crawl1....(but is the crawl or crawler the deepbot? I forget.

Does this mean the update is farther away than we would like?

John_Caius

3:34 pm on May 1, 2003 (gmt 0)

216 is the deepbot, 64 is the freshbot.

PeterHo

2:16 pm on May 2, 2003 (gmt 0)

How do you know that bot's have visited your page?
By the IP-numbers of you visitors?

ncsuk

2:22 pm on May 2, 2003 (gmt 0)

My site has been deep crawled as well and some pages 4 levels deep have been picked up

speda1

5:42 pm on May 2, 2003 (gmt 0)

You cannot tell if its deepbot by the pages that are grabbed. You have to look at the IP of the bot. See John's post above.

tomapple

6:31 pm on May 2, 2003 (gmt 0)

havn't seen the deepbot today, just "freshie" hitting a bunch of pages

taxpod

6:31 pm on May 2, 2003 (gmt 0)

I'm seeing only freshy.

kmtell

8:22 pm on May 2, 2003 (gmt 0)

I have a ton of 216.* hits on my site today.

speda1

8:59 pm on May 2, 2003 (gmt 0)

With a user agent of 'Googlebot'?

markusf

9:11 pm on May 2, 2003 (gmt 0)

I get a lot of these on my site..

216.239.33.5
Nokia6610/1.0 (4.18) Profile/MIDP-1.0 Configuration/CLDC-1.0 (Google WAP Proxy/1.0)

speda1

1:27 am on May 3, 2003 (gmt 0)

That looks human to me.

matrix_neo

4:18 am on May 3, 2003 (gmt 0)

What is the difference between up date and google dance (I believe both are the same). Do you guys mean update is PR calculation after the google Or PR calculation is also happening simultaneously?

speda1

5:24 pm on May 3, 2003 (gmt 0)

The update and the dance are the same thing. Each of the google data centers are updated with the new index. Because Google load balances with DNS, and because it takes some time to update all of the data centers, search results fluctuate during this time, hence the 'dance'.

PR is most likely calculated during the period between the deep crawl and the start of the dance. We are in this period now.

Jesse_Smith

11:54 pm on May 3, 2003 (gmt 0)

freshBot: 64.68.82.* bah, listed for only a few days, short dinner date then it dumps you.
deepcrawler: 216.239.46.* Good bot, it likes you, listed until death do you depart. So don't make him mad or it will divorce your site. You can divorce the Googlebot by using your robot.txt file. It's much cheaper and faster than going to the courts.

If it's the Googlebot it will say

Googlebot/2.1 (+http://www.googlebot.com/bot.html)