homepage Welcome to WebmasterWorld Guest from 54.163.139.36
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
Google Deep Crawling My site Today.
Chief

10+ Year Member



 
Msg#: 12449 posted 3:30 pm on May 1, 2003 (gmt 0)

Crawling again. About 20+ bots on my site. Crawl1....(but is the crawl or crawler the deepbot? I forget.

Does this mean the update is farther away than we would like?

 

John_Caius

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 12449 posted 3:34 pm on May 1, 2003 (gmt 0)

216 is the deepbot, 64 is the freshbot.

PeterHo

10+ Year Member



 
Msg#: 12449 posted 2:16 pm on May 2, 2003 (gmt 0)

How do you know that bot's have visited your page?
By the IP-numbers of you visitors?

ncsuk

10+ Year Member



 
Msg#: 12449 posted 2:22 pm on May 2, 2003 (gmt 0)

My site has been deep crawled as well and some pages 4 levels deep have been picked up

speda1

10+ Year Member



 
Msg#: 12449 posted 5:42 pm on May 2, 2003 (gmt 0)

You cannot tell if its deepbot by the pages that are grabbed. You have to look at the IP of the bot. See John's post above.

tomapple

10+ Year Member



 
Msg#: 12449 posted 6:31 pm on May 2, 2003 (gmt 0)

havn't seen the deepbot today, just "freshie" hitting a bunch of pages

taxpod

10+ Year Member



 
Msg#: 12449 posted 6:31 pm on May 2, 2003 (gmt 0)

I'm seeing only freshy.

kmtell

10+ Year Member



 
Msg#: 12449 posted 8:22 pm on May 2, 2003 (gmt 0)

I have a ton of 216.* hits on my site today.

speda1

10+ Year Member



 
Msg#: 12449 posted 8:59 pm on May 2, 2003 (gmt 0)

With a user agent of 'Googlebot'?

markusf



 
Msg#: 12449 posted 9:11 pm on May 2, 2003 (gmt 0)

I get a lot of these on my site..

216.239.33.5
Nokia6610/1.0 (4.18) Profile/MIDP-1.0 Configuration/CLDC-1.0 (Google WAP Proxy/1.0)

speda1

10+ Year Member



 
Msg#: 12449 posted 1:27 am on May 3, 2003 (gmt 0)

That looks human to me.

matrix_neo

10+ Year Member



 
Msg#: 12449 posted 4:18 am on May 3, 2003 (gmt 0)

What is the difference between up date and google dance (I believe both are the same). Do you guys mean update is PR calculation after the google Or PR calculation is also happening simultaneously?

speda1

10+ Year Member



 
Msg#: 12449 posted 5:24 pm on May 3, 2003 (gmt 0)

The update and the dance are the same thing. Each of the google data centers are updated with the new index. Because Google load balances with DNS, and because it takes some time to update all of the data centers, search results fluctuate during this time, hence the 'dance'.

PR is most likely calculated during the period between the deep crawl and the start of the dance. We are in this period now.

Jesse_Smith

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 12449 posted 11:54 pm on May 3, 2003 (gmt 0)

freshBot: 64.68.82.* bah, listed for only a few days, short dinner date then it dumps you.
deepcrawler: 216.239.46.* Good bot, it likes you, listed until death do you depart. So don't make him mad or it will divorce your site. You can divorce the Googlebot by using your robot.txt file. It's much cheaper and faster than going to the courts.

If it's the Googlebot it will say

Googlebot/2.1 (+http://www.googlebot.com/bot.html)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved