homepage Welcome to WebmasterWorld Guest from 23.23.57.144
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

This 120 message thread spans 4 pages: < < 120 ( 1 2 [3] 4 > >     
Deepbot is in the house....
I just seen deepbot on my server all
teeceo

10+ Year Member



 
Msg#: 11809 posted 10:30 pm on Apr 15, 2003 (gmt 0)

Its deepbot time!

teeceo.

 

ircgeeks

10+ Year Member



 
Msg#: 11809 posted 11:41 pm on Apr 17, 2003 (gmt 0)

On my second deep crawl for this site and its at 1321 hits this time. still trucking Go deepbot go!

I have been seeing alot of Inktomi deep craws to that is a first for me. About 600 hit's so far on this crawl

Oaf357

10+ Year Member



 
Msg#: 11809 posted 11:54 pm on Apr 17, 2003 (gmt 0)

When it rains it pours...

Freshbot has been hitting me up for two days then WHAM... Deepbot is tearing through with Inktomi bringing up the rear.

Smiles.

markis00

10+ Year Member



 
Msg#: 11809 posted 2:55 am on Apr 18, 2003 (gmt 0)

I disappeared from google entirely today...I checked my logs and haven't been spidered by freshie or deep today or yesterday. I checked ww2.google.com, I'm not there, and if I type in my domain I should come up but I don't on anything. I know people get dropped from google sometimes for a couple of days, but question is does their entire domain get dropped too?
Help me out here please.

stuntdubl

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 11809 posted 3:41 am on Apr 18, 2003 (gmt 0)

>When it rains it pours...

My terrible unreliable host that I can't regain my domain name from went down today and hasn't come up yet. I want to scream.

nextblock

10+ Year Member



 
Msg#: 11809 posted 4:48 am on Apr 18, 2003 (gmt 0)

Deepbot has looked at our siteindex and index pages too. But no deep crawl yet, sadly.

I understand it could take 10 days to crawl.

-Adrian

[edited by: Marcia at 5:19 am (utc) on April 19, 2003]
[edit reason] No sigs, please. [/edit]

winterradio

10+ Year Member



 
Msg#: 11809 posted 5:10 am on Apr 18, 2003 (gmt 0)

I know people will think I'm a conspiracy nut, but, it seems google crawls more amoung sites that advertise. I run 5 different sites and the differnce is more than noticable. I knew it was only a matter of time

tim3562

10+ Year Member



 
Msg#: 11809 posted 5:27 am on Apr 18, 2003 (gmt 0)

yes, I see that the googlebot hit my sites.

question-
when a site that was just deep botted show up in the index?
one site is new, and it was fresbotted like last week, disappeared( I know why cause I read here about freshbots)
now when will it be re-indexed?

also on another site,
which is older and bigger, it it showing a really old index page in google's index. ( this domain is available page).. I snatched it up someone let it expire.

this site was debotted too, when will the real page show?

uci_bink

10+ Year Member



 
Msg#: 11809 posted 5:32 am on Apr 18, 2003 (gmt 0)

Question:

The deepbot has been hanging around my page for a couple of days now but seems to be crawling extremely slow. My site has a low PR (it is very new) but has about 5,000 or so pages. This is my second deepcrawl and I was hoping google would gobble all of it...but judging by the freqeuncy of it hitting me it wont. Is this a good way to judge how much the bot likes me...should it be going faster?

Thanks,
Kevin

futureX

10+ Year Member



 
Msg#: 11809 posted 1:32 pm on Apr 18, 2003 (gmt 0)

i'm getting hit pretty slow too compared to inktomi

cowcool

10+ Year Member



 
Msg#: 11809 posted 12:32 am on Apr 19, 2003 (gmt 0)

I deleted hundreds of old pages yesterday in the middle of deep crawl. Will this have bad effect on me?

Night_Hawk

10+ Year Member



 
Msg#: 11809 posted 5:08 am on Apr 19, 2003 (gmt 0)

nextblock,

You need to read the rules for this forum, you can not have a URL in your post.:)

BGumble

10+ Year Member



 
Msg#: 11809 posted 3:48 pm on Apr 19, 2003 (gmt 0)

I really wasn't panicking.. heh.. Deepbots are out in force this morning, looks like normal pace. Thanks for releasing them trillianjedi :)

>>I deleted hundreds of old pages yesterday in the middle of deep crawl.

Some may have been indexed, some may not have been. You might have a few 404's for an update, no big deal.

Farrukh

10+ Year Member



 
Msg#: 11809 posted 4:08 pm on Apr 19, 2003 (gmt 0)

Indeed the deep crawl has started on my site too.

sharpie_79

10+ Year Member



 
Msg#: 11809 posted 10:31 pm on Apr 19, 2003 (gmt 0)

crawling seems really slow compared to last month?

trillianjedi

WebmasterWorld Senior Member trillianjedi us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 11809 posted 9:42 am on Apr 20, 2003 (gmt 0)

I'm glad you guys have got some of them.

Looking through the logs, at one point we have 87 of them together with 4 freshbots in the site for 4 hours continuous at about 4 hits a second.

Madness.

Still getting deepbotted now, but only 3 of them in there and being reasonably well behaved and gentle. So I think they fixed the problem, but I haven't received a phone call or email yet. I'll let you know the outcome.

I never thought I'd see the day where I had my finger hovering over the "ban googlebot" button.

;-)

TJ

tsomers

10+ Year Member



 
Msg#: 11809 posted 1:46 am on Apr 21, 2003 (gmt 0)

I'm new to this. How often does Google crawl?

On what day of the month should you plan to have changes completed on your site to be ready for the crawl?

And how can you monitor the crawl on your site?

Thanks.

eblitz

10+ Year Member



 
Msg#: 11809 posted 5:22 pm on Apr 21, 2003 (gmt 0)

May be you can help your self by scrolling up a few pages
to read all the posted.

Thanks

NexDog

10+ Year Member



 
Msg#: 11809 posted 4:14 am on Apr 22, 2003 (gmt 0)

Deepbot is staill hitting us. Been 7 days now?

futureX

10+ Year Member



 
Msg#: 11809 posted 4:49 am on Apr 22, 2003 (gmt 0)

Still hitting me too...

A few questions about bot algos...

How does a bot move through your site, does one goto a page and then grab all the links then other bots come for those links and so-on? Or does one bot come and visit more than one page at a time, I seem to have about a dozen coming at once, but not sure of their direction :)

Thx

bielle64

10+ Year Member



 
Msg#: 11809 posted 8:02 pm on Apr 23, 2003 (gmt 0)

I'm afraid this full crawl is over and the deepbot only visited my index page.
My site is new and only has PR0 but I don't think it's a good reason, there are a lot of link to the other pages on this index. I'm so disappointed, I have to wait for a month now.
Does it happens a lot?

My site is ossau dot net if one of you can notice a problem... maybe in the robots.txt
I don't think because other bots like Pompos have no problem to crawl my site.

help!

Critter

10+ Year Member



 
Msg#: 11809 posted 10:18 pm on Apr 23, 2003 (gmt 0)

Deep is not over yet...came back to me after a 6 hour lull and has done another 400 or so (and counting fast :) pages.

Peter

Adam_C

10+ Year Member



 
Msg#: 11809 posted 8:42 am on Apr 24, 2003 (gmt 0)

bielle64>>>

I've seen the deep bot hit only the homepage of a PR6 site this time round - very strange indeed.

bielle64

10+ Year Member



 
Msg#: 11809 posted 9:43 am on Apr 24, 2003 (gmt 0)

Maybe they changed something to the robots program.
It's not a good change!

Or maybe it's not over...

jezzer72

10+ Year Member



 
Msg#: 11809 posted 10:43 am on Apr 24, 2003 (gmt 0)

It's definately not over: She's still around. Deepbot must be female as she's so irregular, and comes and goes as she pleases ;-)

Adam_C

10+ Year Member



 
Msg#: 11809 posted 12:01 pm on Apr 24, 2003 (gmt 0)

only one hit from deep bot, but loads/constantly from Fresh.

le_gber

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 11809 posted 12:37 pm on Apr 24, 2003 (gmt 0)

Deepbot came on the 17th, grab the robots.txt and nothing else. No sign of her ;) since. Site is PR0 and here the robots.txt file:
User-agent: *
Disallow: /folder1/
Disallow: /folder2/
Disallow: /folder3/

Is there anything I should be worried about?

Leo

John_Caius

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 11809 posted 12:52 pm on Apr 24, 2003 (gmt 0)

Currently 11 deepbot versions (216) and 1 freshbot (64)

e.g.

IP Address 216.239.46.239
Reverse DNS crawl9.googlebot.com
From / Via Biznessonline
Origin Location New York, USA.
Time Spent 12 min
Hits / Kilobytes 1 / 2.64Kb
Browser Tag Googlebot/2.1 (+http://www.googlebot.com/bot.html)

Each bot seems to take one file but it's a real page, not robots.txt .

meinereiner

10+ Year Member



 
Msg#: 11809 posted 5:42 pm on Apr 24, 2003 (gmt 0)

1500 Hits by GoogleBot today, with about 500 Deepbot and 1000 Freshbot Hits, so he/she is still there definitely and I wish it will never go :-)

*dream-modus on* just index all and every page and put them on no. 1 position next month *dream-modus off*)

Adam_C

10+ Year Member



 
Msg#: 11809 posted 2:44 pm on Apr 25, 2003 (gmt 0)

Any sightings today?

John_Caius

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 11809 posted 2:55 pm on Apr 25, 2003 (gmt 0)

Not as we speak...

Critter

10+ Year Member



 
Msg#: 11809 posted 2:58 pm on Apr 25, 2003 (gmt 0)

Ok, who's tempted to start a "the update has started" thread? :)

Peter

This 120 message thread spans 4 pages: < < 120 ( 1 2 [3] 4 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved