Welcome to WebmasterWorld Guest from 184.73.3.107

Forum Moderators: open

Deepbot is in the house....

I just seen deepbot on my server all

   
10:30 pm on Apr 15, 2003 (gmt 0)

10+ Year Member



Its deepbot time!

teeceo.

2:58 pm on Apr 25, 2003 (gmt 0)

10+ Year Member



Ok, who's tempted to start a "the update has started" thread? :)

Peter

3:32 pm on Apr 25, 2003 (gmt 0)

10+ Year Member



nothing here, although scooter has been busy...
4:09 pm on Apr 25, 2003 (gmt 0)

10+ Year Member



Right, Scooter is much more efficient than Google these days. Alatavista strikes back?
8:40 am on Apr 26, 2003 (gmt 0)

10+ Year Member



scooter crawling is too fast...
+ results don't show up in the index..
i'm going to block scooter..
9:52 am on Apr 26, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Got hit by 216.239.46.165 less than 6 hours ago.
10:05 am on Apr 26, 2003 (gmt 0)

10+ Year Member



Seeing the 216.* as we speak.
11:40 am on Apr 26, 2003 (gmt 0)

10+ Year Member



Deep's in here too.

I agree with the Scooter comment about not showing up in the index...have the same problem (I think) with Ink. Ink crawled 3,500 odd pages and as far as I can see 13 are in the index--boneheaded.

Now, I may be wrong because it hasn't been a full month or so since the crawl, but we'll see.

Peter

12:51 pm on Apr 26, 2003 (gmt 0)

10+ Year Member



Deepie (216.239.46.*) is back after stopping the deep crwal 2 days ago. What is going on? Is google re-crawling the entire web? Maybe the last crawl was no good?

Will this delay the next update?

1:06 pm on Apr 26, 2003 (gmt 0)

10+ Year Member



Deep took a break with me too, and is back in today.

Only doing new pages though, it's not a reset.

Peter

1:11 pm on Apr 26, 2003 (gmt 0)

10+ Year Member



Critter: Only doing new pages though, it's not a reset.

The Deepie is crawling all my sites, mostly old pages?!

If this is a new, complete crawl then it will take another 10 days to complete!?
=>
Definitely no April update!
May update is in doubt?

1:30 pm on Apr 26, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>Right, Scooter is much more efficient than Google these days. Alatavista strikes back?

*Superlol* - scooter calls itself a crawler. Ok, yah, they crawl. However, new pages take ages to make it into the av index ... at least i hope they will get listed some day ...

Why do you think it's more efficient than google? To me, efficient means - crawl frequently - index continiously - show fresh results within days or hours. Sorry, that's what google does - afaik, no other engine currently does the same.

My newest site (est. April 2) has been crawled by google after it's been online for 5 hours and was listed after less than 36 hours. THAT is efficient!

1:43 pm on Apr 26, 2003 (gmt 0)

10+ Year Member



216.239.46.99
216.239.46.55
216.239.46.42
216.239.46.41
216.239.46.36
216.239.46.33
216.239.46.222
216.239.46.172
216.239.46.116
216.239.46.102

*and*
64.68.82.66

have all just hit my site

(scooter 3.x did as well, wonder if they fought?)

2:03 pm on Apr 26, 2003 (gmt 0)

10+ Year Member



What happens if server resets while pages are being crawled? does it start all over or does it pick up from where it got cut off?
3:38 pm on Apr 26, 2003 (gmt 0)

10+ Year Member



Welcome to WW, MaxPower.

Google's DB holds the results of the crawled pages & URL's to be crawled. So a server reset should not mess anything up -- as long as you're not relying to much on session variables. Pages heavily driven by session vars are not very friendly to a multi-spider crawl anyway, as far as I understand.

4:21 pm on Apr 26, 2003 (gmt 0)

10+ Year Member



still haven't had a full deep crawl.

Did get a full freshbot crawl a few days ago though (64.ip), and those pages aren't showing in the SERPS as fresh. Is Google messing with my mind?

4:44 pm on Apr 26, 2003 (gmt 0)

10+ Year Member



I got a fresh listing on Google right now, but the fresh tag isn't there. Dunno.

Peter

5:30 pm on Apr 26, 2003 (gmt 0)

10+ Year Member



anybody ever considered that the clever boys and girls over in the googleplex have figured out a way to start a new crawl cycle while one is still underway? Just a thought. Could be the solution to keeping wihtin a fresh index every 30 days with all those thousands of new pages on the net every second.
8:54 pm on Apr 26, 2003 (gmt 0)

10+ Year Member



Guys,
my site has been being nailed by scooter the last couple of days but hasn't been deep or freshed in nearly two weeks. I disappeared from Google and am wondering, could I have been banned?
10:50 pm on Apr 26, 2003 (gmt 0)

10+ Year Member



Just seen deepbot again(216.239.46.17 [26/Apr/2003:15:46:35), how long will it crawl? And If it crawls this long, when can we expect a update?

teeceo.

12:51 am on Apr 27, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I last got the deepcrawl seven hours ago, chonping away at a section that I put up four days ago! I'm guessing the May update will start between the 10th and 15th.
9:47 am on Apr 27, 2003 (gmt 0)

10+ Year Member



Can someone please reply to my above post
I am worried
12:03 pm on Apr 27, 2003 (gmt 0)

10+ Year Member



I noticed freshbot today picking up dynamic pages with 3 and 4 variables. Looks like something new to me.
12:56 pm on Apr 27, 2003 (gmt 0)

10+ Year Member



My brand spanking new site got deepbotted for the second time this crawl last night, at the same time as a freshie was in there too. Unfortunately my pulled-out hair is still dusting my workstation as they continue to ignore my asp pages with query string variables. And I don't have much left.

Can I perhaps blame my PR NULL <grey bar> for why they won't crawl down into my product pages, or should I start retooling my entire URL framework?

Darryl

1:18 pm on Apr 27, 2003 (gmt 0)

10+ Year Member



Well it looks like Deep's done with me. On the other hand Ink's back in :(

Stupid Inktomi, doing a crawl *again* when they crawled 3,500 pages or so on my site last month and included *13* in the index.

Boneheaded...

Peter

2:23 pm on Apr 27, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>Can I perhaps blame my PR NULL <grey bar> for why they won't crawl down into my product pages

That would be my guess. I have a brand new site just indexed last crawl. The ONLY backlink is the $300 Yahoo listing which gives the site a PR3.

Deepbot grabbed all 100 product pages (*.asp?myid=99) on the first crawl. However, there are only 5 main static pages so maybe it's good to keep Deepbot bored on the static end of a new dynamic site.

7:33 pm on Apr 29, 2003 (gmt 0)

10+ Year Member



I think the Fully is over now :(
Anyone still have visits ?

I just had a visit on my index :'(

8:06 pm on Apr 29, 2003 (gmt 0)

10+ Year Member



I mean I ONLY had a visit on my index page.
8:23 am on Apr 30, 2003 (gmt 0)

10+ Year Member



bielle64 - I have a very well established site that also didn't see deepbot on any page bar the index. However, it did get a full fresh-crawl over a week ago and those pages aren't showing as fresh.

I think Google may have changed servers or something.

Also agree that it seems better with dynamic pages - had 200 dynamic pages crawled on a brand new site this crawl.

7:50 am on May 1, 2003 (gmt 0)

10+ Year Member



Unfortunately no deepbot or freshbot in the last 3 days now :( But hey, he was there before, next googledance near? :D
4:07 pm on May 1, 2003 (gmt 0)

10+ Year Member



Deepbot and Fresh are becoming one....
This 120 message thread spans 4 pages: 120
 

Featured Threads

Hot Threads This Week

Hot Threads This Month