Welcome to WebmasterWorld Guest from 54.196.244.186

Forum Moderators: open

Message Too Old, No Replies

Deepbot is in the house....

I just seen deepbot on my server all

     
10:30 pm on Apr 15, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:June 15, 2002
posts:396
votes: 0


Its deepbot time!

teeceo.

2:58 pm on Apr 25, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 31, 2003
posts:386
votes: 0


Ok, who's tempted to start a "the update has started" thread? :)

Peter

3:32 pm on Apr 25, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 10, 2003
posts:157
votes: 0


nothing here, although scooter has been busy...

bielle64

4:09 pm on Apr 25, 2003 (gmt 0)

Inactive Member
Account Expired

 
 


Right, Scooter is much more efficient than Google these days. Alatavista strikes back?
8:40 am on Apr 26, 2003 (gmt 0)

New User

10+ Year Member

joined:Jan 18, 2003
posts:16
votes: 0


scooter crawling is too fast...
+ results don't show up in the index..
i'm going to block scooter..
9:52 am on Apr 26, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 30, 2002
posts:741
votes: 0


Got hit by 216.239.46.165 less than 6 hours ago.
10:05 am on Apr 26, 2003 (gmt 0)

New User

10+ Year Member

joined:Apr 25, 2003
posts:7
votes: 0


Seeing the 216.* as we speak.
11:40 am on Apr 26, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 31, 2003
posts:386
votes: 0


Deep's in here too.

I agree with the Scooter comment about not showing up in the index...have the same problem (I think) with Ink. Ink crawled 3,500 odd pages and as far as I can see 13 are in the index--boneheaded.

Now, I may be wrong because it hasn't been a full month or so since the crawl, but we'll see.

Peter

12:51 pm on Apr 26, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Nov 25, 2002
posts:207
votes: 0


Deepie (216.239.46.*) is back after stopping the deep crwal 2 days ago. What is going on? Is google re-crawling the entire web? Maybe the last crawl was no good?

Will this delay the next update?

1:06 pm on Apr 26, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 31, 2003
posts:386
votes: 0


Deep took a break with me too, and is back in today.

Only doing new pages though, it's not a reset.

Peter

1:11 pm on Apr 26, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Nov 25, 2002
posts:207
votes: 0


Critter: Only doing new pages though, it's not a reset.

The Deepie is crawling all my sites, mostly old pages?!

If this is a new, complete crawl then it will take another 10 days to complete!?
=>
Definitely no April update!
May update is in doubt?

1:30 pm on Apr 26, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 8, 2002
posts:2012
votes: 0


>Right, Scooter is much more efficient than Google these days. Alatavista strikes back?

*Superlol* - scooter calls itself a crawler. Ok, yah, they crawl. However, new pages take ages to make it into the av index ... at least i hope they will get listed some day ...

Why do you think it's more efficient than google? To me, efficient means - crawl frequently - index continiously - show fresh results within days or hours. Sorry, that's what google does - afaik, no other engine currently does the same.

My newest site (est. April 2) has been crawled by google after it's been online for 5 hours and was listed after less than 36 hours. THAT is efficient!

1:43 pm on Apr 26, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 23, 2003
posts:77
votes: 0


216.239.46.99
216.239.46.55
216.239.46.42
216.239.46.41
216.239.46.36
216.239.46.33
216.239.46.222
216.239.46.172
216.239.46.116
216.239.46.102

*and*
64.68.82.66

have all just hit my site

(scooter 3.x did as well, wonder if they fought?)

2:03 pm on Apr 26, 2003 (gmt 0)

New User

10+ Year Member

joined:Apr 14, 2003
posts:7
votes: 0


What happens if server resets while pages are being crawled? does it start all over or does it pick up from where it got cut off?
3:38 pm on Apr 26, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Mar 3, 2003
posts:306
votes: 0


Welcome to WW, MaxPower.

Google's DB holds the results of the crawled pages & URL's to be crawled. So a server reset should not mess anything up -- as long as you're not relying to much on session variables. Pages heavily driven by session vars are not very friendly to a multi-spider crawl anyway, as far as I understand.

4:21 pm on Apr 26, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Apr 4, 2003
posts:410
votes: 0


still haven't had a full deep crawl.

Did get a full freshbot crawl a few days ago though (64.ip), and those pages aren't showing in the SERPS as fresh. Is Google messing with my mind?

4:44 pm on Apr 26, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 31, 2003
posts:386
votes: 0


I got a fresh listing on Google right now, but the fresh tag isn't there. Dunno.

Peter

5:30 pm on Apr 26, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:June 25, 2002
posts:85
votes: 0


anybody ever considered that the clever boys and girls over in the googleplex have figured out a way to start a new crawl cycle while one is still underway? Just a thought. Could be the solution to keeping wihtin a fresh index every 30 days with all those thousands of new pages on the net every second.
8:54 pm on Apr 26, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Apr 18, 2003
posts:305
votes: 0


Guys,
my site has been being nailed by scooter the last couple of days but hasn't been deep or freshed in nearly two weeks. I disappeared from Google and am wondering, could I have been banned?
10:50 pm on Apr 26, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:June 15, 2002
posts:396
votes: 0


Just seen deepbot again(216.239.46.17 [26/Apr/2003:15:46:35), how long will it crawl? And If it crawls this long, when can we expect a update?

teeceo.

12:51 am on Apr 27, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 5, 2003
posts:807
votes: 0


I last got the deepcrawl seven hours ago, chonping away at a section that I put up four days ago! I'm guessing the May update will start between the 10th and 15th.
9:47 am on Apr 27, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Apr 18, 2003
posts:305
votes: 0


Can someone please reply to my above post
I am worried
12:03 pm on Apr 27, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 20, 2002
posts:139
votes: 0


I noticed freshbot today picking up dynamic pages with 3 and 4 variables. Looks like something new to me.
12:56 pm on Apr 27, 2003 (gmt 0)

New User

10+ Year Member

joined:Apr 9, 2003
posts:19
votes: 0


My brand spanking new site got deepbotted for the second time this crawl last night, at the same time as a freshie was in there too. Unfortunately my pulled-out hair is still dusting my workstation as they continue to ignore my asp pages with query string variables. And I don't have much left.

Can I perhaps blame my PR NULL <grey bar> for why they won't crawl down into my product pages, or should I start retooling my entire URL framework?

Darryl

1:18 pm on Apr 27, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 31, 2003
posts:386
votes: 0


Well it looks like Deep's done with me. On the other hand Ink's back in :(

Stupid Inktomi, doing a crawl *again* when they crawled 3,500 pages or so on my site last month and included *13* in the index.

Boneheaded...

Peter

2:23 pm on Apr 27, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 22, 2003
posts:1483
votes: 0


>Can I perhaps blame my PR NULL <grey bar> for why they won't crawl down into my product pages

That would be my guess. I have a brand new site just indexed last crawl. The ONLY backlink is the $300 Yahoo listing which gives the site a PR3.

Deepbot grabbed all 100 product pages (*.asp?myid=99) on the first crawl. However, there are only 5 main static pages so maybe it's good to keep Deepbot bored on the static end of a new dynamic site.

bielle64

7:33 pm on Apr 29, 2003 (gmt 0)

Inactive Member
Account Expired

 
 


I think the Fully is over now :(
Anyone still have visits ?

I just had a visit on my index :'(

bielle64

8:06 pm on Apr 29, 2003 (gmt 0)

Inactive Member
Account Expired

 
 


I mean I ONLY had a visit on my index page.
8:23 am on Apr 30, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Apr 4, 2003
posts:410
votes: 0


bielle64 - I have a very well established site that also didn't see deepbot on any page bar the index. However, it did get a full fresh-crawl over a week ago and those pages aren't showing as fresh.

I think Google may have changed servers or something.

Also agree that it seems better with dynamic pages - had 200 dynamic pages crawled on a brand new site this crawl.

7:50 am on May 1, 2003 (gmt 0)

New User

10+ Year Member

joined:Apr 24, 2003
posts:6
votes: 0


Unfortunately no deepbot or freshbot in the last 3 days now :( But hey, he was there before, next googledance near? :D
4:07 pm on May 1, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Aug 31, 2002
posts:284
votes: 0


Deepbot and Fresh are becoming one....
This 120 message thread spans 4 pages: 120