Welcome to WebmasterWorld Guest from 50.19.190.144

Forum Moderators: mademetop

Message Too Old, No Replies

Crawlers Out In Force.

     
7:11 am on Nov 28, 2002 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38048
votes: 12


This week, we saw full crawls from:

Fast
Inktomi
Google
Wisenut
Altavista

That's all the major crawlers but Teoma. I think that is the first time in ages where 5 engines have all been actively crawling at the same time.

It's nice to see them all out fighting for it again!

8:05 am on Nov 28, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:May 13, 2002
posts:149
votes: 0


And inktomi just looped and looped while generating 20,000 404 hits. All this due to a handful of links that were out by one folder level. Just makes a huge mess of the logs!

But it sure was a busy week.

8:50 am on Nov 28, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 29, 2000
posts:1133
votes: 0


Brett, I haven't seen Inktomi in my logs, except PFI pages. I have some older sites that have some PFI pages, and get only PFI crawls, and new sites that, despite having good links from other established sites, haven't been getting crawled by Slurp.

Can you tell if the Ink crawling you are seeing includes new sites (three months old or newer), or free pages on PFI sites?

11:59 am on Nov 28, 2002 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38048
votes: 12


Sorry, all older sites Mayor.
3:20 pm on Nov 28, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 3, 2001
posts:1609
votes: 0


Very nice activity from the major players, serps are improving as well.

I'm glad to see that there is still a race!

If AV would kill the poker/toner invitation links after each results page, they might get back faster, credibility is pretty well demolished with that nonsense.

[edited by: john316 at 3:36 pm (utc) on Nov. 28, 2002]

3:27 pm on Nov 28, 2002 (gmt 0)

Senior Member from ZA 

WebmasterWorld Senior Member 10+ Year Member

joined:July 15, 2002
posts:1720
votes: 1


I had Fast and Google taking near enough the same pages at the same time yesterday, and Wisenut is currently having a look around my site right now :)
1:58 pm on Nov 29, 2002 (gmt 0)

New User

10+ Year Member

joined:Nov 24, 2002
posts:4
votes: 0


How can I know that a full craw was done on my site.
I have an old site but optimized it in mid octobre, no changes on the engines so far.

My cached page in google is still the old site.

When can I expect to see my modifications?

thank you.

3:16 pm on Nov 29, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 16, 2001
posts:112
votes: 0


We have Inktomi and LS on our site, every single day. Should I be concerned?
4:48 pm on Nov 29, 2002 (gmt 0)

Moderator from US 

WebmasterWorld Administrator martinibuster is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 13, 2002
posts:13973
votes: 123


Even Robozilla stopped by to see if I was still around!
11:52 am on Dec 3, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 2, 2002
posts:792
votes: 0


All this is very nice, but the relevency of results from all except google is E grade.

Hands up all those that will use anything but google if looking for nightclubs and discount hotels for your next vacation to Spain.

Why don't the search engines understand that 1st it's relevancy . Just read all kinds of nonsense in Inktomi's press release about Geo relevance and blah blah. Typical kind of press release put out by an overpaid & disinterested PR department.

Then some user called "Inktomi" says "we're very excided about the future"...well my friend, we are not!