Welcome to WebmasterWorld Guest from 54.196.145.173

Forum Moderators: open

Message Too Old, No Replies

Slurp out on full crawl?

     
2:01 pm on Jan 13, 2003 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38063
votes: 13


Every site I've looked at this morning has been spidered hard by Inktomi the last few hours.

Is this the start of the first big Yahoo db build?

2:04 pm on Jan 13, 2003 (gmt 0)

Senior Member from ZA 

WebmasterWorld Senior Member 10+ Year Member

joined:July 15, 2002
posts:1721
votes: 4


I am seeing this to on a few of my sites.

Craig

2:05 pm on Jan 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 16, 2001
posts:2059
votes: 0


saw it to

Tor

2:26 pm on Jan 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 31, 2000
posts:786
votes: 0


I see it here in Norway also.
2:54 pm on Jan 13, 2003 (gmt 0)

New User

10+ Year Member

joined:Jan 10, 2003
posts:10
votes: 0


I haven't seen this on my sites yet but maybe it just hasn't gotten to them yet. Probably will soon.
2:57 pm on Jan 13, 2003 (gmt 0)

Moderator

WebmasterWorld Administrator lawman is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:May 28, 2001
posts:3569
votes: 42


They've hit both of my sites. Sorta keeps you on pins and needles about any potential change to Y! results.

lawman

2:59 pm on Jan 13, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 22, 2002
posts:453
votes: 0


And here - just to add my voice to the crowd.

R.

3:00 pm on Jan 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 3, 2002
posts:1590
votes: 0


Same here. It's like a "did you see the UFO over London the other day?" post. ;)
3:03 pm on Jan 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 23, 2002
posts:1052
votes: 0


Yep, slurp has certainly increased its activities here too. Isn't this what we've all expected to see?
3:11 pm on Jan 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 4, 2000
posts:1324
votes: 0


Same Here

-s-

3:27 pm on Jan 13, 2003 (gmt 0)

New User

10+ Year Member

joined:Dec 19, 2002
posts:11
votes: 0


What's the IP address of this bot?
3:30 pm on Jan 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 22, 2001
posts:2044
votes: 0


I've got this one inktomi4-not.server.ntl.com
but it's a few days old.....
4:26 pm on Jan 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member zeus is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 28, 2002
posts:3444
votes: 1


I got a notice a few days ago that I could not see which site has been crawled, but I don think they have spidered any more then the index page, but then spider the whole web now.

zeus

4:28 pm on Jan 13, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 24, 2002
posts:87
votes: 0


Haven't seen them yet. How do the spider anyway? Do they start at low IP-addresses and then go up?
5:04 pm on Jan 13, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 8, 2002
posts:465
votes: 0


Slurp came and got my main site yesterday. Crawled the deepest it has in the past couple of months.
5:55 pm on Jan 13, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 23, 2002
posts:165
votes: 0


Substantial slurp traffic over the weekend. By far the biggest since 7/8 December.
Got individual pages listed for the first time two days before that...
6:42 pm on Jan 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 29, 2000
posts:1133
votes: 0


Anyone seeing slup spider the unpaid pages on a site that has some PFI pages on it?
8:20 pm on Jan 13, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 8, 2002
posts:465
votes: 0


mayor, Yes.
8:50 pm on Jan 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 2, 2002
posts:918
votes: 0


When are we likely to see a refresh of the database?
Any clues?
J
9:10 pm on Jan 13, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:May 5, 2002
posts:68
votes: 0


Same here. I have PFI but now they are spidering the whole site, including pages for which I donīt PFI... and they spider all the stuff under another domain as the one on PFI...
9:11 pm on Jan 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 2, 2002
posts:792
votes: 0


whopee...how does that Inktomi algo go? :)
9:25 pm on Jan 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 23, 2002
posts:1052
votes: 0


Methinks we are a few on this board who are gonna try and learn about inking. ;)
10:27 pm on Jan 13, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Apr 23, 2002
posts:314
votes: 0


Where can I submit to Inktomi?
11:55 pm on Jan 13, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Dec 30, 2002
posts:437
votes: 0


Inktomi has requested 3070 pages on my main site this month.

It seems to have been crawling me since sometime in December almost constantly. Very slow going compared with googlebot, which "slurped" ;) over 50,000 pages in the January crawl, stopping on the 11th.

-Pete

11:58 pm on Jan 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member korkus2000 is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 20, 2002
posts:3732
votes: 0

Ink submit
http://www.hotbot.lycos.co.uk/submit.html
12:12 am on Jan 14, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 29, 2000
posts:1133
votes: 0


Whoohooo ... can't get my logs until around 3AM. U betcha I'll have a fresh pot of coffee ready.
12:15 am on Jan 14, 2003 (gmt 0)

Moderator from US 

WebmasterWorld Administrator robert_charlton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2000
posts:11584
votes: 240


korkus - I tried the hotbot.lycos.co.uk submission form a few days ago. It now submits to Fast, not Inktomi.

For Inktomi submit, this still works:

[submitit.bcentral.com...]

To be considered for inclusion in the Inktomi Search Index, powering the MSN Web Pages section

Note also, though, on the MSN site:

...only one url allowed per site allowed per day.

They really seem to keep track.

3:48 am on Jan 14, 2003 (gmt 0)

Full Member

10+ Year Member

joined:May 24, 2002
posts:284
votes: 0


You guys think we will see free traffic from Inktomi after Yahoo paid 235M or so....I am not that sure....

Yahoo wants Inktomi PFI or even better PPC.

Get ready to pay in order to play.

12:06 pm on Jan 14, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 28, 2002
posts:96
votes: 0


1300 pages crawled out a total of 2500 in our site requested from December, 1st.

Seems that Inktomi crawls at a very low pace, 15 pages/hour in the "fast moments" ;) and 1 in the low.

4:00 pm on Jan 14, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 18, 2002
posts:63
votes: 0


As I wrote in another post I also saw a big increase in traffic from Ink the biggest I ever seen. It was some new domains which had got in for free. 5 days later (sunday) the domains were nowere to be found on Ink. Maybe they are doing this to sell more inclusion...You know show webmasters how much traffic you could get from MSN and etc then turn it off...

/esset

This 37 message thread spans 2 pages: 37