Forum Moderators: open
Here's the URL that I saw in my logs:
[almaden.ibm.com...]
Recently, I haven't seen them much, but it's been "peekin' and pokin'", to use some of my favorite old tyme computer commands.
It seems Almaden is working on some sort of new fangled search algo's of some kind. From my foggy memory of what I managed to glean this summer, it seems like it might be some sort of combination of Natural Language (a la Ask Jeeves) and several other things that I can't remember right now. As of then, I remember thinking that it was a neat idea, but that it didn't look like anyone was interested. (Which is why I assumed they were looking to pitch to Yahoo). If we start seeing more activity in the next few weeks, it likely means that they're freshening up the database to pitch it to someone else that's new (or they've got a major new factor in the Algo and want to test it on new stuff).
In the end, though, it's a pretty agressive bot. I let it scuff the top level of my site just in case it picks up someone big, but keep it out of the deep pages until I know exactly what it is they're doing. It DOES seem to be somewhat smart, like the Googlebot, in that it'll back off if it notes that things are bogging down on your server.
G.
Well, I can't do anything about as I depend on my ISP (Netgate--don't know if they're any good; their service/support is excellent, though) for allowing or disalloing anything at the server level.
What I have noticed is that Almaden is NOT responsible on my site at least, for big loads of traffic. I did notice a sudden increase of my log size that has been going on for about a week now, but when I analyzed the log Almaden couldn't be responsible.
Googlebot has paid me daily visits, the last week, but it hasn't resulted in a dramatic increase of traffic :(. I'm now at about 10,000 visitors a month and some 15,000 page views. After 3 months of trying hard to do what Brett has been saying on these forums all along, I find that disappointing. But that could be me, of course--I'm not experienced.
I thought that would also stop other bots, and I don't want to stop Googlebot! Now that I have his attention, I want it to be kept alive as long as possible ;).
Craig
They may not be a player today, but with the resources they are deploying to crawl (they are very aggresive), I have to think they are serious.
I know they are very active in the search market on enterprise level, i.e. search functionality for intranets and for their Lotus Notes environment. But I didn't know they were also interested in Internet search. Their natural language stuff is said to be quite good, but that's probably too little too late: Gartner and Giga have recently been saying people really don't want to type in lengthy sentences in natural language anymore. They just want to see accurate result lists based on 3 search terms maximum...
Anyway, gentlemen, you all have been very helpful once more. It's 7 PM over here, and I'm going to log off, so you won't hear from me on this subject until tomorrow morning Wetsern European Time (summertime not included ;) ).
Thanks again and see you all tomorrow!
Erik
Craig
My experience with this one is that it has not followed robot.txt, and has nosed around where it shouldn't. That, and it's very aggressive nature, and I stopped it through other means.
dave
