Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: open
<Date: September 18, 2000 06:56 PM
Author: Liane (BareboatsBVI@surfbvi.com)
Subject: Inktomi Url Submissions - Don't do it!!!
I am new to this site but I sure wish I had found it months ago as I see I am not alone in my ongoing Inktomi problems. In March, I launched a new site and after encountering more troubles than I care to remember, I finally got an answer.
I wrote to one search engine after another in an attempt to find out why my site was picked up ... then dropped, then picked up only to be dropped time and time again. I spent hundreds of hours on the internet trying to find a solution to a problem that nobody seemed aware of ... least of all the people who push the buttons to send the canned responses saying things such as, "perhaps your server was down" or "Often the metatags are the source of problems like this" ... and (my favourite), "site submissions are free and not all URL's are automatically accepted".
At first, I was convinced that our host was at fault as we had recently changed hosts. Then I came to believe that our new webmaster had done something not quite kosher and we were being punished. Then I thought it may have been our local ISP as they had experienced trouble with DNS server updating. Finally, the paranoia set in and I became convinced that our site had been blacklisted by Inktomi!
I had the site checked and rechecked every which way from Sunday. I had it rewritten, redesigned, reworked over and over again trying to find the solution to the problem. I couldn't get any straight forward answers from anyone. I was ready to close the doors and go back to the print advertising industry.
Finally, a very kind person at Canada.com took pity on my plight and forwarded my letter to Inktomi. The very next day I rec'd a phone call from Mr. Tim Gilbert of Inktomi's "back end" technical department. Mr. Gilbert took the time to explain what was taking place at Inktomi and why my site (and many thousands of others) would appear and then disappear on Hot Bot, GoTo, Canada.com, 4 Anything, LookSmart, MSN, About.com and many others. He explained that the "submit URL" feature on the various search engines was the source of all the trouble.
Spammers (mostly p*rn*graphy sites) have learned how to circumvent the system and have overloaded the Inktomi database with p*rn sites. In order to combat this problem, Inktomi came up with some sort of algorithm solution which would make the rankings of these sites less relevant in popularity. However, since they do not have any live editors who sort each and every site in their database, their robot is unable to determine what is or isn’t p*rn. ERGO ... anyone using the “add url” feature on the various search engines is automatically penalized in the rankings system.
Mr. Gilbert and I spoke for about 10 minutes or more and although he was very candid and seemed sincerely concerned ... the end result is that there is nothing you can do about it just now.
If you search any sites using Inktomi, you will find that very few new sites appear and that the relevancy of their search results have been degraded to the point of being "less than useful". (At least that has been my experience.) Excite is a good example of a good search engine gone bad. The searches I have done recently were a joke. The relevancy of results I got were laughable.
Mr. Gilbert further explained that the problem was so daunting that their technical department actually staged a one day walk out to protest the problems they were facing and to bring the matter to the attention of management.
Inktomi is (or was) bantering about the idea of introducing a pay to submit programme ... but they sure are taking their time about it. Considering the extent of Inktomi's importance to the e:commerce marketplace, their lack of immediate response and a reasonable solution to the problem is shocking.
In early August, I decided to try LookSmart's "pay to submit programme" and the results were almost excellent. Our rankings in relevant areas are superb and they did a very professional job of editing the description. It appeared in LookSmart's database within 4 days of sending in the submission. It took MSN about 2 weeks to pick it up and the results there were even better! On the down side, it has still not appeared on Excite (nearly 6 weeks now) and it actually dropped drastically in the rankings on Alta Vista and Raging??? It has yet to
[edited by Brett Tabke : note: original author has given permission for posting]
What a find!!!! Thank you so much for the re-post. I am going to hunt down "Liane" and see if we can get him/her to do some follow-up over here.
Again thank you - you get the best find of the year award in my book. And if it all pans out Liane gets the best post award!!!
At last - something that makes sense...
So, if the above posting about Ink is true, and it does appear to be factual, our worst fears are realized ... Ink has basically shut down more or less for good! Maybe they'll come back with a last hurrah trying to collect an exit fee disguised as a paid submission. If spamming is the reason they shut down, I don't see where paid submissions are going to change any of that.
But all is not lost. My site traffic is slowly recovering from the loss of Ink traffic. Could it be that surfers are abandoning the stale Ink portals and moving over to greener pastures, like Alta Vista and Google? People aren't stupid, you know. Does anyone know how long most surfers hang around a the carcus of a dead search engine when the party is over?
If it all pans out, Inktomi gets the stupidest idea of the year award. Who has heard of a search engine database penalizing someone for submitting to their engine when they encourage people to do so? If this is true, this will really further damage their reputation among webmasters (or maybe they don't care?)
What exquisite irony! If it wasn't so pathetic, it would be comical. I mean really, "How does Inktomi identify spam?" - simple. Anything that's submitted via the add url page.
I wonder...who are they going to turn to?? Google?? Altavista?? These seem to be the only two that are functioning RELATIVELY normally...OR maybe this is going to signify a shift and portals are going to start using primarily directories?? Mainly DMOZ and Looksmart??
Hard to tell...all we can do is speculate...??? or how do we go about preparing for the impending changes??? can we??
Just my thoughts...
I've been watching site logs to see if the spidered pages are going to stick, but, as yet, i have no evidence for that.
I'm taking a longer term view to "getting indexed" and expect Ink to sort out this issue in due course, just as AV did after "Black Monday" last October. It took months for the wrinkles to be ironed out of AV's index. I think we can say AV is ok now.
There are other engines/directories in the competitive world out there. If all the engines/directories have the same problem at the same time then that is serious. Currently, that is not the case.
I'm focusing on the other engines and directories without losing sight of the Ink engine and it's importance.
Stay with it, until you get nothing from Ink - just don't spend so much time on it.
Yes. As proof of this use my company name 'makemetop' on HotBot and iWon. Down there in the lower pages you will see pages that I have never submitted at #24 and #27 (a page from Microchannel Technologies and one from cybermarketing.uk.com). The first page was one I wrote and didn't finish (transfering it to my main site) but was linked to Microchannel's main page which has never been submitted to Inktomi. Cybermarketing is a site I've kept in reserve and never developed and has 1 link from another site which is in Inktomi's main DB. Both of these pages have been spidered and are listed without using meta tags.
Imagine what a turnaround this could be. Do nothing and get lots of Ink traffic without having to constantly re-submit dropped pages! Could we be so lucky?
I've got some work to do...
Pan out indeed. I am checking with a friend of a friend to see if this story is valid. One would think that a "one day walk out" by the technical team would *somehow* get to the media, or at least the rumor mill. People talk.
And the rest of it sounds so ludicrous...even though it does explain the erratic activity on Inktomi's partners.
And the rest of it sounds so ludicrous...even though it does explain the erratic activity on Inktomi's partners.>
I don't disagree with the absurdity factor, grnidone, but I felt that this was worthy enough to be bantered around here rather than dismissed out of hand. I value the opinion of the participants here more so than any of the other SEO forums.
I don't claim to have all the answers, but I sure do have a heck of a lot of questions! ;)
I hope that you can either prove or disprove it one way or another, though.
Oooooh, I just thought of something.
(I'm a little slow)
If the above letter is correct, then if you really wanted to hurt your competitors...submit them to Inktomi.
/me ducks to avoid flying objects.
It's not about over submitting your competition. It's about over submitting in general and INK trying to protect their database.
The old "over submit your competitor" to kill his/her site has been long gone from INK.
Liane's reposted post sounds to me like INK is trying to work out a solution to spammy submissions in general and they are having problems with it.
To me, it's one of the most positive things I've heard lately - becasue if it's true, when INK gets it all worked out, we may be back in business again
(Names changed...hell these guys won't even talk to me,
only to my husband..and that is when they choose to.)
NICK_1 and N3 are with the search engine end of it. (not sure what they do) nick_2 is a programmer for Inktomi, but not nec. with the search engine part.
15:00 <NICK_1> as far as addurl goes: it's true that it's mostly congested with spammers
15:00 <NICK_1> BUT, addurl is pretty useless to begin with, because we already make a webmap and crawl most of it anyway
15:00 <nick_2> yeah; the more interesting thing in that posting was the technical department walkout.
15:01 <NICK_1> The "paid inclusion" program mentioned on that page will replace the current "free" system and will just mean we refresh those urls more often. It WONT change the relevance computations.
16:12 <nick_2> oh yeah, definitive answer was there was no walk out, no idea about it.
16:12 <N3> heh, bunch of drivel on there
16:18 <N3> this is all very humorous in that most of it is false
Hope this helps...although I don't think it will.
I was e:mailed and told that somebody had copied my letter from another forum and asked me to take part in this discussion. I am simply a business owner and in no way do I qualify as a webmaster as my knowledge of "everything internet" is very limited.
Having said that, over the past 5 months, I have been forced to learn more about search engines than I ever wanted to know. In fact, I am on a first name basis with at least 3 editors from various directories!
I have read all messages in this string and found them very interesting. Quite frankly, I was surprised that all this seems to come as real news to all of you.
To Mike, I just want to say that I have had a site in the Inktomi data base for more than 2 years without any problems at all. (So where do I collect my share of the ticket sales?)
To Grnidone ... I can only reiterate that the above statements are all true. (My bible is under my right hand and my left hand is over my heart). I'm afraid I have no way to prove what was said during my discussion with Tim Gilbert of Inktomi as I unfortunately forgot to raise the cone of silence in order to flip on the tape machine. (Drat, I do that all the time!)
Of course what I have stated would, in a court of law, be considered hearsay as I have no proof. I can however, forward you several e:mails back and forth between Canada.com and myself which references this discussion and content of same. In addition, someone else made this comment on the other site after reading my post:
This confirms my feedback from two Inktomi portals which also stated '....there is a problem
with the submit URL feature...until is it is sorted results will be unpredictable...'. One of these
was canada.com. This, however, is the first post which explains the problem in depth and I, for
one, will heed the advice.
I certainly have no reason to fabricate any part of my post and I doubt that Mr. Gilbert (of Inktomi) had any reason to tell me anything which wasn't true. What would be the point?
Anyway, my post was merely an effort to share my experiences with others who may be suffering from the same syndrome as I. (My Mom always said sharing was a good thing!) Whomever chooses to believe what I have reported "is all drivel" ... is certainly free to do so. It only took me 4 months of letter writing and pestering to finally get the answer from Mr. Gilbert. It seemed to make sense to me. Perhaps somebody has a better answer they could share which would explain the problems with Inktomi submissions? I'm sure it will all come out in the wash eventually anyway.
I'm certainly open to anything which will help rescue my foundering site!
(No axes to grind or crosses to bare)
P.S. I did an in depth survey of my site and its search results using specific keyword strings on 18 major search engines. The survey clearly points out the trouble on Inktomi related sites. If anyone would like a copy just e:mail me. I'd be happy to supply same.
The subject of the thread was "So does this explain it all?"
To answer the question posted as the subject line of the thread, NO, this does not explain it all. I can just see the "Let's pay for listings!" crowd running wild with this anecdote. I've been working on my Inktomi listings for months. I keep up with more than 60 Web sites. That translates into more than 100 keyword searches.
At this time, Inktomi is returning far more relevant results than I have seen in the past year. That is not to say my sites are all in the top ten. I wish they were. The most critical sites are in the top ten, and that's good enough for me...to finally determine that I've never had much chance of getting traffic from Inktomi. Whoever uses their partners to search the Web doesn't use them to find sites like mine.
Perhaps Inktomi was just relying too much on Yahoo! and Hotbot.
If you want to see how relevant their search results are these days, try the following search terms (none of which will bring up any of my own sites -- I want to be as fair about this as possible).
web site promotion
Every search I try, time and time again, comes up with meaningful, spam-free, relevant results.
"relevant" doesn't necessarily mean MY sites will come up in the top ten. I know who I'm competing with for search engine traffic and if I don't see my own sites I see the competitive sites.
I don't know if this is due to link popularity or something else, but the problems reported in the message cited above are either confined to only some specific search terms or have been corrected.
The results I'm seeing with Ink these days are very impressive. This is what a search engine is supposed to be doing for its visitors.
And they did it without requiring people to pay for listings.
As of this writing, two people followed up this message in the other forum. One said he ran the "tires" search and found most of the first ten results were from Looksmart. I compared the Looksmart listings to the listings on www.freeserve.com (the service he suggested) and found absolutely no correlation. So I have no idea of where he got the idea Looksmart was feeding most of the top results.
The other person asked why all the results before a certain listing for "irish steam trains" were irrelevant. I ran that search on Canada.Com and found relevant results (starting with Yahoo! categories).
Copy the description listed at Freeserve, search for it at Looksmart. There are a few below, just click the links.
And so on and so forth.
Must admit to being unable to find position 1's description at Looksmart, maybe it has been updated since Ink raided the directory.
I think it's significant that the Pirelli Tires listing has no title in the Ink listing but has a title in the Looksmart directory.
Agreed. It seems that Ink "borrows" the description verbatim from Looksmart, but the title is spidered off the current page. The ones with no title listed in Ink normally have the url as the title on the page.
Sent you a local email, link at the top of the page.
First, let me say, I believe you Liane, in that I am sure Mr. Tim Gilbert told you all of those things. There is not a question in my mind that you are telling it as it was told to you.
As to your question, "What would be the point of him telling you something that isn't true", that's easy. He is taking the blame off of himself: 'greasing the squeaky wheel.'
If he tells you that he is doing everything he can, and the technical team even protested with a walkout,and the *whole* team is frustrated with the management for not changing their practices, then it seems you both are "on the same side" of the arguement. You both can blame someone else, in this case, Inktomi management. What does that do for him? It gets you off of his back. You, on the other hand, feel comforted that 'at least he is frustrated as well, and he is trying to do all he can.'
It goes back to the old saying "When you can't beat em, join em."
Now, to get back on topic, I think what the IRC conversation from the three guys at Inktomi tells us is that the idea about using pages already in the database with new links to other pages is the way to go to insert new pages since
"addurl is pretty useless to begin with, because we already make a webmap and crawl most of it anyway"
Enough of my babbling.