Buddy Links update June - 4 - 2000.
This update is also available online with clickable links at: http://www.searchengineworld.com/links/update6-4-2000.htm
We just blew right through 600, 700, and now knocking on the door of 900 sites!
Is that radical or what!
Wow, we spent from October of last year until January struggling to get to 200 sites, and now we add 350 in the last 45 days. Nothing breeds success like success!
As I write this Sunday afternoon, the little Buddy Links spiders are busy downloading your links pages. After they get through, I will go through the sites that failed removing those that did not do the last refresh or are otherwise failed. Numbers on the main BL page may fluctuate for the next few days as sites are added/removed.
Hi New People!
I send out these little updates once every couple of weeks to give everyone the short story of what's going on with the program.
Big thank you to all those who've joined us since the last refresh. New sites may or may not have gotten a validation confirmation email. To make sure, you can login to your site and have a quick look at the 'status' field.
I started sending out validation emails now, but a few of the mid May sites got skipped. We were mentioned in several popular search engine newsletters last month which resulted in three large crushes of new sites added in mid May that I didn't send out emails.
If you new folks are wanting more info, you can get a touch of a historical perspective of Buddy Links by reading the back posts in the Buddy Links subboard in the forum. There are also a handful of these updates cataloged on the Buddy Links signin page.
In the middle: thank you to those people who've been with us since last year. All those refreshes and hard work is starting to pay off. I do tend to take a bit more notice to you older sites who did all the work in the early days.
I'd originally planned to do a refresh this week or next - that is now in a holding pattern until things stabilize with the directory, search engine, and category editors. We've added so many sites recently that more work is needed NOW on the direction of the program.
We are at that point where many sites have expressed concern over the total number of link pages required. With 12 sites in the program myself - I feel your pain.
Even since the last one of these updates, I didn't feel we were ready to make any major decisions about the direction of the program. With the onslaught of new sites, the time is now for fundamental decisions to be made.
1) allowing sites to signup for a specific category, then only receiving links to/from those sites.
Pluses: cuts down on the number of links pages and raises the relevance of the links received. This will help on context sensitive engines such as Google and possibly on Inktomi.
Minuses: cuts down on the raw number of links available to a site. It would also mean some sites would freak out when they realize they just primarily linked to some of their competition through this program. There is one other program on the net that is using a similar system to this idea, but it brings a host of new problems.
From a management standpoint it sounds easy, but it really raises the depth and complexity of the underlying programs two fold.
2) Allowing sites to specify a 'cut off' point at the total number of sites they want to link to.
Pluses: limits the amount of work a site has to do.
Minuses: cuts down on the raw links available to all sites.
From a management standpoint: extremely difficult as we'd have to run shadow databases tracking which link had gone where and tracking numbers.
3) Removal of sites not found in search engines.
Pluses: raises the quality of site.
From a management: attractive and easy.
4) ??? your ideas here.
We started a few threads on the above in the forum. Feedback on the above is encouraged and welcome. Tossing in your own ideas are even better. I'd prefer it in the forum just to see what others thought of the proposals.
Please submit your Buddy Links pages to search engines. It is more critical than ever (see #3 above).
Because of all the new additions, adding category editors, and general work on the database - the system will be offline more than usual in the coming weeks.
Last Chance to Edit Your Site?
With the coming addition of category editors, the ability to move your sites category and possibly even edit it, will be going away. Now, is the time to update your site if you wish.
Self Promotion: New Stuff:
I've built a linkage checker for you to see how many people are linking with you. It checks Alta, Raging, Hotbot, Infoseek, Google, and MSN. It features the ability to track your link numbers via email updates by week or month. Very handy for Buddy Link member sites.
I also recently added a search engine rankings checker, an online version of the good old Weblint, and updated the Keyword Density Analyzer and Spider Simulator programs.
The high linked site as a result of this program is now at 395 links in Altavista. I feel we are running at about 60% of the links being indexed. It will take some time for that last refresh to filter through and for new sites to get their links pages indexed.
After talking with other link programs, they were actually impressed we were running that high. Of course the recent rush of sites will take some time to get submitted.
Submitting Sites to Buddy Links:
The only way a site can be added to Buddy Links is via the standard submission form on Search Engine World off the main Buddy Links page. Although I've done it on special circumstances, it doesn't matter whether you have 1 site or 50, submissions attempted through email are ignored.
The new submission form is working excellent. It is 12 steps and each step must be filled out properly along the way. Some people have mentioned that they can't back up to change a form value - that is your browsers fault, not our systems. If you have some security setting on your browser set at 'paranoid' level, - it can cause you to refill out the previous values when you attempt to back up.
I know, I hate long online forms too, but I feel it is paramount that sites get signed up properly. One of the major headaches in the past was cleaning up bad submissions. So, we went to this 12 step method that explains each step in detail.
The stumper in all this is still Inktomi. In an effort to get us 'in the door' and stay in the door at Inktomi, I've taken to some radical measures. I have cloaked pages on about a dozen servers pointing at nothing but your links pages. This should give the links pages some relevance. I am also going to start including one link on every links page that points at other peoples links pages.
1) I request that your links pages have the links start 'above the fold' in the first 400 vertical pixels.
2) Links pages may or may not include meta tags. That is optional, but I would prefer IF YOU DO add metatags, please make them unique for each page. I still think the best way is to not include keyword tags or a description.
3) Please put a site footer and header on the page. Something that LOOKS like the rest of your site. I am considering removing sites that put up bare pages of links.
We must stay unique. We must stay unique. We must stay unique.
If you feel this free program is of value, please consider stopping into the forum and participating in the search engine and buddy links messages. Thank you to the regulars who've been pitching with topics and conversations in the forum - it really has lightened the email load.
Again, I can not address all concerns via email - it just isn't humanly possible for me to get to all the email in a day, let alone a week or month. I do try to answer those of critical importance. General questions should go to the forum. No I'm not pushing the forum to try an make that successful - I'm pushing the forum to get your questions and comments answered and support the membership base. It is advertising free and will remain so.
Ever hear that old philosophers question, "What color is the blue cup"?
A cup colored blue absorbs every color of the spectrum but blue - it reflects or /rejects blue light. Thus; the only thing we know for sure is that the cup isn't blue!
Welcome to the age of the search engine Word Guru. Did you know that every major search engine has high level semantic, syntax, phonetic, and linguistic experts on staff? Inktomi has several with doctorates in English and various languages. These 'word gurus' set around and study web pages pages and search terms.
This type of research has led to many of the 'natural language' features search engines have been adding. It has also led to some very complex studies of the web itself. There are some very interesting threads on this in the forum right now. See the one post with links leading to the W9 conference and 'term vectors'. It also leads nicely to the next topic:
Keyword Surveillance Tool
One thing that I've always relied on is picking good keywords and keyword combinations. For about two years, I've run a program that recorded search engine searches via the online 'spy' pages like the MetaCrawler Metaspy.
Last week, I put the program online. It is in its initial stages at:
Currently we generate a daily, weekly, monthly, and yearly keyword reports. There is also a 'live' report that is available online each day. You can watch it roll through the keywords during the day. Watching a term rise or fall during the day is fascinating as different types of users get on and off the net.
For instance, Pokemon was stuck at about 100 during the week. Come saturday morning it skyrocketed to #1 for part of the day (saturday morning kids). Where else can you find that info?
We are tracking 5 major search engines and 2 minor ones. The minor ones only add about 3-5% to the total keywords seen. It breaks down approximately:
Webcrawler:25-29%, Other:20-26%, Metacrawler:10-15%, Lycos:10-15%, Excite:7-10%, Magellan:2-5%, Savvy:2-4%
'Other' is a top 10 major search engine I don't want to expose. If you had a back door to a major search engine search log file, would you? Please don't ask for the non-standard urls.
The percentages float a bit since a couple of the engines are dog slow during the day and I reduce the rate of pulling on them during peak loads.
That gives a nice cross section of searches. Our methodology is to pull a search log about twice a minute. We've found that more often than that and you risk getting duplicate data (like some other keyword programs do).
I've also put together a keyword resources page at: http://www.searchengineworld.com/spy/resources.htm
5000+ estimated links pages on net.
876 sites listed in program.
815 validated (about to change in a major way).
54 banned or perm bad (mostly adult sites trying to slip in).
485 sites in the last refresh (April 29, with +-20 sites).
395 links to the high linked site (60% of the last refresh).
599 unique emails (indication of how many unique sites are involved).
...Either we are alone in the universe or we are not; either idea is mind-boggling.