Inktomi Slurp Not Indexing

[mark1615]

I have had a strang experience recently with Y. Let me begin by saying that our other sites enjoy great success with Y. We are working on a new site. We were rankied at 640 for our main KW and and in 100s for secondary KWs. Noticed a few rookie mistakes with onpage optimization and changed them. Jumped 600+ spaces for main KW in two days and 100+ for the secondaries. Basically we were dialing it in. So we let it settle for a few days and then went back and made two really minor tweaks to the index page. The next day two things happened:

1) We dropped back down to our SERPs.
2) All of the robots activity dropped. Y was using 1-1.5M of bandwidth everyday and 125 hits and G was using around 3M and 350 hits. This dropped to 15K and 7 hits and 150k and 25-30 hits.

Weird. Coincidence? I really need some opinions here.

[grandpa]

What were the two really minor tweaks?

<diatribe>
I don't know if it's just me - some bad karma - maybe the weather... but in past few days when I've edited some really simple things on my page, everything goes nuts.

This morning I spent 2 hours looking for a missing semi-colon in a Javascript routine. I was the third time thru my code that I found it, because I knew I didn't change anything in that part of the page. But there it was - OR WASN'T - big as daylight. As soon as I inserted the semi-colon the page functioned as advertised.
So, just before it quit working I did 2 minor changes. I did a search and replace for 2 things, \n and \t. All I was trying to do was to remove some bloat, the page is about tweaked out and works really nice. Somehow, that semi-colon went missing too. And it only caused a problem deep into the page.. not something that would be noticed without going thru the entire page process.
</diatribe>
I had a similar problem a few days earlier, No need to go into details.

My point is to cross the T's and dot the I's, check your validataion, and cross your fingers. All you can do is make sure your part is right.

As far as the movement you saw, one might speculate that it's normal. Maybe the changes that influenced the serps just weren't enough to keep you there.

Just in case you haven't seen a reference to this page [webmasterworld.com] yet, it is arguably valuable reading.

<afterthought>
You might have a look at your Server Headers [webmasterworld.com] too. Here a link to W3C Markup Validation [validator.w3.org].
</afterthought>

[mark1615]

Thanks. G is starting to pick its activity a little bit, but Y and MSN haven't been around for 36 hours which is very unusual for them. More unusual in that that Mon-Tues they came but only had like 10 hits each and then just stopped.

We dropped from #1-#2 in the MSN preview to totally out of site and back to #60 this morning. On Y, last week we were #640 for the same major term, made an onpage tweak and went to #21-#40 for three days. Then on Monday dropped back to 640 at the same time the bots started behaving strangely. And this morning we moved to #610 on Y. It's just strange.

The main question for me is, is there something we are doing that is inhibiting the bots. Maybe the SERPs and the bots issue are unrelated but that would be a weird coincidence.

[steveb]

Yahoo has at least two indexs. I have a page ranking #42 on one and #200+ in the other based on two different caches.

A cattleprod could't motivate Slurp. Not much you can do about it except take up a collection to buy Yahoo an adequate bot.

[RFranzen]

Yahoo has at least two indexs. I have a page ranking #42 on one and #200+ in the other based on two different caches.

steveb,

Could you comment in the other thread [webmasterworld.com] where I speculate how the two different cache's might function? I can't tell whether your experience is the same as mine, but I can repeatedly demonstrate the presence of two caches. (PM me if you are interested in the actual case which shows the cache is selected deterministically, not randomly.)

Mark,

As you try to figure out the weirdness, take a look at the cache'd page which is being used at that particular time. My guess is that your site's position is dependent on the cache contents, and that the cache used is dependent on multiple factors, including search phrase. And I sure hope that minor page tweaks won't shift position by dozens of places.

-- Rich

[mark1615]

steveb,

I guess the thing that strikes me is that the other bots backed off simultaneously (M & G). On 11/1 I started getting around 5% of the hits that I have consistently gotten from Y, M, and G. G has started to pick back up but I am still at only about 20% of normal. I hear what you are saying about Slurp - slow would be a compliment. My biggest concern is that I did something.

As to the cache issue, I am still showing the page as of 10/31 after the tweak was made. It has been changed since.

[mark1615]

Rfranzen

Your comment on the cached pages was great. Are the SERPS dependent on the cached page and not the actual page? Let me go over my time line again:

10 days ago - SERP 640 - made some on page changes.

7 days ago - SERP 21-40

Sunday - SERP still holding in the 20s. Made a little tweak.

Monday - SERP 640.

I just checked the cached pages per your suggestion and it is the page from 10+ days ago!

So again, are the SERPs based on the cache? And if so, was our jump them just a foreshadowing of our SERP when the cache is ultimately updated?

[RFranzen]

Mark,

I replied via PM and the other thread [webmasterworld.com].

-- Rich

[webmktg]

Most of our websites lost rankings in Yahoo today. Most of our sites were on the first page for Main and hihgly competitive search terms. But suddenly we lost everything. Has anyone else experience this?

[Kennyh]

Ouch! I've just seen 80% of my pages disappear from Yahoo, including all of my most profitable ones. Many were ranked 1-5 and provided very healthy traffic. :-(

[mark1615]

On some of my other sites I saw movement this morning for our big important terms, but nothing drastic. I had #3 go to #4, a #2 to a #3 - the worst was a #6 to a #8. Not sure yet why they did it. These are all pages that have been solid in their positions for a long time.

Has anyone analyzed it yet to see what seemed to rank better?

[submitx]

mark1615: maybe Yahoo is done indexing your site for this month and will come back next month again. Yahoo and Google both can detect sites that have fresh daily content and index those more frequesntly. If your site's content does not change that often, then maybe it will make sense that they are done indexing you for this month and we'll come back again in a month or so.

[mark1615]

Thanks. All of the SEs now seem to be making their way back and taking more and more hits everyday. It is very strange.