Very fast indexing on a specific page

What does all this mean about my site, if anything?

[Asia_Expat]

My PR4 18 month old website usually gets new content indexed in 2 to 4 days. Sometimes straight into supplemental, sometimes sticks... variable results basically.
My forum is well optimised (although threads with multiple pages have the same title for each page).

My website is geographically themed and focuses on a specific set of countries. A national of one of those countries is being discussed by a British tabloid newspaper in the last couple of days, including their online version and so I decided to start a forum thread about that person for my members to chat about the topic. I noticed that there was very little information about that person on the net so figured it's be good search engine fodder

I was amazed to see the thread indexed by Google and showing in the results just a few hours after it was started. Not only that, my forum thread was in the number 1 position for every combination of keyword about the celebrity being discussed, even beating this major British tabloid newspaper. Alongside the reult was the caption '6 hours ago'

It's not beginning me loads of traffic, probably only an extra 40 or 50 uniques per day but I'm curious about this. What does it mean? How did it get indexed so quickly? Is this a sign that my website is becoming authority in some way?

I was accepted into the Yahoo directory a couple of days ago but the listing is not yet live, so I don't think that is relevant. Certainly Google seems to be good at picking up on bursty search terms that have little info on the net and indexing relevant content quickly, even in my relatively small niche.

[tedster]

It certainly is a healthy sign for a site to have new urls showing quickly in the index. Has spidering increased in frequency for your site?

You raise the idea of how Google works with bursty terms. I can see that, if a term begins to show activity, the Google crawl team might increase indexing for any urls that they already have indexed with related terminology to the bursty terms - the high co-occurence terms that they already have measured. This could lead to rapid discovery of new pages linked from those urls they already know about.

[Asia_Expat]

I just checked and I see no change in crawl rate. I also see no change in overall referrals and they remain as constant as they have been for the last 9 or 10 months, except for this single topic that has spiked an extra 50 or so referrals/day.

Slightly off topic... but I mentioned the Yahoo Directory above and I now see my website in the listings and I'm delighted with the category they've placed me in, just 3 directories deep (most of my competitors are buried 6 or 7 directories deep) and it tells me I'm doing something right. I just hope it'll be reflected in my traffic soon.

Actually I recall seeing Google bot on the thread in question just minutes after I posted the topic and that's why I was monitoring it (the forum software is set to to list the bots as guest users to me) and new topics are automatically listed in a 'new topics' section of the root domain, so I'm sure this is why the topic was noticed so quickly... but for sure I've never seen it show in the results so fast before.

[dataguy]

I have several custom reports which show me pages that have been added to my site in the previous 7 days which have received visitors coming from Google search. About 2 weeks ago I started noticing that pages were receiving visitors faster... where pages used to start seeing visitors from G in a day or two, it's now often 2-6 hours.

We have been working for years to get up to the minute current events commentary on our site. We are hoping that G has started to recognize some of the fast-breaking commentary and is now indexing it faster.

I hadn't heard any reports of anyone else seeing this, so I thought it was something specific to my site and not a universal change.

Anyone else seeing faster indexing?

[kidder]

Yeah we see it all the time and it seems to vary from day to day. Sometimes pages pop in a matter of hours. You can also drive them a little faster if you use bookmarking.

[Robert Charlton]

I can see that, if a term begins to show activity, the Google crawl team might increase indexing for any urls that they already have indexed with related terminology to the bursty terms - the high co-occurence terms that they already have measured.

This idea is extremely intriguing to me, and I wouldn't rule out that Google might have been crawling topical sites more frequently because of interest in the news.

The timing of this particular example, though, suggests that coincidence must have been a big factor...

I recall seeing Google bot on the thread in question just minutes after I posted the topic

This seems too quick for there to have been any kind of cause and effect, and it strongly suggests to me that Googlebot's presence on this specific thread at this specific time was coincidental... as if the post just happened to hit the right time in the crawl cycle.

Is there any crawl cycle data to suggest that anything about the crawl cycle was unusual?

[Asia_Expat]

Nothing that I can see but this is no coincidence... I have now seen it with another thread... I only just noticed but I'm pretty sure it was index just hours after the thread was started...

www.mywebsite.com/forum/index.php?showtopic=1122334455 - 23 hours ago

... something has shifted in the way my site is being crawled but I don't know what and I don't know why. New forum topics are showing in the results very quick now. I am seeing no change in traffic though.

[Robert Charlton]

Asia_Expat - If there's no change in crawl pattern, it's hard to know what else to call it except coincidence... particularly the "just minutes" example... unless Google's monitoring your keystrokes. ;)

[Asia_Expat]

I webmastertools, the graph tracking the number of pages crawled had spiked in the last few day (more than doubled).
What does this mean?

[dataguy]

I created a few new reports to try to figure out if anything has changed with the crawl patterns on my site.

I don't show any increase in the daily number of pages crawled, but typically I'll have 2 or 3 pages a day get picked up and start receiving traffic within 2 hours of the page coming into existance. Occasionally I'll show some pages getting crawled in less than 10 minutes and some receive traffic in less than 75 minutes.

I've also noticed that many of these pages drop in their rankings after about a week. Seems to me that this could be the result of the "Query Deserves Freshness" shim that was talked about here a few months ago. Not all the pages that receive nearly instant traffic are late breaking news commentary, but I do see new current events types of information getting picked up quickly... for example, Thanksgiving Day information added this week is doing very well, better than the same type of information that has been on the site for years.

What does it all mean? I don't know...

[kamikaze Optimizer]

Asia_Expat: I have that happening on one of my sites. Several of the articles that we posted today got picked up in minutes and are performing much better then expected in the serps.

G-Bot is very active today.

I could'nt be happier about it.

[Asia_Expat]

There is absolutely no doubt whatsoever now that this is going on. I just noticed a new thread get indexed and show in the results less than an hour after it was first posted!

www.mywebsite.com/forum/index.php?showtopic=12345 - 8 minutes ago

It's certainly not happening with every thread and appears to be randomly occurring. I wonder if this is because I am on the cusp of better ranking, or if this is a very clever algo quickly indexing niche search terms.
BTW these threads that index quickly rank very well in the first or second position for the target search terms.

Still no noticeable change in crawl activity. Still no overall change in traffic.

[tedster]

It will be good to know how long those good rankings last - in other words, are they related to the burstiness of the terms, or to the news cycle, or do they just stay indefinitely.

[Asia_Expat]

The thread I mentioned in the original post has now gone supplemental and did so about a week ago. It does not rank at all now.

[Asia_Expat]

... and I can't believe I didn't notice this before!... there is no cache of these pages that are quickly spidered and ranked.

[Miamacs]

Isn't this a part of Universal Search? You know, the original meaning of One box, that they look for relevant data in many ways ( web search being only one ) and mash it up?

...

I mean this sounds like as if the forum was considered some other kind of source than a 'web' result, a kind of which would fit into a vertical like news, blogs, ( forums? there's no such vertical, is there? ) and treated accordingly:

Being crawled, ( not ) cached, quickly indexed and ( not ) clicked in the same fashion as... for eg. universal search news results? Then they're dropped once the news isn't news anymore.

My guess is that while it's the same bot, it's not the same vertical. And that the site is basically indexed twice ( or 'in two different ways' ) for plain old web results and this... news... thing. With the second being less popular among searchers, and quite temporary.