Wonder why your blog posts are not indexing these two days?

[miozio]

If you've been having problems with your new blog posts here is the huge discussion: [google.com...]
I've posted around 20 new posts on my WP blog since the 18th, none of them were indexed... What about you guys?

[dataguy]

frank72: Just wait. It's coming.

[incrediBILL]

Three hours ago i published two articles but nothing...they are nowwhere!

What should i do?

Were they crawled yet?

If they haven't been crawled they won't get in the index.

I've had a couple of people report to me that pages not in the index haven't been crawled.

Today I checked a couple of new posts, not crawled yet, the most recent was 2 hours ago, the oldest is 6 hours old, Google has crawled neither.

Google crawled and indexed my table of contents, and 2400 other old pages, and I can locate the titles of the new content in Google Search any time it's referenced on a pre-existing page.

All content from yesterday is both crawled and indexed properly.

It appears there's no problem getting content indexed once it gets crawled, the problem it just getting it crawled.

There's nothing you can do except wait.

[Robert Charlton]

Status in the Google support discussion... [google.com...]

Posted by one of the thread participants prior to the JohnMu post below...

the new posts are getting indexed almost instantly BUT the ones published in the time range between 20th and 22nd are still not indexed.

Posted by Google's JohnMu on 10/24/10, 6:31am... my emphasis...

This issue should be resolved now. It might take a few more days for everything to catch up, but new content should be getting indexed at the same rate as before now.

Should you continue to see problems with crawling, indexing or ranking, then these are likely unrelated to this issue. In that case, I would recommend starting a thread here in the forum, mentioning your site's URL, any changes you have recently made and detailing the issues that you're seeing. Feel free to link to the thread from here, so that we can take a quick look to double-check.

It's not clear from Google's time-stamps what time zone they reference. JohnMu, in what I've seen, and I haven't read the whole support thread, apparently doesn't indicate specifically whether it was a crawling or indexing problem.

[incrediBILL]

JohnMu, in what I've seen, and I haven't read the whole support thread, apparently doesn't indicate specifically whether it was a crawling or indexing problem.

If you don't crawl new content, which they aren't, you can't index new content!

Several others have shared with me that there is no crawl showing in their logs as well.

So best I can tell they're still at least a day behind because my new content was being indexed in about 30 minutes worse case before this happened.

[Robert Charlton]

Bill, thanks. It sounds like you are observing that Google in fact is still not crawling new content as it should be, though JohnMu's comment seems to suggest that part of the problem was fixed. Keep in mind that he is in Switzerland and on a different schedule than we are.

I mention this because some members don't have access to server logs... they are depending on Google's Webmaster Tools... and if they are not seeing new content appear, they may think it's their site rather than a Google crawl problem ongoing.

[Future]

It seems this problem was not faced by majority of sites ?

[incrediBILL]

It seems this problem was not faced by majority of sites ?

I think it was very widespread, it was even impacting WebmasterWorld front page stories that I was monitoring that weren't being indexed, among many other sites.

I think Google had a major boo-boo and I'm shocked that they don't have internal monitors and alarms that go off when the new content crawl drops below an unnatural level so they can check out the problem before everyone starts to complain.

If they do have those monitor alarms, then they sure took their sweet time reporting and resolving it.

If they knew, why even bother hiding it?

It's not like the rest of the world wouldn't notice within 2 hours of the crawl not happening because of the index not updating.

More transparency.

If Google would just tell what they knew when they knew it there would be a lot less anxious people in a panic that something was wrong with just *their* site.

[dataguy]

I'm surprised that there are still reports of this still being a problem. The problem was resolved for my site 2 days ago.

I believe to understand this problem, you can't think of it as an on/off switch that someone at Google accidentally switched off. If it was, certainly Google would have noticed it right away and flipped it back on.

I think it is much more likely this was the result of an algo change. Google saw that things were different, but they were expecting things to be different. Not all sites were negatively affected. I'd suspect that there was just as many sites that were positively affected, and by only looking at the numbers Google couldn't tell that some sites were negatively affected that should not have been.

It hasn't been that long since WW had a thread on how to get Google to index fresh content. We need to keep these things in mind. When I noticed my fresh content was not being indexed, I spent 3 days tightening things up, which may be why I've seen the issue subside faster than others.

[incrediBILL]

Googlebot picked up new content only 4 hours after publishing today and 2 hours later it's still not indexed. More recent content has also been crawled from 2 hours ago, and of course it's not in the index either.

I think it is much more likely this was the result of an algo change.

I seriously doubt it's an algo change, no reason for any speculation at this time.

Google confirmed it was a bug, that once confirmed fixed still put them way behind with a massive backlog, probably further behind than even they suspected.

It's slowly catching up, but I think there's a mountain of new content slowing them down.

Maybe it'll be back to normal by the end of the week, hard to say.

[CainIV]

Multi-billion dollar improvement to architecture, gone wrong.

This has to be embarrassing, and they will want to fix this fast before too many posts hit the news, people love to talk about this type of stuff.

I am seeing content that was previously cached (and should be) before the rollouts and changes now getting re-indexed again, and new content is (slowly) beginning to get indexed faster on sites I work with.

[mhansen]

Just thought I would jump into this convo and mention that I found a weird thing in WMT's today, when using "Fetch as Googlebot" on a newer website, versus an older, well established site.

I also started a post over in the tools forum on this to see if others see the same thing. [webmasterworld.com...]

- New Site is -2 months old, but well indexed.
- WordPress backend on my own domain, XML sitemap
- Published new article, pinged.
- Went into GWT and used the fetch-as-googlebot feature to see what was returned.
- Article still not indexed.
- Using the site:command.tld, and sorting by past 24 hours, shows no results.

Right now, more than an hour later, the status of the fetch-as-gbot in GWT is still "Pending". I have used the fetch-as-gbot as recent as 10/16 with no problems and nearly-instant results on this same site.

Update: The fetch-as tool worked right after I posted this. It took roughly an hour from the time requested the fetch.

Then I went to an established site (+5 years old) and added a new article.

- WordPress backend on my domain and server, xml sitemap, etc
- Very well indexed site
- Published a new article, pinged
- Article indexed within minutes
- Went to Fetch-as-gbot in WMT and the tool worked flawlessly, showed result within seconds.
- Using site:domain.tld, sort by 24 hours showed the article and a cached copy in serps.

I would be interested to know if others are seeing anything similar in GWT using the fetch-as tool in labs.

MH

[incrediBILL]

I think it's almost back to normal, Googlebot crawled only 2 hours after new content was posted today, I'm already seeing same day content in the index.

Whew! Nice to see the status quo return!

[socialwebexpert]

we are having the same problem on our site too
The site as only fresh and unique content and its been in google news since 2 years and now its only indexing 20% of my post

the site is in wordpres - can this be the problems?

any help please? we are extremely dropping traffic and we are having issues with our writers as not money are coming in from advertisment :-(