Fastest way to get a new page indexed

[robzilla]

The recently updated "Fetch as Googlebot" feature in Google Webmaster Tools allows you to submit the URL for crawling as soon as it has been successfully "fetched". I think that's a great way to tell Google a page is either new or updated, but I'm wondering if that's really the quickest way to get it indexed. Frankly, I'm worried that this method will somehow disturb the natural flow of indexing, and consequently might actually delay it. It's an unfounded fear, however.

It seems quite impossible to beat the newspaper giants with my niche site (PR6), in terms of the speed with which fresh pages are indexed, but I do enjoy trying. I have a decent following on Facebook and Twitter, where I generally publish these pages as soon as they go live. The giants have a much greater following, however, and their news stories tend to be syndicated in real-time by hundreds of other sites. All I've got for Googlebot is the site itself, an RSS feed, Facebook and Twitter. If giant A publishes a story half an hour later, it's still likely to be picked up sooner than mine.

Aside from getting more links, how do I compete?

[robzilla]

From Google Webmaster Tools:

When you submit a URL in this way Googlebot will crawl the URL, usually within a day. We’ll then consider it for inclusion in our index.

When you submit a URL manually, there's obviously no linkage data (yet), no source, whereas normally Googlebot finds new pages via links. That's why I fear it might actually slow down the process. Any ideas?

[tedster]

You can update your XML Sitemap and ping Google that there's a new version. This was a correct example of the ping address, last I checked:

http://www.google.com/webmasters/sitemaps/ping?sitemap=http:%3A//www.example.com/sitemap.xml

If your site is a blog, there's also a blog ping service for Google:

http://www.google.com/help/blogsearch/pinging_API.html [google.com]

-----

Here's the big question for me, however -- is your site published in Google News? I think that would be essential for competing with larger newspapers in any effective way.

[seoskunk]

Socially bookmark it, sitemap.xml and links

[mhansen]

I like Pingler for fast indexing. Non-commercial freebie tool.

[robzilla]

is your site published in Google News? I think that would be essential for competing with larger newspapers in any effective way.

You're probably right, but no, it's not published in Google News. I always thought it would not be eligible for inclusion, but now that I've taken a closer look at the requirements, I should probably give the application a shot -- though it might be a longshot, considering that the site's main focus isn't on news.

I thought I'd dive into the access logs to see how long it took Googlebot last time. To my great surprise, it only took 1 minute and 12 seconds before Googlebot came along to fetch the page. For a new page that wasn't a news story (not spread to our followers), it took about 40 minutes. And yet, with the news story, it took at least 4-6 hours for the page to turn up in the search results. Googlebot's not the problem, then. The inclusion of the new page just isn't prioritized, for a lack of authority, I assume. That's not something I can improve on without getting more links and/or a bigger footprint in social media.

The next time I launch a story, I'll use the

tail

command below to time Googlebot, and then occasionally I'll check the search results to see if any of the new efforts (page submitted via GWT, a sitemap, and some social bookmarking) help speeden up the process of inclusion. Hopefully, the more signals Google receives about a new page, the higher it's prioritized. Wishful thinking?

tail -f /var/log/example.com_access_log | grep "Googlebot"

[robzilla]

You're absolutely right. I published a story simultaneously with one of those news giants. They're connected to Google News, obviously, and so their article showed up within minutes. Googlebot came crawling to my story within a minute, but it took at least 2 hours for it to be displayed in the SERPs. The sitemap, social media and URL submission did not help much, so I'll be looking into getting accepted for publishing with Google News. As you say, it's probably the only effective way.

Thankfully, I do outrank most other news sites for the story-related keyphrases -- it just takes too long to get there.