Google not indexing new pages quickly enough

[GloriousMorning]

We have a very large site(strong site ~2000000pages) with tens of thousand of new property pages added daily (we are in the real estate industry) Google is not indexing our new property pages fast enough.
We created a seperate sitemap with new listings and set the frequency to hourly - but doesnt seem to help.
In our regular sitemap the first 10 page are only reporting about 50-60% indexed (eg:30,000 out of 50,000) and when we do random checks many property pages are not in googles index at all.
How can we get Google to crawl and index new property pages quicker? (Others seem to manage..)

[Andy Langton]

I am a bit of a curmudgeon when it comes to sitemaps. Google doesn't really care about your sitemaps. The calculated crawl schedule is what determines how your sitemaps are interpreted. I would suggest looking at existing crawl data. How often does Google request your homepage? How many new pages in results for the last month from a site: search?

If you have 2 million pages of essential content, then either you have serious technical problems, or Google does not regard those pages as requiring frequent indexing. In most cases where large sites don't get crawled enough, the problem is that Google does not view a site as warranting a large allocation of resources.

[GloriousMorning]

Hi
Thanks for your response.
How can I check how many of the page in site:search are new in the last month? We have pages that expire so it's not just adding pages (comparing site:search this month over site:search last month).
I definitely think we have a problem with Google not allocating enough resources to the site - our industry requires that large number- our competitors are zillow/trulia/realtor.com - all have tens of millions of pages...
According to Search console Google is crawling an average of 648,238 pages daily.
How can I check how often Google requests my homepage? (I can see the number of impressions the homepage gets in search analytics report)
Our biggest issue right now is how to get new property pages indexed quickly.
Thanks!

[Andy Langton]

How can I check how often Google requests my homepage?

You can check server logs, which would be the best place to start analysing if you want a good picture of Google's crawl behaviour. Otherwise, a "quick" test is to just check the cache of the homepage, and record the date. If you check periodically you will have a reasonable idea of how often Google is visiting.

How can I check how many of the page in site:search are new in the last month?

You can do a quick test with a date restricted site: search (the parameter tbs=qdr:m would equal the last month):

[google.co.uk...]

For large sites, you can also assess per directory:

[google.co.uk...]

This isn't perfect, but should help you understand behaviour a little better.

To put all that aside, however, it seems likely that you don't have enough links to your site to support that number of pages. Sites with millions of pages that are well-regarded will have tens of thousands of sites linking to them, and that's what is required to rank those amounts of pages.

[tangor]

At no time are you guaranteed a crawl by g (or any search engine). From a technical standpoint they have to manage their resources just like we do, only difference is g is an order of magnitude larger than anything else on the web.

Real Estate is more difficult than other niches in that the content is not evergreen, or even semi-evergreen, thus there's no burning reason to crawl something that won't be there the next month (or so). I do not say that is what they are doing, just being realistic about processing loads and user value of content available at any given time.

[GloriousMorning]

Thanks - yes properties get listed and then sold - but that's really like eCommerce.
As I said - other sites seem to manage to get listed and indexed (we do in some cases too - would like all )and most of our property pages do get indexed but many only after a week which in this industry is too late!

Any ideas how we could get Google to index new pages quickly? What we could do to limit the resources so google sees the new pages (we can't cache too long as changes are made all the time..)

We do have a lot of links (maybe not enough) but not many to the new pages - although there are some properties that people have to credit us with the listings so we get lots of links and even then some of our competitors who are giving us credit are getting indexed before we do..

[Marketingdude]

For the pages that we really want indexed, it's as simple as pinging the page or adding a link to another page that's already indexed. This assumes you want to get some properties indexed over others (but it doesn't sound like it).

[GloriousMorning]

Does linking to indexed pages really help? Or do you need the indexed pages to link to the new page?
We added breadcrumbs which links to pages already indexed.. hasnt helped so far...
Pinging the page - how? We tried submitting some to Search console - most got indexed straight away but we can't do that for so many pages..

[lucy24]

Another thing to consider: Do humans arrive at your not-yet-indexed pages by other means? Once you've got yourself established as a go-to site for whatever your business is, indexing individual pages may not even matter. Humans will type in (or bookmark) yoursite dot com in the way that they type in amazon dot com or wikipedia dot org, and then use your site's internal site search to find what they're looking for.

Er... you do have an excellent internal search, right? With millions of pages, you absolutely need something of your own. Don't rely on search-engine crawling and site: operators; you know your own site and your own customers better than anyone.

I don't know about other humans, but if I already know what site I want, I'll always use their own search rather than Google. The only exception is if I know from past experience the site's internal search is hopeless. Naming no names.

[tangor]

Your home page and category (community, city, etc) are what need to rank the highest in the search engines and then your internal linking will get your customer where they want to go.

Meanwhile, you can try "fetch as google" to see if that speeds things up. Some say it does.