Googlebot Ignoring a New Site, in spite of good backlinks

[DXL]

I launched a site several weeks ago, and linked to it from several PR4 related pages. Yahoo indexed the site over a week ago, but according to the site statistics, Googlebot has made only one appearance 10 days ago.

There's no robots.txt file, or tag related to robots in the site's coding (same as similar sites I've launched), nothing that would tell Googlebot to go home and not come back. So why the infrequent spidering? Other sites I've launched recently have been indexed in a matter of 1-2 weeks, why is Google ignoring this one?

[tedster]

In my experience, Google spidering happens on a variable schedule, rather than some kind of predictable time scale. In other words, during some periods Google dedicates more resources to spidering than at other periods.

Did the spidering of your home page result in that page showing up in a site: operator search? If so, the spider dance has begun.

[AnkitMaheshwari]

I also believe that the Google spidering is slow from past 1 week as mostly whenever we make any change on the site or add new pages, Google immediately spiders them with a span of 24 hours, however, we added around 3K pages last week and only 1 is cached till date...

[DXL]

Did the spidering of your home page result in that page showing up in a site: operator search? If so, the spider dance has begun.

I don't know what "site: operator search" is. Since most of my sites are indexed within a week, I generally don't do any special searches on Google. I also do the occasionally linkdomain inbound link lookup on Yahoo, but don't know what the equivalent is on Google.

[MLHmptn]

Google site operator = site:www.yoursite.com

I also do the occasionally linkdomain inbound link lookup on Yahoo, but don't know what the equivalent is on Google.

link:www.yoursite.com

[edited by: MLHmptn at 11:41 pm (utc) on May 25, 2009]

[Robert Charlton]

...what "site: operator search" is...

The site: operator allows you to limit your search to a specified domain. The syntax is:

site:example.com searchterm

There should be no space between the colon and your domain name.

If you leave out the searchterm, site:example.com by itself should return all pages for the example.com that are indexed in Google.

Note that indexed by Google is not the same as ranking in Google.

Also, note that the site: operator provides an approximate indication only of how many pages for a given domain Google has indexed. Results for the site: operator are often erratic. This approximation has been a great concern to webmasters who watch the number of indexed pages closely.

To get an idea of how much of a concern, do a site search of WebmasterWorld, entering this in your Google search box...

site:webmasterworld.com google site: operator

You can also use the WebmasterWorld site search (see top menu), in which case you don't need to include "site:webmasterworld.com".

There's further discussion of the site: operator in our Hot Topics [webmasterworld.com] section, pinned to the top of the Google Search forum home page, which is always a good place to look for core topics we discuss here.

[Robert Charlton]

A PS to the above...

There's also a section in Hot Topics about the Google link: operator, which you should read.

I prefer Yahoo's Site Explorer to Google's link: operator.

Google's link: operator returns only a small and randomly selected sampling of a domain's backlinks. Yahoo's Site Explorer is where Yahoo redirects its linkdomain: searches, so you might as well just use it in the first place. If anything, Yahoo returns too many backlinks for a domain, including tracking strings from ads, etc.

Very occasionally, you will see a backlink on Google that Yahoo hasn't picked up, because their spidering patterns are different.