Forum Moderators: Robert Charlton & goodroi
It's not just a matter of not being in the index. I can't find googlebot in the logs anywhere. No snips on the robots.txt, nothing at all.
The site has links pointing to it.
The site is all over adwords.
Other spiders have been crawling and indexing just fine.
The domain was parked between 2003 and early 2005. Besides that, I can't find any evidence that anything dodgy happened with the domain.
I've acquired domains in all sorts of situations, and I've never seen one that was completely ignored. Maybe I've just been lucky.
I've had the domain since 2006.
Is it possible that between 2005-2006, someone might have used noindex or disallowed the site in robots.txt? If so, wouldn't googlebot still come recheck the robots.txt or index page?
I've seen talk before of an expired domain filter. Does googlebot think this domain is poison? When I see the expired domain issue talked about, I always thought it was a "no index" issue, and not a "no crawl" issue. Was I wrong about that? Anyone have domains that are absolutely not touched?
This also makes me wonder how googlebot works. Does it crawl the web, gather potential links to crawl, then only crawl those links later after it filters them with a no-crawl list?
It looks like the bot looked for robots.txt and then did a header check on the index page. Nothing since.
Incoming links are actually pretty good for this site. Meanwhile, I have another domain that googlebot is all over, with only one low value incoming link. The domain that doesn't get spidered probably has 15 unsolicited (and unreciprocated) links at this point.
In the past, visiting a new site with a toolbar installed, or one incoming link would get some spider action.
I've never contacted a search engine about anything, unless I was paying them. Even then, almost never. I've bought some rather muddy domains before, too. Those were always spidered, at least.
I guess I don't have much of a choice anymore. I've done about 300 hours of work on this site, and it's a pretty nice domain too. At least since I've owned it, there's not any reason why it should be totally snubbed.