I'm trying to get a client's site ranked in google but when I search for him nothing comes up and searching his url doesn't immediately rank him highly.
In fact, google displays this site, but why does it show no meta description or page title?
I originally assumed that google may have penalised him for something but I can't find out what! I've checked with bad-neighbourhood.com, looked at his site with lynx to see if I can find any cloaking - nothing. I have even used the lynx browser and changed the user agent to Googlebot/2.1 to see if I got redirected and nothing.
Is there something that I am missing?
[edited by: tedster at 8:35 pm (utc) on May 24, 2010]
[edit reason] no personal URLs or seerach terms, please [/edit]
I know, I just can't see what!
Oh and I should also mention that the site also had this in the robots file:
User agent: *
But that got changed at least a month ago to:
User agent: *
Does anyone know of any other techniques or tools that I can test to locate the problem?
Do the logs show the googlebot through in the last month?
It seems unlikely, as a month is a long time in crawl time, but it would explain the lack of caching.
Something strange is going off.. Doesnt show anything from any pages, just URL's?
I'd start from scratch, remove robots file and any custom stuff (theme etc) and put everything back to the default..
THEN see if google re-caches your site and includes content..
If not then im stuck!
[edited by: tedster at 8:37 pm (utc) on May 24, 2010]
[edit reason] remove specific domain name search [/edit]
When Google (Yahoo or Bing, too) shows only URLs in the search results, this does mean that they haven't spidered or indexed the URL -- the robots.txt problem you noted can do it, or robots meta tag nofollow, or other kinds of search engine blocks at the server level. At any rate, when you see this kind of SERP, it means that the search engines only know about the URLs existence because of links pointing to it.
Four weeks is a relatively long time to wait after fixing that error, but it's not unheard of. what I'd suggest is setting up Webmaster Tools accounts at Google and Bing, and a Yahoo SiteExplorer account. This allows you to verify the site with each of the search engines. In addition, it gives yu a way to understand how they currently see the website.
One excellent tool that Google offers is "fetch as googlebot" - that can uncover problems in present time. In addition, you will see records of past crawl problems and a host of other data.
I'll will look into it, with your suggestions! :)