Site indexed by G but not ranking

[goodbyedee]

Hello all,

I'm facing a rather strange situation and wonder if anyone has experienced the same situation before.

Data:
- The keyword is competitive but not too much. From the top 10 ranking pages only 3 or 4 have spend some time on SEO
- Our site has been optimized about 3 months ago (on site optimization and link building)
- The site uses rewrite rule (with 301 redirect) in order to make seo friendly urls
- The site is available in 2 languages, english and greek. The keywords we target are just the greek ones.
- When user types the domain http://example.com he redirects to http://example.com/en/welcome.html . - For our main keyword we wish Google to index and rank http://example.com/gr/welcome.html
- The site has a lot of unique content.
- The domain is about 18 months old

Problem:
- The site ranks very poorly (Obviously), but what makes me see this case as unique is that:

1. Google does not display the (greek)home page in the results, but instead it displays services.htm, faq.htm or contact.htm
2. Even worse, for the main keyword it displays faq.htm in the 3rd page and services.htm in the 5th page!
3. Trying the site:example.com - "main keyword" in google.com, the home page (which is the one i wish to rank first) displays without the keyword in the title being bold, although the title includes exactly the same keyword. It is like Google doesn't recognize the characters in the title of the home page. Moreover, instead of faq.htm the first result is services.htm.
4. When i search the greek keyword in google.com, the english page ranks top 5. But when i search the same keyword in google.gr i get the results described in 1.
5. Speaking from my experience and taking into consideration the keyword competitiveness and the competitor's sites our site should definitely rank higher. At first i thought that this would be an easy one...

Sorry for the long post... :-\

[tedster]

It seems to me that a redirect from the domain root is a poor idea - and it probably is at least part of your problem. See this thread for advice straight from a Google spokesperson: [webmasterworld.com...]

[goodbyedee]

Yes, the redirect is definitely a poor idea but for the time being we have to leave with it.

Do you believe that this could be responsible for the symptoms i describe above? I think i have encountered these phenomena in cases where the site is new or it has just been indexed by Google, but in our case the site has been indexed 3 months ago... Could this be related to the low crawling and indexing rate that has been noticed during the last months?

[goodbyedee]

or is it the rewrite rules that makes crawling and indexing slower than normal?

[milosevic]

Rewrite rules, properly set up shouldn't affect crawling and indexing speed.

Perhaps there is an issue with the ordering of your rewrite rules - test the http headers received when GETting the homepage to see if there are multiple redirects - if there are, you need to re-order your rules so that most specific come first.

Redirecting from site root is always risky. There's clearly a problem with your home page from the symptoms you are experiencing.

[goodbyedee]

Thanks for the suggestion milosevic... I tried what you suggested and maybe we are getting somewhere:

I checked the http headers through 4 different applications + fetch as GoogleBot (from WT)+ seo-browser. All applications show normal (i think) results like:
HTTP/1.1 302 Found
Location: en/welcome.html
X-powered-by: PHP/5.2.10
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Content-type: text/html; charset=UTF-8
Connection: close
Server: Apache/2.2.3 (Red Hat)
Date: Thu, 13 May 2010 07:37:56 GMT
Set-cookie: PHPSESSID=u3mc2vb21f2sari8vs7ng8htf7; path=/
Cache-control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache

HTTP/1.1 200 OK

X-powered-by: PHP/5.2.10
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Content-type: text/html; charset=UTF-8
Connection: close
Server: Apache/2.2.3 (Red Hat)
Date: Thu, 13 May 2010 07:37:56 GMT
Set-cookie: PHPSESSID=3jq8l27celkdm3j8sdbsq3tqg7; path=/
Cache-control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache

except from 1 application which shows:

#1 Server Response: http://[my site]
FollowRedirects=False; Server requested redirection, but sent no new location.

What is this and how do i change it to True?

Also, another info that might be important. There is a script that redirects the user according to the browsers language.

[edited by: tedster at 8:11 am (utc) on May 13, 2010]
[edit reason] fix unintentional link [/edit]

[milosevic]

It sounds like you have redirects disabled in IIS configuration - there is a checkbox called "allow redirects" somewhere in the options that must be checked.

[goodbyedee]

i'll check it out, but how come "Fetch as Googlebot" or many other http headers check sites don't show any problem?

[milosevic]

Most agents (web browsers etc) ignore the server directive that does not allow redirects, and follow the redirects anyway.

This does not apply to Google or other search engine robots that follow server directives more strictly.

(What should have been edit to my previous post):

Whoops - I just saw you are using Apache rather than IIS when I looked at your headers more closely (I thought this was an IIS issue). I am not aware of how this option is enabled or disabled in Apache but a search through the documentation should help

[goodbyedee]

This does not apply to Google or other search engine robots that follow server directives more strictly.

Shouldn't then "Fetch as Googlebot" identify the problem?

[milosevic]

Yes it should do, but it seems that it doesn't.