Welcome to WebmasterWorld Guest from 54.163.40.152

Message Too Old, No Replies

Multilingual website - Crawling issue for in different languages

     

hayden

10:40 am on Apr 18, 2014 (gmt 0)



I have one website with two versions. One for arabic and one for english.
The arabic urls are indexed and crawled by google but the english urls are not indexed by google.
I have done everything to resolve this issue but still the english urls are not indexed and crawled.

Can anyone help me out?

aakk9999

11:23 am on Apr 19, 2014 (gmt 0)

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month



I presume you had a look at your server logs and you can see English pages are not crawled.

Have you tried to test a few URLs in Google Webmaster Tools via Fetch as Googlebot and submitting to index?

wilderness

3:43 pm on Apr 19, 2014 (gmt 0)

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Google allows (Bing, Yahoo and the others do not, unless you pay a fee) the submission of a custom (uniquely named) site map file.

It's been some time since I created the file and don't recall the submission specifics, however Google did index (over time) all the files it was missing (and accurately), while the remainder of the SE's (pay a fee) are left with inaccurate results.

lucy24

8:06 pm on Apr 19, 2014 (gmt 0)

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



You said "arabic URLs and english URLs". Did you mean that literally? The language is part of the URL, not created by behind-the-scenes javascript or cookies? Do the arabic pages have direct html links to the english pages?

Does your robots.txt include one or more sitemap lines? Most search engines read this information.

Something is preventing g### from crawling the pages. They don't pick and choose; being #1 means that you crawl everything.

The arabic urls are indexed and crawled by google but the english urls are not indexed by google.
<snip> but still the english urls are not indexed and crawled.

Emphasis mine. The English URLs are not indexed. Are they being crawled?

hayden

9:40 am on Apr 23, 2014 (gmt 0)



aakk9999 I tried to fetch the english webpages from google webmasters and it was done successfully. Shall I submit it to index over there? Will that help?

hayden

9:44 am on Apr 23, 2014 (gmt 0)



http://www.example.com/PageError/error.html?aspxerrorpath=/Language.aspx

This is present in sitemap file code. can anyone tell me what this line means?

[edited by: aakk9999 at 11:51 am (utc) on Apr 23, 2014]
[edit reason] Examplified - Members with "New User" status no URLs allowed [/edit]

aakk9999

11:08 am on Apr 23, 2014 (gmt 0)

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month



Are you saying this is in your sitemap.xml?

You should not have this in Sitemap. What you are seeing is the URL that .NET reports when something has gone wrong (in this case it was within the script Language.aspx)

Sometimes you get this if your session has expired.

If you ever get this when browsing the site, you may see what the error was (depending on how the application was coded).

How did you get these URLs into your sitemap?
 

Featured Threads

Hot Threads This Week

Hot Threads This Month