homepage Welcome to WebmasterWorld Guest from 54.163.72.86
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Multilingual website - Crawling issue for in different languages
hayden




msg:4664100
 10:40 am on Apr 18, 2014 (gmt 0)

I have one website with two versions. One for arabic and one for english.
The arabic urls are indexed and crawled by google but the english urls are not indexed by google.
I have done everything to resolve this issue but still the english urls are not indexed and crawled.

Can anyone help me out?

 

aakk9999




msg:4664375
 11:23 am on Apr 19, 2014 (gmt 0)

I presume you had a look at your server logs and you can see English pages are not crawled.

Have you tried to test a few URLs in Google Webmaster Tools via Fetch as Googlebot and submitting to index?

wilderness




msg:4664411
 3:43 pm on Apr 19, 2014 (gmt 0)

Google allows (Bing, Yahoo and the others do not, unless you pay a fee) the submission of a custom (uniquely named) site map file.

It's been some time since I created the file and don't recall the submission specifics, however Google did index (over time) all the files it was missing (and accurately), while the remainder of the SE's (pay a fee) are left with inaccurate results.

lucy24




msg:4664443
 8:06 pm on Apr 19, 2014 (gmt 0)

You said "arabic URLs and english URLs". Did you mean that literally? The language is part of the URL, not created by behind-the-scenes javascript or cookies? Do the arabic pages have direct html links to the english pages?

Does your robots.txt include one or more sitemap lines? Most search engines read this information.

Something is preventing g### from crawling the pages. They don't pick and choose; being #1 means that you crawl everything.

The arabic urls are indexed and crawled by google but the english urls are not indexed by google.
<snip> but still the english urls are not indexed and crawled.

Emphasis mine. The English URLs are not indexed. Are they being crawled?

hayden




msg:4665208
 9:40 am on Apr 23, 2014 (gmt 0)

aakk9999 I tried to fetch the english webpages from google webmasters and it was done successfully. Shall I submit it to index over there? Will that help?

hayden




msg:4665209
 9:44 am on Apr 23, 2014 (gmt 0)

http://www.example.com/PageError/error.html?aspxerrorpath=/Language.aspx

This is present in sitemap file code. can anyone tell me what this line means?

[edited by: aakk9999 at 11:51 am (utc) on Apr 23, 2014]
[edit reason] Examplified - Members with "New User" status no URLs allowed [/edit]

aakk9999




msg:4665226
 11:08 am on Apr 23, 2014 (gmt 0)

Are you saying this is in your sitemap.xml?

You should not have this in Sitemap. What you are seeing is the URL that .NET reports when something has gone wrong (in this case it was within the script Language.aspx)

Sometimes you get this if your session has expired.

If you ever get this when browsing the site, you may see what the error was (depending on how the application was coded).

How did you get these URLs into your sitemap?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved