Forum Moderators: Robert Charlton & goodroi
My client have more 5 million pages website, he want to me index his site on google/yahoo...
Can anyone guide me that how can i index all pages?
What should i do? or tool i used...?
I don't want spam way, just white hat
Regards,
Mianoor
Huh, could you enlighten me about verry big array etc ....?
Well ebay own shopping sites like shopping.com and dealtime.co.uk do it better but mostly rely on serving and optomising two pages at once. For instance "dealtime" would be one phrase and "dealtime-" would be second.
[edited by: Pirates at 1:46 am (utc) on Nov. 24, 2006]
[edited by: Pirates at 1:50 am (utc) on Nov. 24, 2006]
oh, a grand total of 1,900 pages indexed by Google, amazing heh
[edited by: theBear at 2:08 am (utc) on Nov. 24, 2006]
I'm currently working with a site that has 400,000 urls indexed, growing up from 200,000. Their newly redesigned domain actually has over 30 million URLs, but we will not "expose them" to googlebot all at once. Clearly, if MSN got nuked for a mere 10 million at once, then even a respected business with that many URLs all designed for visitors, will still have a problem dumping them all at once.
There's a lot of detailed planning required just to be able to show these new URLs gradually -- including the link structure as well as the savvy use of url re-writing and robots.txt. (I'm getting paid OK for the job, but not nearly 2 million. I can only wish!)
for most terms supplemental is fine
That's an important insight. It's especially true because the URLs can still be returned for the long tail searches, which is all you need for that kind of deep content. There's little value in millions of URLs if the key "fat belly" search terms start to tank.
Does a site really need to get that many pages indexed?
I mean, does a shopping site really need to get every product indexed with every colour variation, every size variation, every design variation, every different manufacturer variation, every purchase-quantity variation, and every other type of variation, published each as individual pages?
I think not.
.
Additionally, I am troubled by the fact that you have seemingly taken this job on, and are getting paid for it, and then you have to ask in the forum what to do. Does the client know that you don't know what to do?
[edited by: g1smd at 2:30 am (utc) on Nov. 24, 2006]
Additionally, I am troubled by the fact that you have seemingly taken this job on, and are getting paid for it, and then you have to ask in the forum what to do. Does the client know that you don't know what to do?
1. I have not taken the job on
2. If I did I could probably do it without help. But if I needed forum help I would ask for it as there are some great people here.
[edited by: jatar_k at 3:13 am (utc) on Nov. 24, 2006]
To get that many pages ranked is essentially about a 2 million dollar plus project.
Why do you believe that it will take this amount of financial resources to get it going?
new site?
bad history, already got removed?
urls so GET stringed out that a bot can't follow?
mianoor, a bit more info about why it isn't indexed would help get better answers. Unless you mean ranking it, which is something all together different.
as Ted said, a gradual approach is the best way.
Hi there, just a question, how did you do your page count?
site:www.ebay.com yields a max of 23,000 yahoo, 1,500 msn,
1900 google
I guess i am missing quite a lot of stuff here,
I will have to amend some techniques
Since I am in charge of optimizing some of the sites you are talking about I know this quite well.
@ thebear
Hi there, just a question, how did you do your page count?site:www.ebay.com yields a max of 23,000 yahoo, 1,500 msn,
1900 googleI guess i am missing quite a lot of stuff here,
I will have to amend some techniques
Try site:ebay.com, eBay uses sub domains. When you envoke a site:www.ebay.com you only see pages indexed on WWW.
i get 1830 showing in google's main index and 2620 with the supps so i don't think you are missing anything.
I have a couple of questions for people with sites that have millions of pages. Are they all selling things, with millions of products. Who has written the content for the millions of pages, is it valuable unique content etc?
Unless there is a massive product database i don't see how people could have such a large site.
If you were able to write 20 pages of content a day and you wanted a 1,000,000 page site it would take you 136 years to get the content done. How can anyone do this? Even if you wrote 100 pages a day it'd still take 27 years. Saying that, this is talking about one person doing it, not 100 people, so i know it's achievable, but if anyone who's running these sites can give me a bit more detail i'd appreciate it.
They start with a 250,000 SKU database, without even counting category, sub-category, sub-sub-category pages, comparison pages, product details, online manuals, and so on. There are many businesses like this learning to make good use of the web, and playing catch-up in many cases.
There's no doubt that their existing customers now want to be able to access all this information online - and I'm sure new customers will too. The challenge is not to show a similar profile to the search engines as a spam play while all this is made available.
Unless there is a massive product database i don't see how people could have such a large site.
Really?
We've just built a site with over a million pages and it isn't a product database, and yes before you ask all pages are unique (not just slicing and dicing.)
