Forum Moderators: Robert Charlton & goodroi
My client have more 5 million pages website, he want to me index his site on google/yahoo...
Can anyone guide me that how can i index all pages?
What should i do? or tool i used...?
I don't want spam way, just white hat
Regards,
Mianoor
But I simply cannot see why anyone would want to do it that way. For example, we buy a lot of wire products from Anixter. They have over 300,000 basic wire and cable products. But all the spec sheets etc are on PDF files, not webpages. Only the product search and selection area has it's own pages. They have a massive database, but if you want to see the details about a particular product, the link points to a catalog PDF page.
It's not ( only ) that a web site shouldn't have 5.000.000 different pages served for each and every product variation, function, feature combination of the site's search and comparison features but...
That it shouldn't need all of them indexed by Google.
That's it... you can have this many pages if you have to... but i don't see why you, Google or Google users would need them to be indexed.
On-site search features, pages combined to have more weight and useful info without having to clickedy click through a million of nav links, well thought up category and sub category pages should do the trick.
I mean i'm all in for long tail, IF it's reasonable but a startup with 5 million pages...? Get Google index the first 2500. Users should be able to decide where to go on your site from that many landing pages :P
Not to mention that if these pages aren't as unique as you like to believe... indexing them in an innermost area will get them to sub sub supplemental PR0 state, in which case any hobby site will outrank them at will. Again there's no point.
Furthermore URLs appearing in masses will flag you for eternity. And for a good reason ( spam ). Adding them gradually means adding them at a rate which the trust and history of the domain allows...
What do you think a startup has in those departments?
'scuse me for the rant. :P
If these pages are visually or information-wise unacceptable in any other way than breaking them down into so many pieces, i'll take everything back. But if they could be combined without compromising - or even to increase - usability, then do it. For your own-, your customers' sake.
Tedster,
in that case you will have no problems. Worst case scenario it trips a wire, a google engineer looks at it and lets it go through. Google will actually be thrilled as they love good informational pages.
Now for those with borderline spammy--as defined by Google--pages...good luck adding tens or hundred of thousands of pages.
That's it... you can have this many pages if you have to... but i don't see why you, Google or Google users would need them to be indexed.
Theoretically ... yes.
But there is the convenience factor. I need to lookup many different things quite often from several unrelated databases that can be searched directly, but which are indexed in SEs, too. I am lazy. There are no bookmarks (different browsers at different workstations), and I can't remember all those specific longwinded database-site-lookup-ULRs (old age, sorry ...).
So I type my specific search into google or search.yahoo, and ... aah there on the SERP on position 7 I see the desired result for the site I was interested in but didn't rembember exactly.
Just one search interface for all and everything. Fast and easy user experience.
Count me on those users who want these pages to be indexed.
Kind regards,
R.
for most terms supplemental is fine
That's an important insight. It's especially true because the URLs can still be returned for the long tail searches, which is all you need for that kind of deep content. There's little value in millions of URLs if the key "fat belly" search terms start to tank.
What's an eternity? Six months, one year, or forever?
I've got 3 large sites that are somewhat related I'm putting out. The first one has 250k pages, the second has 2.5M, and the third has 7.5M. I'm going to favor my users over G...in expectation of more direct hits over time than search engine hits. As a result, when a site goes live I'm putting up every page at once. The first site has been up for 2 months and has 1600 pages indexed. Almost all of them are supplemental. I'm hoping to finish the next two sites before the end of the year.
I'll let you know how it goes...how many pages get indexed, how many are supplemental, what page rank the home page gets, etc. It'll be a fun experiment for the G be ¦]@^^&¦] approach.
No it really doesn't. For the site I was working on the supp result came up first with just a part number.
Someone ask a question about arrays. What you do is create descriptions and keywords in a huge array, then let coldfusion edit the pages. You can do the same for titles and even content.
The real issue i think goes back to what Tedster said. What most of these companies are trying to do is dominate a SERPS category for part numbers or variations. They would actually have better conversion rates by concentrating on building less pages, with better (read:more compelling) content, and then list the part numbers in logical groups.
It's the nature of the search engines that a search box alone on your site won't do the trick, so you have to put the part numbers and color, size variations in the content, but again, if logically sequenced your conversion rate will actually rise. All search engine algos work this way (size of content/density ratio) you will not find an exception.
1. Good internal navigation.
2. Lots of deep links.
3. Time.
Go to one of your product pages at random, as if that product was links from a post on some forum. You can expect that page to be linked the first month that the link exists.
Now look at every link off that page. Most of those pages will be linked the second month that the link exists. If you have good internal navigation you will get multiple deep pages as well as the high level pages such as the home page, which would already be indexed.
The next month, you can plan on having the pages that are one click away from the previous month's pages in the index.
It may happen faster than this, but I never plan on it. Googlebot may also go more than three clicks from a deep link, but don't count on that either. Strong links will get you more than three clicks, but it is the weaker links that will end up doing most of the work due to the quantity and variety.
One way to slowly help things along is to put some "featured products" on your home page with direct links from there. This will send the bot deep to a few areas from what is probably your strongest page.
Whatever you do, don't try and rush it.
