100k pages - Which technology is best to serve this?

ASP.net or off the shelf app

[jerseyboy]

I am publishing a site with 100,000 pages. At first glance I thought to use Magento and disable the shopping cart parts, so I could have an SEO strong, but this seems to be very memory use intensive which raises hosting costs above the $30/mo I have in mind for this site.

If you were to publish 100k pages (revenue from ads), and you wanted a low hosting cost yet have reasonably fast page loads, what would be your tech stack? (i.e., ASP.NET or off the shelf app)?


JB

[ergophobe]

The number of pages on a site is perhaps the least important factor in choosing a way to manage that content.

Any CMS can handle that though if I recall correctly, traditionally WordPress has performance issues with large numbers of "pages" (but not large number of "posts"). Sometime like that.

Anyway, what's really going to matter is how the pages are put together (how many DB tables are you drawing from, are they indexed, can the queries and/or pages be cached and so on), how many concurrent users you have at *peak* load, and what you have for hardware ($30/mo can get you a decent VPS if you don't need a lot of support) and software (opcode caching or not) etc.

If I were expecting a lot of users and complex or numerous DB queries, I would want to be sure the system had good caching. This might be simple or complex. On some sites, I've just grabbed the output buffer in PHP and if there's no static version of the page, cache one. On the next request, you're serving static pages with just a quick check for whether not the cached version exists.

If you don't have a lot of dynamic content, you can get very fast page loads even if the initial page generation is processor/memory intensive.

[jerseyboy]

What tech stack would you use to publish 100,000 pages with minimal DB table queries? I am thinking of a site similar to schooldigger.com, which has as many pages.

Also, I've only used cpanel not VPS - is it difficult to learn for a novice?

JB

[jinxed]

Magento guzzles memory, I'd give that a miss.

[jerseyboy]

What would you propose Jinxed? ASP.NET custom build?

[bhartzer]

Actually, I would consider looking at WordPress. There are plenty of sites that are using WordPress and don't have any issues. Pages can be easily cached, which helps with server load.

[jerseyboy]

OK, but 100,000 pages? That is the size of my database I want to publish.

[jinxed]

I think you're confusing a CMS and the language the CMS is built in.

PHP, ASP etc all do the job.

To be honest, I would look into Wordpress. I don't personally use it but it is the market leader for a reason and easy to tailor to your personal needs.

[bhartzer]

One consideration is whether or not you'll have to hit the database each and every time a page is loaded for a user.

When you're talking server load and having to server up lots of page views, it's best to using caching--and WordPress can handle that by caching the content.

[jerseyboy]

Magento caches content very well which is why I started using it for this project. I also stated writing my own database driven site in ASP.NET, but went to Magento since it exists and has the caching, indexing, CMS features, and SEO tools. But it is very slow.

[ergophobe]

JB - you keep mentioning the number of pages. If it's a simple query (which a cached page in Wordpress or Drupal should be), then 100,000 pages is a *small* number, especially if you have your DB tables properly indexed and you measure and slow queries (if possible on your hosting setup).

The number of concurrent users is a far more important number and that's where the overhead of something that is highly dynamic starts to hurt. For example, the "who's online" feature of bulletin boards and forums tends to cause huge overhead in and of itself if there's a large user base.

So when you say Magento is slow, that is not likely because you have 10 pages, 10,000 pages or a million pages. It is most likely simply because generating any page in Magento takes many queries or a few very slow queries. That's the part that will make scaling difficult.

So like bhartzer says - if you use WP and implement agressive caching (WP SuperCache for example) or Drupal (with Boost), your bottleneck will likely be on the front end - CSS and JS cruft. Then it's a matter of a theme that keeps things simple, and not adding features that add complex and slow DB queries.

[jerseyboy]

ergophobe - very awesome description. makes perfect sense now. thank you!

[ergophobe]

You're welcome - doesn't solve your problem or answer your question, but hopefully it helps you focus on the questions that will matter the most.

[Rickvb]

Have you looked into WordPress with SQLite? Also important: does the content change over time a lot? You might also want to look at Sphinx Search Engine to speed up your full text searches (from Wordpress search functions):

[wordpress.org...]

Also remember that with popular posts, you will want to have query caching activated. But if all posts get visited the same amount of times, you might not want to do activate this.

[Vuffy]

Any forum software can easily handle that much and more.