Forum Moderators: phranque

Message Too Old, No Replies

How to update a big site fast

How can you update thousands of pages with no delay for the visitor?

         

dirkz

9:59 am on Aug 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi all,
I'm running a big site, consisting of pages with a number near 70,000 or so. They are created "offline", so in the end there is just a bunch of pure html. Since the production takes some time, I usually "procduce" into a test directory, view it, and when it's allright, I copy it to the destination directory where apache has access to.

How do you update? Is there a faster way? E.g., using symlinks and telling httpd to use a different directory after production, which would effectively mean to swap directories.

Mark_A

4:45 pm on Aug 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Hi dirkz

that is a lot of pages .. well I wont ask why you need so many ...

"with no delay for the visitor?"

Well if its just a change to a site wide menu or something you could use something like SSI (server side includes) and then when you made a change to the included code, it would work across all pages as soon as it was made.

But I will ask how you create the pages offline as I also create some sites offline also in a semi automated way .... by building a database of contents .. various page templates, inc menus, drill down page queries etc then merge them all into static html pages - I can test most of the sites offline and then just ftp the lot up to the webserver ....

But .. these are perhaps 300-500 page sites ... I would not do that for 70,000 pages as page creation and ftp etc would just take too long ... what I might do (and in this I am not yet expert by any means) is move a contents database online with templates etc and allow the server to create the pages on demand from users.

Then when I wanted to update the site I would update the database online or offline no matter .... anyhow updating just the database plus any changes to the templates ... rather than creating all the 70k pages ....

does that make sense for your application?
if not what is it that you are doing?

trillianjedi

4:49 pm on Aug 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



E.g., using symlinks and telling httpd to use a different directory after production, which would effectively mean to swap directories.

I'm not sure you need to use the symlink do you? You just port the new version up onto a different drive volume and tell httpd that's where the site is?

TJ

netcommr

4:52 pm on Aug 29, 2003 (gmt 0)

10+ Year Member



just change the document_root in the web server and restart it

or

just swap the 2 folder names containing the old and new sites.

or

get creative and add a folder name (such as 'foo') to all your urls, then do a mod rewrite to change /foo/ in the requested url to the day of the year, then in the folder named /187/, which would be tomorrow, you have tomorrows site. Then your site changes automatically at midnight... get the idea, lots of possible ways.

dirkz

5:25 pm on Aug 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Mark_A, at the start I thought of using PHP or Perl or Servlets and accessing the database in realtime, but thought this would be too costly in terms of load on the server on really high traffic. I want my users to experience my site as fast, no matter how many are accessing at the same time. The database and the building engine is of course running on the server, but the update still takes time.

Btw, I use a self-written engine, using Java, XML, MySQL and Velocity.

trillianjedi, I would need the symlink for accessing the "stable" content from the freshly generated. If I publish to a different location and want to test it online, I have to symlink the stable stuff like images and css.

netcommr, hadn't thought about rewriting. How can I achieve the switch fast? I thought of "httpd restart", but this also takes some time. It could mean that "you pull a page from a viewer's butt" if he tries to load it in the same time. Is there a fast ways to tell httpd that the document root or the rewriting rules have changed? How are the monster sites doing this (if they use static html)?

piramida

10:59 pm on Aug 29, 2003 (gmt 0)

10+ Year Member



dunno about the monster sites, but httpd would reread it's conf on a received SIGHUP just like any other good daemon. just do "kill -HUP <httpd thread>".

and there should be a friendlier way for this in some CPs?

bilalak

5:47 am on Aug 30, 2003 (gmt 0)

10+ Year Member



Dirk, I think you can use .htaccess for some time to rewrite the URLs then you can change the configuration when you think that your traffic is lower and this does not take more than a minute for the whole process.

I suggest that you use XML with CSS for your web site. It might be better to use a database for the content but you export it into XML files every now and then. This is much faster and easier to maintain.

Luck!

dirkz

6:17 am on Aug 30, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



bilalak, piramida, thanks for the input!
Yes, I'm using XML for the html stuff of the pages, the database only for media content. Mainly because I can edit my XML stuff much faster with my favorite editor. Absolutely agreed.

piramida, what do you mean by "CP"?

Yes, I have a script on my box called "httpd", it supports the "restart" command.

Guess I really have to wait for the traffic slowing down. Problem is, that my site is pretty much aimed at international visitors so there is no such thing as fixed slow down, but maybe I can spot some calmer hours.

dirkz

6:19 am on Aug 30, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Sorry, I meant "httpd reload". It should be using a HUP and be faster than "restart".

bilalak

7:17 am on Aug 30, 2003 (gmt 0)

10+ Year Member



CP is the control panel to manage a host account I think
It is a web Administration Interface