Dedicated Server Advice

[MarkWarner]

Hi,

My main problem is I work pretty much in a bubble and have no "IT friends" or watercooler to help me get advice.

I run a self-financing hobbyist comic book site. Our dedicated server is being switched off 30 June, so I am on a hunt for another server. As usual with me, I am going round and round in circles.

What I am after is to be used as an image server. We have a lot of data (3.5tb 1.5m images). Ideally I'd like 8tb BUT at a stretch could use 4tb. The main issue (as always) is a limited budget v reputable service.

I realize that individual hosting companies can't be discussed on this forum .But any guidance about what features to look for to get a good deal I'd be eternally grateful. In other words, can anyone point me in the right direction.

Many thanks!

[lucy24]

It sounds like you know most of what you need. But alongside server space you also need to think about bandwidth: how much up-and-down movement is there in a given time period? Are all of those 3.5tb of images accessed regularly, or do many of them sit in storage for weeks on end? Another variable is processor load, if it’s a dynamic site.

It's been a while, but I do remember that when I moved to my present host, there was no limit on storage space, but they charged for excess bandwidth. (afaik, they no longer do, but may charge for processor use instead.)

[MarkWarner]

it is a static file server. So I upload every month or so,

Unlimited space on shared hosting does not actually exist. There are caveats about "fair use" with maximum set for both disk space and inodes (number of files and folders). If it really did my life (and other owners of very large sites) would be so much easier! What happens is you get an email from "Compliance" followed by bright lights, lack of sleep and you "accidentally" falling down the stairs,

[lucy24]

Ouch. But the question was about downloading, not uploading; I assume the files are there for other people to access. And then it makes a difference how much they access and how often, in much the same way that your electric bill isn't determined by how many lamps you own but how often you turn them on.

If you want to split hairs, unlimited space doesn't exist *anywhere*, since even the most gigantic server must fill up eventually.

[MarkWarner]

No my problem is file space, not bandwidth That is what I am asking.

Look around for machines which could actually house 3.5tb let alone allow for inevitable expansion. Most providers do not actually have such machines available. Those that are can be very expensive/

Sorry i am not splitting hairs. OF course unlimited space doesn't exist anywhere. If you have a 1tb disk then it can hold so much. That you know and can watch.

It is not about bandwidth for the providers it is about disk space. Shared hosting they want to cram as many sites as possible onto each machine. So if there is one with 1000 times more files than an average blog they'll boot it off. Been there!

[thecoalman]

I have to ask, have you optimized the images? Since you mentioned comics I'd even see what kind of results "lossless" 256 color palette with webp produces. It can be superior in some aspects when you have line type drawings as long as it can accommodate the color range. Specifically line edges will remain crisp relative to compressed .jpg.

[MarkWarner]

Hi They are indeed jpgs and I have optimized, by shrinking dimensions for the online viewer.

They are out of archives. Basically hobbyists scan books. comics then zip them. Not for compression as there is none, but to create a single file. Viola, a primitive, but very effective ebook. There are readers available!

I also have these original archives for download. Which in fact take up most the space (but not the inode count).

Converting to webp I guess will save some space, but judging by images I have converted on the site not a huge amount. It will not alter the number of files!

I think I am on the home stretch ordering a new server. Just performing "due diligence" (aka stalling spending a few K)

[MarkWarner]

I went to alter a "viola" to "Voila" and ended up adding another post and now can't edit the one above. Hey ho! I also forgot I had a "few thousand" OTR mp3 files as well!

Hopefully have will be migrating in a day or 2. Fingers crossed!

[thecoalman]

Hi They are indeed jpgs and I have optimized, by shrinking dimensions for the online viewer.

In addition to the resolution when using .jpg or .webp there is a quality setting, 90% is practically identical to 100%. and there is significant savings. This works well with photo realistic images but less so with comic type images because of the text and lines which will get fuzzy, the lower you go the more artifacts you will see.

I'd suggest again you experiment with 256 color palette with webp, lines/text will remain perfectly sharp but you lose the color range which may not be such a big issue with comics. Ideally you are scanning to lossless true color source file like .png for archiving and then convert to whatever format for publishing. Also not there is tools like pngcrrush for substantially reducing the size of .png files.

Hey ho! I also forgot I had a "few thousand" OTR mp3 files as well!

MP3 has quality slider too, 120kbps is usually suitable for decent quality however mp3 is a pretty old codec. I have not experimented with webm but it's going to produce much smaller file for same quality.

They are out of archives. Basically hobbyists scan books. comics then zip them. Not for compression as there is none, but to create a single file

Compressing binary files into .zip like images, audio or video isn't going to do much for you anyway. It might only be a few % less.

[MarkWarner]

Yes as I said the zipping isn't for compression, but to create a single file. Which is then read as a book. A very smart idea from way back when.

These files are created by hobbyists everywhere and it's up to them what the standard of the images are. These are actually the biggest user of space as my individual files are resized down and resaved. The mp3 files again are made by hobbyists. They are old radio shows and were mainly made a few years ago when disk space was at a premium. So they tend to be rather small.

I had ponder before about remaking mps and the image archives will just mean that there are duplicates handing around. And the effort was just worth the savings. And as I said file numbers is easily as big a problem as file sizes.

As it is I've now got a server and files are copying across now. Not as much space as I wanted but enough. In budget and in fact enough for me to get a 2nd machine when the time comes. So we're hopefully good to go for a few years!

[Kendo]

Amazon S3 is good for file storage. Then you can host your website anywhere, even on a shared hosting service. Setting up on Amazon can be tricky bit there are a lot of devs on freelance sites that can help you get going (ours might be available). That would work out cheaper in the long run and there are no limits. You get billed for what you eat.

[MarkWarner]

Hi Kendo, I've actually got the site moved and up and running. So far am VERY pleased with my hosting company. Came in at a great price and been faultless. ({Famous last words!) I did have a quick look at AWS ... but looked like a bit of a learning curve (which being a software developer I am not afraid of) . I signed up an account a while back to investigate. But so much to do .... and so little time and money lol!