uploading large files like the file hosts

uploading multi hundred byte files 100mb

[phazei]

Ok, I searched the forums and found about 20 threads asking about large files (i guess they didn't search first, hehe).

I understand that http will usually timeout if the file is over 80-100mb, but how do the large file hosts (rapidshare, zshare, megaupload, depositfiles, upload.to, etc, etc) do it? They have files of up to a gig.
They have a web interface...

Is there a way to do this with php? With a nice progress meter? (apc might not work if it's not a http post)

[coopster]

Timeouts can occur either because of PHP settings or Apache settings, both of which can be set to meet your needs in the configuration for each. As far as a progress meter, there are ways to accomplish this as well but it means sending some information back and forth with the server.

[phazei]

I'm familiar of all the php settings needed (max_input_time, post_max_size, upload_max_filesize, memory_limit) and APC and uploadprogress PECL extensions for progress meters.

What I'm asking is how those sites generally do it. Do they just use a regular upload, or do they use some flash plug-in, or do they have some type of ftp wrapper, or something I have no clue about.

From all the posts I read on this thread, there seems to be a consensus that http can't be used for files upwards of 80-100mb.

But clearly there is some stable method to get it accomplished since there are so many dozens of file hosting sites where there are 10's of thousands of files uploaded through them that are easily over 500mb.

I'm not sure why memory_limit should make a difference; does it really store the entire file in memory, or does it just stream it to disk?

[pontifex]

There are some sites who allow huge uploads, yet http just takes very long due to the overhead. We limit uploads via http to 100 MB as well and switch to FTP for larger files. But I do not see any reason why not to allow PHP post max to 1GIG. From my experience (and I manage an upload site with serveral terabytes) most users are NOT aware of the HTTP overhead and that it will take them long to upload.

A good alternative is the "rad ftp" applet to give your users an FTP way to upload. Also a good idea is to use pureFTPd on the server side to create FTP logins in a database and manage those with PHP.

There are many combinations possible of which I tried most. The best results are dynamic FTP accounts like described above.

ok?

P!

[rocknbil]

I'm not sure why memory_limit should make a difference; does it really store the entire file in memory, or does it just stream it to disk?

For straight upload, it really doesn't, timeout is the larger issue with PHP.

Memory usage becomes an issue when most image upload implementations are paired with functions of the GD toolkit. GD takes a compressed image, say, .jpg or .gif, and creates a full bitmap in memory to do any manipulation. As soon as you cross the "2MB limitation" of PHP, this because a resource hog and you have to start increasing the allowed memory for PHP, or, switch to the less memory intensive IMagick/ImageMagick methods.

In your case, it sounds like we're talking about video, so the memory usage is not an issue. How do other sites do it? It depends. Facebook, for example, appears to use a Java-based uploader that apparently does not use http protocol for uploading. When you start dealing with uploads of that size, you're probably going to have to step outside the vanilla methods of php or even perl and do something else.

[punisa]

I had to construct an upload form for my company's clients.
Basically these were zip and video files cca 300-500 MB.
I've sorted the timeout limitation and file size.
But it still gave me headaches with custom errors from time to time.

In the end I resorted to implementation of a small java app that works beautifully!
Shows the current data transferred in KB, percentage, time, speed... everything.
The only problem is that I'm not 100% sure how safe java actually is.
Thus we use it only for our well known clients and don't expect any mischief from them.
But I'm not sure if I'd be all happy in making it public. What do you think?
Maybe its just me, I have huge doubts about technologies and languages I don't understand : P

Anyway it's a free script, you can PM for links if you want.

[rocknbil]

The only problem is that I'm not 100% sure how safe java actually is.

My knowledge of Java is limited, but one of the down sides is Java applications can read local files, and do all sorts of nasties if it's poorly programmed and has vulnerabilities. Or if it's coded maliciously.

So it really depends on the author, and how well it's built. Does your application have any sort of a message or support board? Did you Google it to see if it's been abused or has security holes? Similar to "phpBB vulnerability," "VBulletin vulnerability," if it's in widespread use you can find this out right away with very little Java knowledge.

Even if it has "vulnerabilities" if you put it behind a login you will probably have no troubles.

[phazei]

Actually, it's for uploading csv's. With millions of records. Even compressed many are hundreds of megs. My script lets them be uploaded, sorted, then inserted using load data infile in a fork proccess.

It all works very well for smaller files.

Could a java applet POST data to the next page including the file name once the upload was completed? I'm guessing it probably could. My current upload script will add an incrementing number to the end of the file name in case of duplicate file.

I would probably need a custom java applet. I don't think I could do one from scratch, but does anyone know of an open source java upload package? I'm pretty sure I know enough java to mod it.