Massive Data Feeds In .csv, .xls

How the heck do you open them?

[skibum]

There are some truely HUGE data feeds out there that cometimes come in Excel format but it seems almost impossible to get them to open even with a gig of RAM and a high speed processor.

Any secret to opening them, or just pump up the RAM? Can Excel even handle a file in excess of 100,000 KB?

Is there anything that will work well for getting them into a site? Will WebMerge handle something that large?

[Procyon]

I've had the same problem. Mine weren't quite that big, but excel did EVENTUALLY open it.

Makes you think - these datafeeds make our computers struggle, and google has to deal with thousands of these new feeds per day, if not more. :)

Ah, to be a search engine...

[plumsauce]

Strange.

I've been playing with 120MB files in the last couple of days. The editor opens them in less than 5 seconds for manual examination during debugging, and I'm feeding the data into SQL in about 3 seconds. 30 fields per line, about 120K records per file. The maximum memory utilisation performance monitor is just under 250MB.

Is it perhaps drive subsystem performance?

[Rogi]

I thought MS Excel had a 65,000 record maximum?

[MovingOnUp]

I've never seen any in XLS format. Most of the ones I've seen are CSV or pipe-delimited. It's easiest to just load datafeeds directly into a server-based database. Or, even better, write a program that processes the datafeed, reformats it to a standard format, and writes it to the database. Most programming languages have libraries for interacting with databases.

[antirack]

Excel opens CSV files, but this doesn't mean you have to open it with excel.

[badone]

Bash, PHP, Postgresql is how I handle files of this ilk. OTOH, I have no experience with large datafeed files as such, although I have a lot of experience with databases and massaging data files so they can be sucked into the database.

While were at it can anyone point me at a datafeed file lying around on the net somewhere or email me one? Doesn't have to be current, just want to have a play :-)

Cheers,
BAD

[growingdigital]

Import the .xls/.csv file into MySQL. Extract the results with PHP, or just view them using phpMyAdmin.