homepage Welcome to WebmasterWorld Guest from 54.161.175.231
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Code, Content, and Presentation / PHP Server Side Scripting
Forum Library, Charter, Moderators: coopster & jatar k

PHP Server Side Scripting Forum

    
Trying to Open 712MB XML file
Please help me to open this
CWebguy




msg:3491794
 7:57 pm on Oct 30, 2007 (gmt 0)

Hello, thanks for all your support and help! This might sound stupid, but I am having trouble opening a 712MB XML file from Amazon.com. This is the smallest version that they offer and when I run my PHP script on a smaller file (25MB) it opens fine. By the way I'm using the SimpleXML function built into PHP 5. Yet when I try to open the larger file through the PHP script I get error messages after about 3-5 minutes about allocated memory and SimpleXML being unable to parse.

I just recently upgraded my computer's RAM to 1.25 GB but it doesn't seem to fix the problem (but did seem to help somewhat). I have no background programs running except the basics and Apache & MySQL too. Yet, it still won't open. I don't really have much money to upgrade my computer anymore at the moment. Is there anything that anyone could recommend to help? Thanks!

Keep coding!
Peace!

[edited by: CWebguy at 8:01 pm (utc) on Oct. 30, 2007]

 

whoisgregg




msg:3491862
 8:50 pm on Oct 30, 2007 (gmt 0)

Check your php settings regarding max_execution_time [us3.php.net] and memory_limit [us3.php.net]. Both values would affect your ability to process a file that size.

CWebguy




msg:3491940
 10:02 pm on Oct 30, 2007 (gmt 0)

Yes, I put

set_time_limit(0);

in the script for the timeout factor. I'll see about the other one here in a minute.

CWebguy




msg:3491957
 10:24 pm on Oct 30, 2007 (gmt 0)

In the PHP.ini file the memory limit was set to 128MB, but that can't be right because whenever I run the script it uses the entire system's memory. Not only can I hear my computer chugging hard (kind of scary) but the memory availabe drops into the single digits :( . I tried upping the memory_limit to 700MB in the .ini file and restarted Apache but got same warning messages and results:

Warning: simplexml_load_file() [function.simplexml-load-file]: ....7770953: parser error : Memory allocation failed : xmlSAX2Characters ....on line 10

and a couple others. But like I said, it will do the smaller file which is in the same format so the script is not the problem. Is there anything else that can be done? Maybe a function in PHP or another setting that would work to open this large XML file and parse it?

Thanks!

hughie




msg:3491964
 10:34 pm on Oct 30, 2007 (gmt 0)

things always get messy dealing with LARGE text files, have you tried splitting it and uploading it in chunks?

CWebguy




msg:3492146
 2:13 am on Oct 31, 2007 (gmt 0)

Thanks! Is there a way that you can recommend to split the XML file? (For instance using PHP or another method?)

Any help is greatly appreciated! Thanks!

CWebguy




msg:3492203
 4:18 am on Oct 31, 2007 (gmt 0)

By the way, the reason I ask is I'm having trouble opening in a text file (it just goes on and on loading). Maybe there is a way to split it without having to open and manipulate it. Thanks!

hughie




msg:3492688
 3:53 pm on Oct 31, 2007 (gmt 0)

Any filesplitter will do the job, you've just got to be careful not to split it in the middle of a record.

I've got a good free one on my computer called "filesplitter" but for the life of me i can't find it on the web. but i'm sure there are plenty around.

CWebguy




msg:3493751
 4:43 pm on Nov 1, 2007 (gmt 0)

Thanks! Just wondering, but why can't I split in the middle?

Thanks!

hughie




msg:3494121
 10:29 pm on Nov 1, 2007 (gmt 0)

well if your record is

<name>fred</name>
<email>fred@fred.com</email>

and you've split the data then theres a possibility that

<email>fred@fred.com</email>

will become
<email>fred@

then in the next file
fred.com</email>

Which is fine if you're going to join the two back together again, but with a 700mb file you're far better to deal with each individual file and whack it in a database rather than rebuild the xml file and try and work with that.

ncw164x




msg:3494163
 11:17 pm on Nov 1, 2007 (gmt 0)

Try "Large Text File Viewer", it was created for viewing large 1GB text files

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Code, Content, and Presentation / PHP Server Side Scripting
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved