Forum Moderators: coopster
I'm having problems with my PHP script eating up memory. I'm not sure if it's because of my code or an actual PHP bug (some have been reported in the past).
Error:
Fatal error: Allowed memory size of 104857600 bytes exhausted (tried to allocate 700161 bytes) in #*$!#*$!x on line 168
This is even with me setting:
ini_set("max_execution_time",120);
ini_set("memory_limit","12M");
My PHP version is: 5.2.4
The script involves a php file which does a simple SQL query.
A class instance is created.
Then there is a while loop going over the record set and passing a value to the class methodA.
Class methodA runs another query, loops andcreates an array, then passes the array to methodB.
methodB then loops over the array, running a query, connecting and disconnecting from the db each time and passes back to methodA.
From what I can gather, most of the memory is being used up by the loops in the php file and in the class. Say at a rate of 1MB per 8 database rows returned. So it's going over the 12MB limit pretty quickly.
I have found some articles relating to memory leaks in php5.1, but we have 5.2 and they should have been fixed by then.
[bugs.php.net...]
[bugs.php.net...]
[webmasterworld.com...]
I've looked into using unset() and avoiding looping in the fashion they state, but it only reduces it by a few MBs, nothing major.
Have you any ideas what I can do to reduce the memory usage of the script?
I appreciate any pointers you can think of.
[edited by: eelixduppy at 3:01 pm (utc) on Sep. 9, 2009]
[edit reason] snipped some links [/edit]
thanks
I had a situation where I read a text file (thousands of lines of data, only a few megabytes in size). My script read the file, and processed the data through a series of loops... creating arrays... etc. End result? Simply processing that ~3mb text file into a few arrays ended up needing something like 100mb of memory. I learned after doing this, that creating an array to hold data... though being incredibly useful, easy to deal with, manipulate, etc... it actually consumes a lot more memory than you'd expect!
I would scrutinize the process of what it is you are trying to do. Most likely (without seeing your code), you might be creating arrays that are being stored in memory with each loop (which might not be needed). A solution to this could be to simply discard (unset/etc) any arrays that are created and not needed. Remember that PHP will create "new memory space" for each new instantiation of a variable/object; PHP does not inherently do "references"; and it will keep these items in memory until told otherwise or the script terminates. It is also possible that the total memory used by your array could very well exceed the memory limit (if huge amount of data).
Obviously, I cannot see your code... nor do I have the time to trace it... but it is obvious something in your loops is probably getting kept and increasing memory with each loop.
Here is my code:
The PHP file is looping through a list of companies that distribute in several locations. It's passing the company ID to a class that will return a list of all the location names that that company distributes to.
PHP file (memory is ok until this point. It's this line that racks it up about 1MB every 15 loops):
$fullLocationList = $cd->loadLocationByCompanyId($col['CompanyId']);
Class:
<?PHPclass CompanyDistribution {
# run queries on db
private function runQuery($sql)
{
$dbc = DBConnect(DB_DSN_X,true,true);
$qry = DBQuery($sql, $dbc, false);
DBDisconnect($dbc);
unset($dbc);
unset($sql);
return $qry;
}
# return all locations a company distributes to
public function loadLocationByCompanyId($mid)
{
$sql ="SELECT LocationCode FROM CompanyDistribution WHERE CompanyId = {$mid} ";
$qry = $this->runQuery($sql);
$locationCodes=array();
while($row = DBFetchArray($qry))
{
$locationCodes[] = $row['LocationCode'];
}
$companyLocationNames = $this->loadLocationByCode($locationCodes);
unset($qry);
unset($locationCodes);
return $companyLocationNames;
}
# return location name from location code
# this is also used directly by the PHP file
public function loadLocationByCode($locationCodes)
{
$locationNames = array();
foreach($locationCodes as $x=>$code)
{
$sql = "SELECT LocationLevelName AS LocationName
FROM LocationLevel
WHERE LocationLevelCode = '{$code}'
";
$qry = $this->runQuery($sql);
while($row = DBFetchArray($qry))
{
$locationNames[$code] = $row['LocationName'];
}
}
unset($locationCodes);
return $locationNames;
}
}
I've been on this another few hours this morning, but whatever I try I'm hitting the error. My memory limit is currently 16MB, but I had it at 120MB and it still broke. My search is currently returning 450 rows, but the app is supposed to return up to 2500....... eek!
try creating some indexes in yot databse tables to speed up the things a little
foreach($locationCodes as $x=>$code)
{
$sql = "SELECT LocationLevelName AS LocationName FROM LocationLevel WHERE LocationLevelCode = '{$code}' ";
$qry = $this->runQuery($sql);
while($row = DBFetchArray($qry))
{
$locationNames[$code] = $row['LocationName'];
}
unset($locationCodes);
return $locationNames;
}
Hve you tried altering your logic here?
Perhaps you can construct your sql like :
SELECT LocationLevelName AS LocationName
FROM LocationLevel WHERE LocationLevelCode IN '$code_list_of_id'
if you can of course (where $code_list_of_id would be something like $code_list_of_id='9,12,456,32,123' - comma seperated list of ids
Your "process" is...
Your process seems to be...
- Call 'loadLocationByCompanyId'. This then...
- runs a query;
- creates an array 'locationCodes';
- runs a loop thru database results (temporarily creates an array for each DB result);
- then call 'loadLocationByCode' inputting array 'locationCodes';
- + this then runs a loop on locationCodes
- + - runs an SQL query for each location code
- + - then loops each of these query results
- + - + sets up array 'locationNames' with the name from the result array
- + - returns the location names array
- original method call then returns this same names array
Not overly complicated. But consider if you had hundreds/thousands of companies with hundreds/thousands of location codes. You end up nesting a few arrays/loops/queries that really isn't as efficient as it could be.
As suggested above, you could save A LOT of processing (and memory) by passing the work on to SQL. It seems to me, you could run a single query using JOINS or such to retrieve what you want, instead of running it as you do. I haven't done a lot of SQL lately, but given the 2-table design you have, this should be feasible I imagine. (join the tables, and query what you need from the combined result; etc)
I re-structures my SQL with joins, that helped.
If the class structure is "not awful" then then problem must be due to the amount of rows returned.
I'm displaying memory_get_usage() after all the SQL and class sections and it's only 8MB.
But then I'm displaying 2500 rows on on page and each row is using 10KB... so I think this is just due to the amount of rows returned.
I've increased my memory_limit to 128MB and that's helped a lot. It just takes ages to load the page (if hitting 2k+ rows). But I suppose there's not much can be done about that unless I return fewer rows on different pages.
shmop functions [php.net]
thread on bugs.php.net about array sizes [bugs.php.net]
