homepage Welcome to WebmasterWorld Guest from 54.211.73.232
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Code, Content, and Presentation / PHP Server Side Scripting
Forum Library, Charter, Moderators: coopster & jatar k

PHP Server Side Scripting Forum

    
PHP arrays - max size, max amount
unless you know a better way to do this report
Frank_Rizzo

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 554 posted 9:03 pm on Aug 24, 2002 (gmt 0)

I need to run some reports against a flat file database. More on this in a second. What I want to know is, is there a maximum size and amount of arrays?

I have a flat file database (MySQL and stuff are not an option at the moment). I need to run off some reports which subtotals categories.

fields:

fruit,location,colour,size,good crop,price

data:

banana,south,yellow,big,no,2.5
banana,north,yellow,big,yes,2.5
apple,south,green,small,no,5
banana,west,green,small,yes,3.2
plum,south,blue,medium,yes,1.8

Ok, its a lot more complex than that. Like 10,000 lines of data.

What I need to do is to count the number of good crops for each fruit, also summarise the total price for each fruit. I was thinking of an array

banana[0] = the total number of bananas
banana[1] = the total number of good crop
banana[2] = the total price paid

so in this example

banana[0] = 3
banana[1] = 2
banana[2] = 8.2
apple[0] = 1
apple[1] = 0
apple[2] = 5

etc.

So if you are still awake do you think this is a good idea?

----
simplistically:

open the database
...loop for each 10,000 records
.....if fruit = banana
.......banana[0] ++
.......if crop is good
.........banana[1]++
.......}
.......banana[2] =. price
.....}
.....if fruit = apple
.......apple[0] ++
.......if crop is good
.........apple[1]++
.......}
.......apple[2] =. price
.....}
...}

print "there were $banana[0] bananas. $banana[1] were good crop. You paid a total of $banana[2] for them"
print "there were $apple[0] apples. $apple[1] were good crop. You paid a total of $apple[2] for them"

-----

I could create a function rather than have a dozen or so if fruit = and prints in the report.

But I'm wondering if I'm going to hit on some memory issues. There will be about 30 fruit in total.

Driving me nuts trying to work this one out. I have done it easily offline using Foxpro no problem, but that is easy because I can use the indexes to index on the fruit,good crop,price etc. Unfortunately the flat file database does not have indexing.

What you reckon guys? Free bananas for anyone who can help me out here :)

 

Knowles

10+ Year Member



 
Msg#: 554 posted 9:24 pm on Aug 24, 2002 (gmt 0)

I would think 30 is a bit much it would probably slow down the page. I coudlnt find anyting on PHP: Array Funtions [php.net] that said there was a max amount. Have you already created the script? Is it working and what kind of time does it take? I did see array_count_values [php.net] I am not sure this would work but it may help some. I would be interested in seeing your final script as well I have always wondered about this.

jatar_k

WebmasterWorld Administrator jatar_k us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 554 posted 6:44 am on Aug 25, 2002 (gmt 0)

Frank_Rizzo,

I have been thinking about this since you posted it.

As Knowles said it will probably not make a very fast loading page. I don't know of any array limits either. It just doesn't seem like a very tight way to go about it. I looked through filesystem functions, array functions and a bunch of other things but couldn't really come up with a better solution.

I thought of the possibilty of loading the whole file and then using regex to look for occurences of various things in the file, might work. You will have to tighten up that loop though.

I am not overly familiar with using flat files and I can think of a million ways to do it with sql. Not totally sure but I think for now you may have the answer.

andreasfriedrich

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 554 posted 7:49 am on Aug 25, 2002 (gmt 0)

You might want to try using some kind of berkely db format and php´s database abstraction layer (http://www.php.net/manual/en/ref.dba.php). This will speed up your code a lot since the data will be stored in a binary hash or btree format.

If your php wasn´t build with berkeley db support then you might consider not using your flat file db for each computation but an intermediate format instead. Use the algo you came up with to create your arrays from your db, then serialize these arrays and store them in a file. Next time you need to do some computation read the file, unserialize the data and you have your arrays back.

Those are just some thoughts that came to my mind when reading the previous posts. Not really sure whether they help since I don´t really know what you need this for.

Frank_Rizzo

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 554 posted 11:11 pm on Aug 25, 2002 (gmt 0)

I'm actually using dBase flat files. This works well with PHP but the lack of indexing functions is the problem. I'll take a look at that Berkley stuff.

I guess MySQL is the answer but I dont have the time to sort this out yet.

I've done some tests using arrays. It works fine. Speed is only a problem with some of the bigger databases - I was getting PHP exceed a 30 second limit. The answer to this is to limit the size of the database the users can access.

The ram caching is good and the server utilization is ticking over nicely. The acid test is when dozens of the users are running the reports.

Anyway, this is how I have implemented it. I'll show a quick overview of the code and then snippets of the actual code itself. It looks kinda over the top but thats the only way I know how to implement this.

re: creating summary reports for a flat file database where users can filter the report for dozens of options.

1. loop through database and identify matches with user form
2. run check_vars function for each summary report. Fill array with results
3. perform string formatting on each array
4. Display each array.

------
#loop through the database and find matches

for($i=1; $i <= $count; $i++) {

$rec = dbase_get_record($fid, $i);

if($rec[0] >= $month_min && $rec[0] <= $month_max &&
.
.
. 15 other conditions to be checked here.
.
(isset($spposa) ¦¦ preg_match("/$rec[21]/", "$s_sppos"))) {

#This calls the check_vars func for each summary report. the y93 is the name of the array, the $rec[22] the record in DB to match.

#There are about 30 summaries in total, each summary will have 3-10 entries, therefore approx 180 arrays will be created.

check_vars("1993", "y93", "$rec[22]");
check_vars("1994", "y94", "$rec[22]");
check_vars("1995", "y95", "$rec[22]");
check_vars("1996", "y96", "$rec[22]");
check_vars("1997", "y97", "$rec[22]");
check_vars("1998", "y98", "$rec[22]");
check_vars("1999", "y99", "$rec[22]");
check_vars("2000", "y00", "$rec[22]");
check_vars("2001", "y01", "$rec[22]");
check_vars("2002", "y02", "$rec[22]");

check_vars("A", "ssnA", "$rec[23]");
check_vars("B", "ssnB", "$rec[23]");
check_vars("C", "ssnC", "$rec[23]");
check_vars("D", "ssnD", "$rec[23]");

check_vars("A", "lrA", "$rec[29]");
check_vars("B", "lrB", "$rec[29]");
check_vars("C", "lrC", "$rec[29]");
check_vars("D", "lrD", "$rec[29]");
check_vars("E", "lrE", "$rec[29]");
check_vars("F", "lrF", "$rec[29]");

}
}

#html code here

#next some string formating of each array is performed

calc_vars("y93");
calc_vars("y94");
calc_vars("y95");
calc_vars("y96");
calc_vars("y97");
calc_vars("y98");
calc_vars("y99");
calc_vars("y00");
calc_vars("y01");
calc_vars("y02");

calc_vars("ssnA");
calc_vars("ssnB");
calc_vars("ssnC");
calc_vars("ssnD");

calc_vars("lrA");
calc_vars("lrB");
calc_vars("lrC");
calc_vars("lrD");
calc_vars("lrE");
calc_vars("lrF");

#more html here - the report main headings.

#each array is then displayed

display_vars("1993", "y93");
display_vars("1994", "y94");
display_vars("1995", "y95");
display_vars("1996", "y96");
display_vars("1997", "y97");
display_vars("1998", "y98");
display_vars("1999", "y99");
display_vars("2000", "y00");
display_vars("2001", "y01");
display_vars("2002", "y02");

display_vars("Winter", "ssnA");
display_vars("Spring", "ssnB");
display_vars("Summer", "ssnC");
display_vars("Autumn", "ssnD");

display_vars("1 to 7 days", "lrA");
display_vars("8 to 14 days", "lrB");
display_vars("15 to 28 days", "lrC");
display_vars("29 to 56 days", "lrD");
display_vars("57+ days", "lrE");
display_vars("First Time Out", "lrF");

#end html stuff here

#ok, this is the meat. Need to use a log of ${"var"} stuff.

function check_vars($value, $array, $field) {

global ${"$array"}, $rec;

if($field == $value) {
${"$array"}[0]++;
if($rec[10] == 1) {
${"$array"}[1]++;
${"$array"}[2] = ${"$array"}[2] + $rec[11];
}
}

}

function calc_vars($array) {

global ${"roi$array"}, ${"pro$array"}, ${"wpc$array"}, ${"iv$array"}, ${"rc$array"}, ${"$array"};
global $t_wins, $t_runs;

if(${"$array"}[0] > 0) {
${"pro$array"} = ${"$array"}[2] + ${"$array"}[1] - ${"$array"}[0];
${"roi$array"} = 100 * ${"pro$array"} / ${"$array"}[0];
${"iv$array"} = (${"$array"}[1] / $t_wins) / (${"$array"}[0] / $t_runs);
${"wpc$array"} = sprintf("%8.1f", 100 * ${"$array"}[1] / ${"$array"}[0]) . '%';
}
${"roi$array"} = sprintf("%8.1f", ${"roi$array"}) . '%';
${"pro$array"} = '&pound;' . trim(sprintf("%8.2f", ${"pro$array"}));
${"iv$array"} = sprintf("%8.2f", ${"iv$array"});

}

function display_vars($value, $array) {

global ${"roi$array"}, ${"pro$array"}, ${"wpc$array"}, ${"iv$array"}, ${"rc$array"};
global ${"$array"};

if(${"$array"}[0] > 0) {
print '<TR><TD>&nbsp;</TD><TD CLASS="' . ${"rc$array"} . '">' . $value . '</TD><TD CLASS="' . "r${"rc$array"}" . '">' . ${"$array"}[1] . '</TD><TD CLASS="' . "r${"rc$array"}" . '">' . ${"$array"}[0] . '</TD><TD CLASS="' . "r${"rc$array"}" . '">' . trim(${"wpc$array"}) . '</TD><TD CLASS="' . "r${"rc$array"}" . '">' . trim(${"iv$array"}) . '</TD><TD CLASS="' . "r${"rc$array"}" . '">' . trim(${"pro$array"}) . '</TD><TD CLASS="' . "r${"rc$array"}" . '">' . trim(${"roi$array"}) . '</TD></TR>';
}

}
#end

Looks a bit long winded but it works. What concerns me is that about 180 arrays are going to be created. The speed is not a problem yet. But the memory could be.

If you want to see it run, see profile. I'm only have 3 summary reports so it could be that the script will be 10 times slower when all 30 implemented.

jatar_k

WebmasterWorld Administrator jatar_k us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 554 posted 11:23 pm on Aug 25, 2002 (gmt 0)

have you thought about 1 multi dimesional array?

Instead of 180 arrays you could have one array. There are a lot of good array functions [php.net] that may be useful.

I am not 100% sure but you would probably save mem that way. You would also have to rewrite that beast. It is worth looking into though.

Frank_Rizzo

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 554 posted 6:27 pm on Aug 26, 2002 (gmt 0)

multidimension array sounds good. That should make it easier to maintain too.

I'm now upto 63 arrays of 3 cells each. I guess that is not much in actual 'bytes' total but I'm just concerned about 'resources' or something.

I trust those clever people behind PHP and Linux for making it resilient though. If this was using M$ products I'm sure many a blue screen would be displayed.

I'm just thinking if there is a way to see what is going on, like some kind of debug mode for PHP?

I tried TOP and what is worrying is the CPU States. This whacks upto 99% when the script is running. But another report I have which shows load average is just fine - usually around 0.5.

jatar_k

WebmasterWorld Administrator jatar_k us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 554 posted 6:32 pm on Aug 26, 2002 (gmt 0)

I can't remember off the top of my head how to do the debug mode.

Just go to php.net and search the "whole site", default is function list, for debug or debug mode. There were 122 docs when I did it, it's in there somewhere.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Code, Content, and Presentation / PHP Server Side Scripting
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved