Shell script optimization

Shell commands within a PHP script

[iceman22]

Now I have a simple PHP search script, that uses shell commands mainly to perform the search.

Now after I have exploded the filter in and filter out variables into arrays ($elements is the max number of search terms used), here's the code:

for ($i = 0; $i <= $elements; $i++) {
if (!(empty($search[$i])) ) {
$shell = ($shell." ¦ grep -i ".$search[$i]);
}
if (!(empty($notsearch[$i])) ) {
$shell = ($shell." ¦ grep -i -v ".$notsearch[$i]);
}
}

For the shell part, I've tried three different commands, the fastest at the top.

$searchresults = `head -[totalno. of lines] [path] $shell`;
$searchresults = `sed '' [path] $shell`;
$searchresults = `grep \ [path] $shell`;

Basically, what I need is a simple command that'll do nothing, so I can tack on the " ¦ grep -i $seachstring" search terms. I'm trying to keep the script as fast as possible, it is already slow.

Thanks.

[mcavic]

I'm not sure what you're trying to do, but when you're using pipes, cat is a command that does nothing.

In other words, "grep hello" is equal to "cat ¦ grep hello".

[quesera]

I'm not sure what you're trying to do either.

cat will work, as mcavic mentioned, as long as you're directing the searched content to the standard input (STDIN) of the pipe, which you seem to be..

But you might not need the null command at all -- many shell commands default to reading from STDIN. grep, cat, sed all do. Commands that don't will usually take "-" as a file argument to read from STDIN.

If you can avoid the extra overhead, you should...but if you can't, cat is probably the best choice. It's small and very fast.

[iceman22]

Thanks for the replies, cat was exactly what I was looking for. Unfortunately I really need to lose the overhead, using "head" is a little faster, which was surprising.

If the file I am searching was 100 lines, names "file.txt" in the root my server, this is what it'd look like if I searched to filter in 'one' 'two' 'three' and to filter out 'four'.

"head -100 /home/user/public_html/file.txt ¦ grep -i one ¦ grep -i two ¦ grep -i three ¦ grep -i -v four"

The way it's coded is to just add a pipe for each filter in or out term, in the form " ¦ grep -i [search term]".

I don't know enough about STDIN to do without the first pipe.

[mcavic]

The problem with speed is that you're calling the shell from PHP, which shouldn't be necessary just for searching.
PHP has fast search functions.

[mcavic]

Try this:

$file = "myfile.txt"; 
$fp = fopen("$file", "r"); 
# Read lines from the file 
while ($line = fgets($fp)) { 
    $m = 0; 
    # If it matches any of these, then it's a match 
    foreach ($search as $s) { 
        if (preg_match("/$s/", $line)) { 
            $m = 1; 
            } 
        } 
    # If it matches any of these, then it's not 
    # a match, afterall 
    foreach ($notsearch as $s) { 
        if (preg_match("/$s/", $line)) { 
            $m = 0; 
            } 
        } 
    # Display it if it is a match 
    if ($m == 1) { 
        print "$line"; 
        } 
    } 
fclose($fp); 

[iceman22]

Thanks for the nice snippet of code, I had to add an if statement because if the $notsearch arrays are empty it will yield a match. It took a while figuring out why m was always returning a zero value. Surprisingly this is actually slower than using grep. I did a search with grep and it took 2.65 seconds, replacing just the search code with the following and it took 8.25 seconds. It also searches OR for the search terms, rather than AND. The search file is 20MB with hundreds of thousands of lines, so I'm not sure what would be the fastest method of searching it.

$file = "file.txt";
$fp = fopen("$file", "r");
while ($line = fgets($fp)) {
$m = 0;
foreach ($search as $s) {
if (preg_match("/$s/i", $line)) {
$m = 1;
}
}
foreach ($notsearch as $s) {
if (!(empty($s)) ) {
if (preg_match("/$s/i", $line)) {
$m = 0;
}
}
}
if ( $m == 1 ) {
$searchresults = $searchresults.$line;
}
}
fclose($fp);

[mcavic]

I used to have a web database like that. Mine was only about 1.5 megs, but it was still slow with multiple search terms.

MySQL is the way to go for a database that big. It integrates very nicely with PHP too.

[iceman22]

I knew I'd have to use MySQL sooner or later. I parsed the database file and I'll leave it importing/uploading overnight, it takes a while.

I've never used MySQL before what kind of PHP code would you use for the fastest possible search for multiple terms?

Sorry this is now getting even more off-topic.