homepage Welcome to WebmasterWorld Guest from 54.147.196.159
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Hardware and OS Related Technologies / Linux, Unix, and *nix like Operating Systems
Forum Library, Charter, Moderators: bakedjake

Linux, Unix, and *nix like Operating Systems Forum

    
Shell script optimization
Shell commands within a PHP script
iceman22

5+ Year Member



 
Msg#: 1323 posted 2:11 am on Dec 25, 2004 (gmt 0)

Now I have a simple PHP search script, that uses shell commands mainly to perform the search.

Now after I have exploded the filter in and filter out variables into arrays ($elements is the max number of search terms used), here's the code:

for ($i = 0; $i <= $elements; $i++) {
if (!(empty($search[$i])) ) {
$shell = ($shell." ¦ grep -i ".$search[$i]);
}
if (!(empty($notsearch[$i])) ) {
$shell = ($shell." ¦ grep -i -v ".$notsearch[$i]);
}
}

For the shell part, I've tried three different commands, the fastest at the top.

$searchresults = `head -[totalno. of lines] [path] $shell`;
$searchresults = `sed '' [path] $shell`;
$searchresults = `grep \ [path] $shell`;

Basically, what I need is a simple command that'll do nothing, so I can tack on the " ¦ grep -i $seachstring" search terms. I'm trying to keep the script as fast as possible, it is already slow.

Thanks.

 

mcavic

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 1323 posted 5:00 am on Dec 25, 2004 (gmt 0)

I'm not sure what you're trying to do, but when you're using pipes, cat is a command that does nothing.

In other words, "grep hello" is equal to "cat ¦ grep hello".

quesera

10+ Year Member



 
Msg#: 1323 posted 8:14 am on Dec 25, 2004 (gmt 0)

I'm not sure what you're trying to do either.

cat will work, as mcavic mentioned, as long as you're directing the searched content to the standard input (STDIN) of the pipe, which you seem to be..

But you might not need the null command at all -- many shell commands default to reading from STDIN. grep, cat, sed all do. Commands that don't will usually take "-" as a file argument to read from STDIN.

If you can avoid the extra overhead, you should...but if you can't, cat is probably the best choice. It's small and very fast.

iceman22

5+ Year Member



 
Msg#: 1323 posted 1:15 pm on Dec 25, 2004 (gmt 0)

Thanks for the replies, cat was exactly what I was looking for. Unfortunately I really need to lose the overhead, using "head" is a little faster, which was surprising.

If the file I am searching was 100 lines, names "file.txt" in the root my server, this is what it'd look like if I searched to filter in 'one' 'two' 'three' and to filter out 'four'.

"head -100 /home/user/public_html/file.txt grep -i one grep -i two grep -i three grep -i -v four"

The way it's coded is to just add a pipe for each filter in or out term, in the form " grep -i [search term]".

I don't know enough about STDIN to do without the first pipe.

mcavic

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 1323 posted 3:29 pm on Dec 25, 2004 (gmt 0)

The problem with speed is that you're calling the shell from PHP, which shouldn't be necessary just for searching.
PHP has fast search functions.

mcavic

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 1323 posted 3:44 pm on Dec 25, 2004 (gmt 0)

Try this:


$file = "myfile.txt";
$fp = fopen("$file", "r");
# Read lines from the file
while ($line = fgets($fp)) {
$m = 0;
# If it matches any of these, then it's a match
foreach ($search as $s) {
if (preg_match("/$s/", $line)) {
$m = 1;
}
}
# If it matches any of these, then it's not
# a match, afterall
foreach ($notsearch as $s) {
if (preg_match("/$s/", $line)) {
$m = 0;
}
}
# Display it if it is a match
if ($m == 1) {
print "$line";
}
}
fclose($fp);

iceman22

5+ Year Member



 
Msg#: 1323 posted 5:34 am on Dec 26, 2004 (gmt 0)

Thanks for the nice snippet of code, I had to add an if statement because if the $notsearch arrays are empty it will yield a match. It took a while figuring out why m was always returning a zero value. Surprisingly this is actually slower than using grep. I did a search with grep and it took 2.65 seconds, replacing just the search code with the following and it took 8.25 seconds. It also searches OR for the search terms, rather than AND. The search file is 20MB with hundreds of thousands of lines, so I'm not sure what would be the fastest method of searching it.

$file = "file.txt";
$fp = fopen("$file", "r");
while ($line = fgets($fp)) {
$m = 0;
foreach ($search as $s) {
if (preg_match("/$s/i", $line)) {
$m = 1;
}
}
foreach ($notsearch as $s) {
if (!(empty($s)) ) {
if (preg_match("/$s/i", $line)) {
$m = 0;
}
}
}
if ( $m == 1 ) {
$searchresults = $searchresults.$line;
}
}
fclose($fp);

mcavic

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 1323 posted 5:48 am on Dec 26, 2004 (gmt 0)

I used to have a web database like that. Mine was only about 1.5 megs, but it was still slow with multiple search terms.

MySQL is the way to go for a database that big. It integrates very nicely with PHP too.

iceman22

5+ Year Member



 
Msg#: 1323 posted 3:16 pm on Dec 26, 2004 (gmt 0)

I knew I'd have to use MySQL sooner or later. I parsed the database file and I'll leave it importing/uploading overnight, it takes a while.

I've never used MySQL before what kind of PHP code would you use for the fastest possible search for multiple terms?

Sorry this is now getting even more off-topic.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Hardware and OS Related Technologies / Linux, Unix, and *nix like Operating Systems
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved