If this is a Linux box, using the top command from the shell will tell you which processes are hogging your memory (press the < or > on your keyboard until the list is sorted by %MEM). First and foremost, look carefully for processes you don't need, and disable those. Then monitor the process list for a while to see if there are any processes that take up a disproportionate amount of memory. Also check your cron tab for any jobs that are non-essential.
If you tell us a bit more about the server and website environment, we might be able to look into this in more detail.
What is happening here is it searches the entire file system twice and is extremely resource intensive. See post #4274745 in this thread [webmasterworld.com] for one efficient way to fix this. It's a pretty good place to start. :-)