Welcome to WebmasterWorld Guest from 22.214.171.124 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Pubcon Platinum Sponsor
script that meets these needs warrior01 msg:1289304 12:00 am on Mar 27, 2003 (gmt 0) hi,
Does anyone know of a search engine that lets you restrict the directories to seach and that is php and GPL or GNU (free for redistribute)?
Thanks in advance,
mavherick msg:1289305 5:13 am on Mar 27, 2003 (gmt 0)
That one is a good start. PHPdig [ phpdig.toiletoine.net]
It's free (GNU/GPL) and it even seems (never used it myself, simply scanned the feature page) to follow the robots.txt file which should allow you to configure all the directory exclusion based on user-agent.
hope that helps