Welcome to WebmasterWorld Guest from 126.96.36.199 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
mod_rewrite for slurp frritchi msg:1515995 4:44 am on Apr 1, 2005 (gmt 0) Hi all, is it possible to send Slurp a 404 when requesting domain.com but send the normal page when requesting www.domain.com?
jdMorgan msg:1515996 5:07 am on Apr 1, 2005 (gmt 0)
Yes, but Slurp never was any good with 404s -- it just keeps asking for them, sometimes for more than a year. A better bet is to redirect all requests for your non-www domain to your www domain, and prevent this problem from happening in the first place. Thankfully, slurp understands 301s better than it understands 404s and 410s.
dcrombie msg:1515997 10:03 am on Apr 1, 2005 (gmt 0)
I see Slurp generating 2-300+ 301's each day on just one of our servers, and it's been doing that for at least a year with no sign of learning.
I complained at one point when it made >100 requests for one robots.txt file in 24 hours - they said it was 'unavoidable' because the spider is 'distributed'. How can you reason with that? ;)