homepage Welcome to WebmasterWorld Guest from 54.227.160.102
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Code, Content, and Presentation / Apache Web Server
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL & phranque

Apache Web Server Forum

    
mod_rewrite for slurp
frritchi

5+ Year Member



 
Msg#: 3253 posted 4:44 am on Apr 1, 2005 (gmt 0)

Hi all, is it possible to send Slurp a 404 when requesting domain.com but send the normal page when requesting www.domain.com?

thanks
Frank

 

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3253 posted 5:07 am on Apr 1, 2005 (gmt 0)

Frank,

Yes, but Slurp never was any good with 404s -- it just keeps asking for them, sometimes for more than a year. A better bet is to redirect all requests for your non-www domain to your www domain, and prevent this problem from happening in the first place. Thankfully, slurp understands 301s better than it understands 404s and 410s.

Jim

dcrombie

10+ Year Member



 
Msg#: 3253 posted 10:03 am on Apr 1, 2005 (gmt 0)

I see Slurp generating 2-300+ 301's each day on just one of our servers, and it's been doing that for at least a year with no sign of learning.

I complained at one point when it made >100 requests for one robots.txt file in 24 hours - they said it was 'unavoidable' because the spider is 'distributed'. How can you reason with that? ;)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Code, Content, and Presentation / Apache Web Server
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved