Ink/Slurp

why is slurp the only one.....

[The Contractor]

Why does slurp do this but no other crawler. Every link and subdirectory has the trailing slash but slurp always trys to crawl without the trailing slash - it then trys a second time and gets it right. Example below:

66.196.72.51 - - [29/Dec/2002:13:33:02 -0500] "GET /example/myexampledirectory HTTP/1.0" 301 279 "-" "Mozilla/5.0 (Slurp/cat; [e-mail]; edited out link)"

66.196.72.51 - - [29/Dec/2002:13:33:05 -0500] "GET /example/myexampledirectory/ HTTP/1.0" 200 16737 "-" "Mozilla/5.0 (Slurp/cat; [e-mail]; edited out link)"

[Brett_Tabke]

Ink has a great dupe content finder. That is one of the things they do to look for the index file.

[The Contractor]

Brett,
Thanks ;)
That explains it then...