Forum Moderators: open

Message Too Old, No Replies

Yahoo Crawls wrong URLs

Crawling Partial URLs

         

8cool

7:05 am on Feb 26, 2004 (gmt 0)

10+ Year Member



Yahoo! Slurp is crawling my website in wrong way. has anyone noticed this before

But I dont have this problem with other crawlers

Actual url:www.site.com/aa/bbb/widgets.html
It tries to crawl:www.site.com/aa/bbb/widg

66.196.90.49 - - [25/Feb/2004:23:41:35 -0700] "GET /site-map.html HTTP/1.0" 200 34860 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; [help.yahoo.com...]

66.196.90.49 - - [25/Feb/2004:23:41:33 -0700] "GET /aa/bbb/widg HTTP/1.0" 302 292 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; [help.yahoo.com...]