Forum Moderators: open
My site is visited regularly by slurp. The thing is that it is highly interested in my robots.txt file and the ratio of fetching the robots.txt file to any other file is about 1:1 (for every fetch of robots.txt it fetches another file… such a waist of time – it could be reading things more interesting then my robots.txt file :).
Is it a regular behavior of slurp or is something wrong with my site inktomi wise? Googlebot and the other bots manage to crawl fine.
Thanks for your help
