Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: mademetop
I can see hitting the root once in awhile to check for dead links, but today I had FIVE different inktomi spiders hit my site, each requesting either the robots.txt, the root directory or both (one spider sent TWO requests EACH for robots.txt & /)... and NOTHING ELSE! I can't imagine they need to verify a site's exitence 10 times a day...
WHAT'S THE POINT????
Please, someone enlighten me...
Now that would be cool if it only happened once every month or so, but they come in every day, several thousands of times day after day.
Small suggestion, try changing your index page and see if it hits any of your sub-pages.
The repititious spidering of Ink is baffeling to me. Day-in day-out with a different agent name. It just makes me wonder what they are really doing with that data. like repackaging data for someone else privatly (govt? has always been my guess with Ink - they have the highest world coverage of any se).
They have actually over loaded my server a couple of times and have made me have reconfigure apache to tighten down with 'MaxClients' restrictions.
Actually, the day I posted that message, I had changed my index page... actually added new text links to my main sub-pages in addition to my navigation buttons. Still nothing but a hit on my root & index page.
The bait & switch theory makes the most sense to me, but it does seem like it would be a terrific waste of their company resources to spend so much time doing it.