I just notice inktomisearch bot index some area on my site that I don't want to. Now I changed the robot.txt to exclude that area, but what about the existing index that inktomi has? will they remove automatically based on the new robot.txt?
10:46 pm on Aug 17, 2003 (gmt 0)
I'd guess that Ink will remove your excluded pages faster if you are a paid subscriber. Otherwise, it may take awhile, possibly several months. It all depends on how often Slurp visits your site and re-indexes the disallowed pages.