Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

content removal via robot.txt



8:46 pm on Aug 17, 2003 (gmt 0)

10+ Year Member

I just notice inktomisearch bot index some area on my site that I don't want to. Now I changed the robot.txt to exclude that area, but what about the existing index that inktomi has? will they remove automatically based on the new robot.txt?


10:46 pm on Aug 17, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member


I'd guess that Ink will remove your excluded pages faster if you are a paid subscriber. Otherwise, it may take awhile, possibly several months. It all depends on how often Slurp visits your site and re-indexes the disallowed pages.



Featured Threads

Hot Threads This Week

Hot Threads This Month