it keeps requesting for the robots.txt what should i add in the robots file to make inktomi index my pages?
seth_wilde
2:58 pm on Apr 15, 2003 (gmt 0)
You don't need to add anything (or even have a robot.txt). Just make sure that if you do have a robot.txt it's written properly and not blocking spiders.
