Forum Moderators: open
For March 05 my log shows 47 robot.txt hits and 17 "other" hits by Slurp.
Is it common for Slurp to request robot.txt so many times?
relevant metas:
META NAME="ROBOTS" CONTENT="INDEX, FOLLOW"
META NAME="REVISIT-AFTER" CONTENT="30 DAYS"
META NAME="ROBOTS" CONTENT="NOARCHIVE"