Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: open
Do you think that the SE's hidden bots, those who can detect us doing cloacking (using none referenced ip, hidden agent and hidden host) will still access the robots.txt?
volatilegx explained he also used some additional check of his own to identify specifics SE behave, such as not accessing the css files etc...I also read somewhere else that SE must always access the robot.txt file, Do you think this method based on the robot.txt 's access could be confident?
I'm just afraid this would specificly some behave they would not reproduct when they tried to hide themself, when the goal of their visit is to compare result between an hidden bots and their official bots?
any opinions on this matter?