Welcome to WebmasterWorld Guest from 126.96.36.199
Forum Moderators: goodroi
My affiliate's spider visits my site every night to check if their ads are placed on my site. Just recently, they are saying that they were not able to crawl through even though my server log says http code 200 next to their spider log. It appears to be that the page has been downloaded as well.
How could this be possible? Are they pulling one on me?
robots.txt is not at all a 'freestyle' file, and you need to be extremely careful about spelling, syntax, record order, blank lines after every record, and robot-specific record order. See A Standard for Robot Exclusion [robotstxt.org], and read it very carefully.