Msg#: 3476768 posted 4:06 pm on Oct 13, 2007 (gmt 0)
I hope someone here could give me some explanation of this.
My affiliate's spider visits my site every night to check if their ads are placed on my site. Just recently, they are saying that they were not able to crawl through even though my server log says http code 200 next to their spider log. It appears to be that the page has been downloaded as well.
How could this be possible? Are they pulling one on me?
Msg#: 3476768 posted 4:54 pm on Oct 13, 2007 (gmt 0)
Perhaps your robots.txt file is invalid, since they appear to have downloaded it and received a 200-OK server response code. I'd suggest you use a robots.txt validator to check it. There are several on-line, as well as the one available in Google Webmaster Tools.
robots.txt is not at all a 'freestyle' file, and you need to be extremely careful about spelling, syntax, record order, blank lines after every record, and robot-specific record order. See A Standard for Robot Exclusion [robotstxt.org], and read it very carefully.