http code 200 mystery

is the page served or not!?

[dailyclicker]

I hope someone here could give me some explanation of this.

My affiliate's spider visits my site every night to check if their ads are placed on my site. Just recently, they are saying that they were not able to crawl through even though my server log says http code 200 next to their spider log. It appears to be that the page has been downloaded as well.

How could this be possible? Are they pulling one on me?

BTW: I have "disalow" in my robot text but they are for my javascript, stylesheet and image folders which has nothing to do with the ad on the page.

dailyclicker

[jdMorgan]

Perhaps your robots.txt file is invalid, since they appear to have downloaded it and received a 200-OK server response code. I'd suggest you use a robots.txt validator to check it. There are several on-line, as well as the one available in Google Webmaster Tools.

robots.txt is not at all a 'freestyle' file, and you need to be extremely careful about spelling, syntax, record order, blank lines after every record, and robot-specific record order. See A Standard for Robot Exclusion [robotstxt.org], and read it very carefully.

Jim

[dailyclicker]

<quote>I'd suggest you use a robots.txt validator to check it.</quote>

Thanks very much Jim. I didn't know that order of things could mess it up. I've swapped some lines over so there are no mistakes so I'll see what they'll say.

dailyclicker