Welcome to WebmasterWorld Guest from 54.147.134.218

Forum Moderators: goodroi

Message Too Old, No Replies

http code 200 mystery

is the page served or not!?

     
4:06 pm on Oct 13, 2007 (gmt 0)

New User

5+ Year Member

joined:Oct 13, 2007
posts:2
votes: 0


I hope someone here could give me some explanation of this.

My affiliate's spider visits my site every night to check if their ads are placed on my site. Just recently, they are saying that they were not able to crawl through even though my server log says http code 200 next to their spider log. It appears to be that the page has been downloaded as well.

How could this be possible? Are they pulling one on me?

BTW: I have "disalow" in my robot text but they are for my javascript, stylesheet and image folders which has nothing to do with the ad on the page.

dailyclicker

4:54 pm on Oct 13, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Perhaps your robots.txt file is invalid, since they appear to have downloaded it and received a 200-OK server response code. I'd suggest you use a robots.txt validator to check it. There are several on-line, as well as the one available in Google Webmaster Tools.

robots.txt is not at all a 'freestyle' file, and you need to be extremely careful about spelling, syntax, record order, blank lines after every record, and robot-specific record order. See A Standard for Robot Exclusion [robotstxt.org], and read it very carefully.

Jim

5:14 pm on Oct 13, 2007 (gmt 0)

New User

5+ Year Member

joined:Oct 13, 2007
posts:2
votes: 0


<quote>I'd suggest you use a robots.txt validator to check it.</quote>

Thanks very much Jim. I didn't know that order of things could mess it up. I've swapped some lines over so there are no mistakes so I'll see what they'll say.

dailyclicker

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members