homepage Welcome to WebmasterWorld Guest from 54.197.183.230
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
http code 200 mystery
is the page served or not!?
dailyclicker

5+ Year Member



 
Msg#: 3476768 posted 4:06 pm on Oct 13, 2007 (gmt 0)

I hope someone here could give me some explanation of this.

My affiliate's spider visits my site every night to check if their ads are placed on my site. Just recently, they are saying that they were not able to crawl through even though my server log says http code 200 next to their spider log. It appears to be that the page has been downloaded as well.

How could this be possible? Are they pulling one on me?

BTW: I have "disalow" in my robot text but they are for my javascript, stylesheet and image folders which has nothing to do with the ad on the page.

dailyclicker

 

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3476768 posted 4:54 pm on Oct 13, 2007 (gmt 0)

Perhaps your robots.txt file is invalid, since they appear to have downloaded it and received a 200-OK server response code. I'd suggest you use a robots.txt validator to check it. There are several on-line, as well as the one available in Google Webmaster Tools.

robots.txt is not at all a 'freestyle' file, and you need to be extremely careful about spelling, syntax, record order, blank lines after every record, and robot-specific record order. See A Standard for Robot Exclusion [robotstxt.org], and read it very carefully.

Jim

dailyclicker

5+ Year Member



 
Msg#: 3476768 posted 5:14 pm on Oct 13, 2007 (gmt 0)

<quote>I'd suggest you use a robots.txt validator to check it.</quote>

Thanks very much Jim. I didn't know that order of things could mess it up. I've swapped some lines over so there are no mistakes so I'll see what they'll say.

dailyclicker

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved