homepage Welcome to WebmasterWorld Guest from 107.22.78.233
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
http code 200 mystery
is the page served or not!?
dailyclicker




msg:3476770
 4:06 pm on Oct 13, 2007 (gmt 0)

I hope someone here could give me some explanation of this.

My affiliate's spider visits my site every night to check if their ads are placed on my site. Just recently, they are saying that they were not able to crawl through even though my server log says http code 200 next to their spider log. It appears to be that the page has been downloaded as well.

How could this be possible? Are they pulling one on me?

BTW: I have "disalow" in my robot text but they are for my javascript, stylesheet and image folders which has nothing to do with the ad on the page.

dailyclicker

 

jdMorgan




msg:3476781
 4:54 pm on Oct 13, 2007 (gmt 0)

Perhaps your robots.txt file is invalid, since they appear to have downloaded it and received a 200-OK server response code. I'd suggest you use a robots.txt validator to check it. There are several on-line, as well as the one available in Google Webmaster Tools.

robots.txt is not at all a 'freestyle' file, and you need to be extremely careful about spelling, syntax, record order, blank lines after every record, and robot-specific record order. See A Standard for Robot Exclusion [robotstxt.org], and read it very carefully.

Jim

dailyclicker




msg:3476792
 5:14 pm on Oct 13, 2007 (gmt 0)

<quote>I'd suggest you use a robots.txt validator to check it.</quote>

Thanks very much Jim. I didn't know that order of things could mess it up. I've swapped some lines over so there are no mistakes so I'll see what they'll say.

dailyclicker

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved