homepage Welcome to WebmasterWorld Guest from 54.235.227.60
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Google & Robots.txt
Basic Robots.txt question
robdawg




msg:705545
 10:42 pm on Jun 24, 2005 (gmt 0)

In your robots .txt file can you specify certain directories that you do not want the spider searching? Also if someone has linked to a page in a directory that you have specified the spider not to see, will the spider check for the robots.txt first or will it follow the link and try to index the page anyway?

Thanks

Rob.

 

robdawg




msg:705546
 4:17 am on Jun 25, 2005 (gmt 0)

Any one?

Clint




msg:705547
 9:29 am on Jun 25, 2005 (gmt 0)

This is covered in many other posts in the robots.txt forum, and you should post this in the correct forum, which is the robots.txt forum. The "My threads" area under our control panel area here isn't working correctly. I posted this info before and recently, but the thread isn't showing up! (what's up with that mods?) It was in the robots.txt forum.

Ok after some searching I finally found it, message #2.
[webmasterworld.com...]

I can't answer your second question.

dmorison




msg:705548
 10:42 am on Jun 25, 2005 (gmt 0)

Also if someone has linked to a page in a directory that you have specified the spider not to see, will the spider check for the robots.txt first or will it follow the link and try to index the page anyway?

Any self-respecting robot will obey robots.txt regardless of how the link was found. Googlebot certainly does, and i'm sure every other mainstream search engine crawler does too. There would be little point in robots.txt if this were not the case.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved