homepage Welcome to WebmasterWorld Guest from 54.145.183.169
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
frames and robots.txt
snowfishin

10+ Year Member



 
Msg#: 68 posted 8:28 pm on Feb 3, 2003 (gmt 0)

say i have a frame page and it is

/page/index.htm

one frame in this page is
/dir1/bottom.htm

and in my robots.txt file I disallow
googlebot from /dir1/

does /page/index.htm get indexed?

 

JamesR

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 68 posted 9:05 pm on Feb 3, 2003 (gmt 0)

I think so since it is in another directory that you have allowed. You can put code on that page to try and keep it from being indexed (not sure which robots obey it)

<meta name="robots" content="noindex">

Macguru

WebmasterWorld Senior Member macguru us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 68 posted 9:10 pm on Feb 3, 2003 (gmt 0)

Hi snowfishin,

JamesR is right. SE looks at individual pages called by framesets as what they are, individual pages.

Any good stuff for se spiders to eat in your /page/index.htm page?

Most se will abide by <meta name="robots" content="noindex"> command.

snowfishin

10+ Year Member



 
Msg#: 68 posted 9:15 pm on Feb 3, 2003 (gmt 0)

slurp chokes on the meta noindex and when I disallow the slurp from /dir1/

/page/index.htm is disallowed by both of these cases I was wondering if googlebot was as stupid as slurp

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved