Welcome to WebmasterWorld Guest from 54.167.46.29

Forum Moderators: goodroi

Message Too Old, No Replies

frames and robots.txt

     
8:28 pm on Feb 3, 2003 (gmt 0)

New User

10+ Year Member

joined:Nov 26, 2002
posts:5
votes: 0


say i have a frame page and it is

/page/index.htm

one frame in this page is
/dir1/bottom.htm

and in my robots.txt file I disallow
googlebot from /dir1/

does /page/index.htm get indexed?

9:05 pm on Feb 3, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 23, 2000
posts:1277
votes: 0


I think so since it is in another directory that you have allowed. You can put code on that page to try and keep it from being indexed (not sure which robots obey it)

<meta name="robots" content="noindex">

9:10 pm on Feb 3, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 30, 2000
posts:3300
votes: 0


Hi snowfishin,

JamesR is right. SE looks at individual pages called by framesets as what they are, individual pages.

Any good stuff for se spiders to eat in your /page/index.htm page?

Most se will abide by <meta name="robots" content="noindex"> command.

9:15 pm on Feb 3, 2003 (gmt 0)

New User

10+ Year Member

joined:Nov 26, 2002
posts:5
votes: 0


slurp chokes on the meta noindex and when I disallow the slurp from /dir1/

/page/index.htm is disallowed by both of these cases I was wondering if googlebot was as stupid as slurp

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members