| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
|Subscribe to WebmasterWorld|
|frames and robots.txt|
| 8:28 pm on Feb 3, 2003 (gmt 0)|
say i have a frame page and it is
one frame in this page is
and in my robots.txt file I disallow
googlebot from /dir1/
does /page/index.htm get indexed?
| 9:05 pm on Feb 3, 2003 (gmt 0)|
I think so since it is in another directory that you have allowed. You can put code on that page to try and keep it from being indexed (not sure which robots obey it)
<meta name="robots" content="noindex">
| 9:10 pm on Feb 3, 2003 (gmt 0)|
JamesR is right. SE looks at individual pages called by framesets as what they are, individual pages.
Any good stuff for se spiders to eat in your /page/index.htm page?
Most se will abide by <meta name="robots" content="noindex"> command.
| 9:15 pm on Feb 3, 2003 (gmt 0)|
slurp chokes on the meta noindex and when I disallow the slurp from /dir1/
/page/index.htm is disallowed by both of these cases I was wondering if googlebot was as stupid as slurp
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
© Webmaster World 1996-2013 all rights reserved