Control indexing of content by se

[johnn]

Hi,
I's newbie to se. I would like to know how to control what part of the webpage can be indexed by search engines, in another word, how to make some content of a webpage visible or invisible to search engines?

Thanks,
John

[martin]

Use <meta name="robots" content="<pick from the list below>" /> in the head of your document.

Values for the robots tag:
noindex: don't index the page (also index)
nofollow: don't follow links on the page (also follow)

These two can be combined, separated with a comma. I've also seen the value "all" - it should mean index, follow and all other if available.

[Marcia]

martin, is this:

content="<pick from the list below>" />

in the W3C specifications? Is there an official reference for it?

I've never seen anything related, even at Google's site in their webmaster instructions. I don't believe that would be honored by Googlebot.

[Key_Master]

The Robots META tag resource.
[robotstxt.org...]

Googlebot will honor this robot meta tag.

[johnn]

Thanks. I'm not sure what to do...

Well, say if a client would like to display on home page a product of the month special sale, and then remove it when sales ends, and put another product on sale, since this content is kinda dynamic, do search engines index this part of a page? I wish it doesn't see this part of page but the rest. May I have an example on how to take care of this?

Thanks,
John

[deejay]

johnn, if you want the search engines to simply not list a particular page at all, use the meta tags or robots file.

If you are talking about not wanting them to index that particular PART of the page, I don't think I'd worry about trying to hide that sort of thing.

If anything I want the search engines to notice that that information is changing regularly, and maybe they'll deem me worthy of more regular indexing. :)

Ok, possibly the odd person will come to your page because they saw something relating to last week's special on a search engine, but i wouldn't say that's a enough of a negative to want to hide content.

[Marcia]

Thanks Key_Master, just thinking about the possibility of copying and pasting. :)

Is there a posssibility there can be some sections of the page that remain static and the part that changes be restricted to one section?

Aside from browser issues, would iframe work for the part that changes? I did a site search and here's one thread on it with a link to a tutorial:

[webmasterworld.com...]

[Grumpus]

IFRAME would do it, but you also lose traffic from older browsers.

I wouldn't worry too much about specials expiring - so long as you've got another one that shows up. Putting the start and end dates up will also help the customer find their way. If they go to the "live" page, they'd see the new special, and if they go to the cache page, they'd see the old one that clearly states the special runs through a certain date.

You might also try at "content-expires" meta tag (I think that's it - I don't have my tag list handy - it's something along those lines). That SHOULD make google put the page in the index and pull it once the special passes - unfortunately, with the rate of google's update, unless you manage to get a "fresh" tag, the content is likely to expire before it even makes the index.

Regardless, though - if a customer is looking for something, they're probably actually looking to buy it. So what if they missed last week's special? If they need it, they'll buy it at full price - just make sure they can find it and click the "BUY ME!" button.

G.

[martin]

>IFRAME would do it, but you also lose traffic from older browsers.

And not only from old browsers, there're many security issues with frames, and lots of people dislike them. I am using Opera and it has an option to disable frames ;-)

>unless you manage to get a "fresh" tag
Btw, what's a fresh tag and how does one get it?