| Welcome to WebmasterWorld Guest from 126.96.36.199 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|Is there such thing as html markup to tell googlebot what to crawl|
and what not to?
I wonder if there is such thing as document makeup you can put in your html file to tell google bot what part of a file to crawl and what part to ignor.
(though there is some code that tells google which part of a page it should use to judge what adsense adverts to display)
Yes there is: GoogleBot will crawl everything from the <HTML> to the </HTML> :-)
is there such thing as html markup to tell googlebot what to crawl
no. though I always get a chuckle when i see:
'robots revisit' or 'robots all' in the metas.
and what not to?
robots.txt can help exclude 'good' robots. search this site for robots.txt; you should end up at formum 93.
If you look at Brett's robot.txt about 20 lines down you'll see:
Be very careful with this; Brett has chosen to exclude everything. Read up on it before implementing.
> sadly not.
Though it is a shame. I have one page in particular that looks as if it is keyword-stuffed in the open text - I'd love a <noindex> </noindex> markup tag.
What about a server-side include for the stuff that's not to be indexed?
SSI is inserted into the document before it leaves the server. The bot has no idea how the page was generated, so it all gets indexed, SSI or not.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved