Is there such thing as html markup to tell googlebot what to crawl

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Is there such thing as html markup to tell googlebot what to crawl

and what not to?

newbies

4:22 am on Mar 28, 2006 (gmt 0)

I wonder if there is such thing as document makeup you can put in your html file to tell google bot what part of a file to crawl and what part to ignor.

nedguy

10:56 am on Mar 28, 2006 (gmt 0)

sadly not.

(though there is some code that tells google which part of a page it should use to judge what adsense adverts to display)

frox

11:07 am on Mar 28, 2006 (gmt 0)

Yes there is: GoogleBot will crawl everything from the <HTML> to the </HTML> :-)

nickied

1:22 pm on Mar 28, 2006 (gmt 0)

newbies:

is there such thing as html markup to tell googlebot what to crawl

no. though I always get a chuckle when i see:

'robots revisit' or 'robots all' in the metas.

and what not to?

robots.txt can help exclude 'good' robots. search this site for robots.txt; you should end up at formum 93.

If you look at Brett's robot.txt about 20 lines down you'll see:

User-agent: *
Disallow: /

Be very careful with this; Brett has chosen to exclude everything. Read up on it before implementing.

Phil_Payne

3:01 pm on Mar 28, 2006 (gmt 0)

> sadly not.

Though it is a shame. I have one page in particular that looks as if it is keyword-stuffed in the open text - I'd love a <noindex> </noindex> markup tag.

What about a server-side include for the stuff that's not to be indexed?

tedster

3:08 pm on Mar 28, 2006 (gmt 0)

SSI is inserted into the document before it leaves the server. The bot has no idea how the page was generated, so it all gets indexed, SSI or not.

Kufu

5:29 pm on Mar 28, 2006 (gmt 0)

You can use JavaScript to write that section of the text. Since the spiders don't have JavaScript activated that part will be 'invisible' to them.