Welcome to WebmasterWorld Guest from 54.146.248.111

Message Too Old, No Replies

Is there such thing as html markup to tell googlebot what to crawl

and what not to?

     

newbies

4:22 am on Mar 28, 2006 (gmt 0)

10+ Year Member



I wonder if there is such thing as document makeup you can put in your html file to tell google bot what part of a file to crawl and what part to ignor.

nedguy

10:56 am on Mar 28, 2006 (gmt 0)

5+ Year Member



sadly not.

(though there is some code that tells google which part of a page it should use to judge what adsense adverts to display)

frox

11:07 am on Mar 28, 2006 (gmt 0)

10+ Year Member



Yes there is: GoogleBot will crawl everything from the <HTML> to the </HTML> :-)

nickied

1:22 pm on Mar 28, 2006 (gmt 0)

10+ Year Member



newbies:

is there such thing as html markup to tell googlebot what to crawl

no. though I always get a chuckle when i see:

'robots revisit' or 'robots all' in the metas.

and what not to?

robots.txt can help exclude 'good' robots. search this site for robots.txt; you should end up at formum 93.

If you look at Brett's robot.txt about 20 lines down you'll see:

User-agent: *
Disallow: /

Be very careful with this; Brett has chosen to exclude everything. Read up on it before implementing.

Phil_Payne

3:01 pm on Mar 28, 2006 (gmt 0)

10+ Year Member



> sadly not.

Though it is a shame. I have one page in particular that looks as if it is keyword-stuffed in the open text - I'd love a <noindex> </noindex> markup tag.

What about a server-side include for the stuff that's not to be indexed?

tedster

3:08 pm on Mar 28, 2006 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



SSI is inserted into the document before it leaves the server. The bot has no idea how the page was generated, so it all gets indexed, SSI or not.

Kufu

5:29 pm on Mar 28, 2006 (gmt 0)

10+ Year Member



You can use JavaScript to write that section of the text. Since the spiders don't have JavaScript activated that part will be 'invisible' to them.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month