homepage Welcome to WebmasterWorld Guest from 54.205.197.66
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Is there such thing as html markup to tell googlebot what to crawl
and what not to?
newbies




msg:707156
 4:22 am on Mar 28, 2006 (gmt 0)

I wonder if there is such thing as document makeup you can put in your html file to tell google bot what part of a file to crawl and what part to ignor.

 

nedguy




msg:707157
 10:56 am on Mar 28, 2006 (gmt 0)

sadly not.

(though there is some code that tells google which part of a page it should use to judge what adsense adverts to display)

frox




msg:707158
 11:07 am on Mar 28, 2006 (gmt 0)

Yes there is: GoogleBot will crawl everything from the <HTML> to the </HTML> :-)

nickied




msg:707159
 1:22 pm on Mar 28, 2006 (gmt 0)

newbies:

is there such thing as html markup to tell googlebot what to crawl

no. though I always get a chuckle when i see:

'robots revisit' or 'robots all' in the metas.

and what not to?

robots.txt can help exclude 'good' robots. search this site for robots.txt; you should end up at formum 93.

If you look at Brett's robot.txt about 20 lines down you'll see:

User-agent: *
Disallow: /

Be very careful with this; Brett has chosen to exclude everything. Read up on it before implementing.

Phil_Payne




msg:707160
 3:01 pm on Mar 28, 2006 (gmt 0)

> sadly not.

Though it is a shame. I have one page in particular that looks as if it is keyword-stuffed in the open text - I'd love a <noindex> </noindex> markup tag.

What about a server-side include for the stuff that's not to be indexed?

tedster




msg:707161
 3:08 pm on Mar 28, 2006 (gmt 0)

SSI is inserted into the document before it leaves the server. The bot has no idea how the page was generated, so it all gets indexed, SSI or not.

Kufu




msg:707162
 5:29 pm on Mar 28, 2006 (gmt 0)

You can use JavaScript to write that section of the text. Since the spiders don't have JavaScript activated that part will be 'invisible' to them.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved