homepage Welcome to WebmasterWorld Guest from 174.129.130.202
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / WebmasterWorld / Webmaster General
Forum Library, Charter, Moderators: phranque & physics

Webmaster General Forum

    
how to AVOID pages being indexed
[spun off from another thread]
tedster




msg:373678
 5:43 am on Oct 29, 2000 (gmt 0)

this thread was copied from another in the Google search engine forum, but it's a topic that might stand on its own as a general webmaster issue. I hope I've been able to salvage the pertinent posts here. -RCJ

rencke and rc,

Do you have any thoughts about keeping spiders away from the content pages and only allowing them to feed on the NOFRAMES content of the frameset?

 

Marcia




msg:373679
 7:35 am on Oct 29, 2000 (gmt 0)

What tedster asked - I need that, too (I have pages that need to be hidden).

eljefe3




msg:373680
 8:15 am on Oct 29, 2000 (gmt 0)

Is there any way to "hide" the contents inside the noframes tag? My competitors would sure "view source" and see the noframes keyword coding. I'm sure I have seen this done somewhere, I just can't remember how it was done.

Thanks,

eljefe3

rcjordan




msg:373681
 4:01 pm on Oct 29, 2000 (gmt 0)

>Do you have any thoughts about keeping spiders away from the content pages and only allowing them to feed on the NOFRAMES content of the frameset?

Well, the site I'm referencing had no concerns w/ hiding the content, and my philosophy re spidering is generally more is better. Since the content was dynamically generated (using a query "?"), I assumed that it was not going to be spidered anyway.

>hiding noframes
No way that I know of. There was a thread yesterday that I believe discussed some tricks to make it less visible, but it was still there in the source.

rencke




msg:373682
 5:18 pm on Oct 29, 2000 (gmt 0)

>>Do you have any thoughts about keeping spiders away from the content pages and only allowing them to feed on the NOFRAMES content of the frameset?

If you have nothing to hide, why put all your eggs into the same basket? The more pages (with different titles and descriptions), the better chance that one of them may rank high in a search reply and bring people into the site.

If, on the other hand, you do have something to hide - well, sorry I have no other idea than the meta robots tag - but I am told that not all engines respect that one.

Marcia




msg:373683
 7:35 pm on Oct 29, 2000 (gmt 0)

This is good to know. The pages in question for me involve some "helps" pages at a domain site (with more to go up after editing) which were also put up at a helps site done in conjunction with volunteer work - never intended to be spidered or listed, never submitted. I didn't want them spidered or indexed - they were for "community" use only, to help members and avoid a lot of duplicate posting of information and answers.

BUT - other members of the community linked to the pages on message boards, which I wasn't expecting and didn't even think of, and indeed, the boards were spidered, and therefore those pages. I just checked Google, and every page on my "community site" is indexed. The issue is having duplicate content up.

I'll just get them 404'd. How interesting - how to avoid listings instead of how to get them.

rcjordan




msg:373684
 9:13 pm on Oct 29, 2000 (gmt 0)

So, in this new thread on the subject, the question is:
how to avoid listings instead of how to get them
...and not just in noframes.
rcjordan




msg:373685
 9:30 pm on Oct 29, 2000 (gmt 0)

I've had fair success in stopping email harvesters by using js to write the mailto tags. It seems using document.write to code links to pages could provide a similar "stop" for spiders.

Purposely loading the page with words spiders view negatively, such as "links, resource, directory" might also add some defense without getting you marked as a spammer.

As for boards linking to the help pages. How about a js that performs a redirect (to a 404, perhaps) unless the referrer matches an approved page?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Webmaster General
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved