Forum Moderators: phranque

Message Too Old, No Replies

How can I block those urls from getting spidered

         

zeus

3:50 pm on Jul 20, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hello

How can I block those endings/versions of pages from getting spidered.

Ex.

/gallery/page
/gallery/page?noCount=1

ex.
everything that has this in the url slideshow.php

/gallery/slideshow.php?set_albumName=page

ex.
everything that has this at end ?page=1

/gallery/newpage?page=1

here another situation

/page/TV-widget/endpage
/page/TV-widgets/endspage


here i would like to block TV-widget so only TV-widgets are in the serps.

wilderness

4:03 pm on Jul 20, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



pages from getting spidered.


from reputable SE's or pests?

here i would like to block TV-widget so only TV-widgets are in the serps.


Nearly all the major-SE's offer FAQ's which provide the correct references for inserting meta-tags into individual pages.

robots.txt will also work for compliant bots.

zeus

4:32 pm on Jul 20, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



robots.txt is a option the dumb thing is just that google are extreme slow in removing pages they already have indexed. How would you write a robots.txt to block those mentioned.

wilderness

6:11 pm on Jul 20, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



There's an entire forum at Webmaster World devoted to such topics:

Sitemaps, Meta Data, and robots.txt [webmasterworld.com]