Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Sitemaps and Robots.txt

Omitting certain bulk pages

         

lizardlips

4:43 pm on Aug 7, 2006 (gmt 0)

10+ Year Member



I have recently created a sitemap that includes groups of pages I really don't want indexed. I need advice on how to omit this group of pages. Each member on our site has a listing, in each listing there are 4 links I don't want crawled or mapped. Here is an example of the one I do want in a listing and the ones I don't. I am hoping to this in bulk!

Want crawled

1. http://www.example.com/idirectory/inspectors.php?st=ar
2. http://www.example.com/idirectory/view.php?id=1401&page=0&cat=1

Don't want crawled.

http://www.example.com/idirectory/search-3.php?category=&firmname=&business=&address=&zip=
&phone=&fax=&mobile=&im=&manager=&mail=&www=
&reserved_1=&reserved_2=&reserved_3=&filterbyday=
&loc_one=1&loc_two=6&loc_three=&loc_four=&location_text=&page=2

http://www.example.com/idirectory/out.php?ID=799

http://www.example.com/idirectory/mail.php?id=799&cat=&page=

So I do need some pages crawled in the /idirectory and some I don't. How can I configure my robots.txt and sitemap generator to by pass them with out entering each URL?

There are 100s

Thanks for any help

<edited to use "example.com">

[edited by: tedster at 4:52 pm (utc) on Aug. 7, 2006]