Forum Moderators: Robert Charlton & goodroi
Want crawled
1. http://www.example.com/idirectory/inspectors.php?st=ar
2. http://www.example.com/idirectory/view.php?id=1401&page=0&cat=1
Don't want crawled.
http://www.example.com/idirectory/search-3.php?category=&firmname=&business=&address=&zip=
&phone=&fax=&mobile=&im=&manager=&mail=&www=
&reserved_1=&reserved_2=&reserved_3=&filterbyday=
&loc_one=1&loc_two=6&loc_three=&loc_four=&location_text=&page=2http://www.example.com/idirectory/out.php?ID=799
http://www.example.com/idirectory/mail.php?id=799&cat=&page=
So I do need some pages crawled in the /idirectory and some I don't. How can I configure my robots.txt and sitemap generator to by pass them with out entering each URL?
There are 100s
Thanks for any help
<edited to use "example.com">
[edited by: tedster at 4:52 pm (utc) on Aug. 7, 2006]