Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- robots , duplicate content and google


fsmobilez - 8:07 pm on Sep 23, 2008 (gmt 0)


well i have search in yahoo

the robots.txt urls cached pages are just turning to

We're sorry, but we could not process your request for the cache of http://example.com/demo.php?cat_id=ahhf&jtype=1. Please click here to check the current page or check for previous versions at the Internet Archive.

So what u say is it properly blocked by yahoo as well

and yes most of the pages are crawled with

jtype=1

example below

www.example.com/demo/jokes_category.php?cat_id=78&jtype=1

as i have blocked them and now i want all search engines to crawl the urls without extra parameter

one of the example is

www.example.com/demo/jokes_category.php?cat_id=78

so can i submit a new sitemap with new requried urls or just wait for few weeks,months

as i dont know whether robots blocked urls will be removed from search engines or not.


Thread source:: http://www.webmasterworld.com/robots_txt/3750192.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com