| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
|Subscribe to WebmasterWorld|
|Matching patterns in robots.txt|
| 9:08 am on Nov 4, 2008 (gmt 0)|
I have two dynamic URL pages;
I want to allow robots to crawl the first page but i don't want robots to crawl the page with "&start"...How can i do this.
If I use
"Disallow: /index?id" will block both the URL patterns. So How can i be specific..
In my robots.txt:
I have added,
Is this correct....
Please help me..
| 12:16 pm on Nov 7, 2008 (gmt 0)|
Welcome to WebmasterWorld kiransarv!
I would not include index in the Google robots.txt line. I would just have Disallow: /*start*. That will exclude all urls with start in it.
| 2:39 pm on Nov 9, 2008 (gmt 0)|
Do you want to disallow all URLs that include start, or just those with index or with id in them?
In any case, the trailing * is not required.
I might use:
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved