| Welcome to WebmasterWorld Guest from 22.214.171.124 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|Matching patterns in robots.txt|
I have two dynamic URL pages;
I want to allow robots to crawl the first page but i don't want robots to crawl the page with "&start"...How can i do this.
If I use
"Disallow: /index?id" will block both the URL patterns. So How can i be specific..
In my robots.txt:
I have added,
Is this correct....
Please help me..
Welcome to WebmasterWorld kiransarv!
I would not include index in the Google robots.txt line. I would just have Disallow: /*start*. That will exclude all urls with start in it.
Do you want to disallow all URLs that include start, or just those with index or with id in them?
In any case, the trailing * is not required.
I might use:
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved