Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

I think there is a problem

Robots problem

9:21 am on May 16, 2008 (gmt 0)

New User

5+ Year Member

joined:Jan 10, 2008
posts: 13
votes: 0

Hi All,

Do I have a problem with this file. I think the second line might be causing more restrictions than it should. Should it be /*basket.ashx*
And will the /*?* and /*? cause bots to stop indexing page with ? in the url.

Please advice.

User-agent: *
Disallow: /basket.ashx*
Disallow: /*?*
Disallow: /*?

# Google Image
User-agent: Googlebot-Image
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Allow: /*

# Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /

# digg mirror
User-agent: duggmirror
Disallow: /

10:39 am on May 17, 2008 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
votes: 67

Welcome to WebmasterWorld!

I assume you are trying to use wildcards aka pattern matching.

The line "Disallow: /*?" will block googlebot from access ing all URLs that include a question mark in them.

To block all urls that end with basket.ashx use this line: "Disallow: /*basket.ashx$"

ps dont include the quotation marks :)

Please remember that wildcards also known as pattern matching are not officially part of robots.txt. It is something extra that Google, Yahoo and MSN allows. Each engine handles it slightly differently. Most of the smaller bots will not be able to handle the wildcards. Good luck and make sure to monitor how spiders index the site after you make changes to your robots.txt.