homepage Welcome to WebmasterWorld Guest from 54.198.130.203
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
I think there is a problem
Robots problem
AllanS




msg:3651671
 9:21 am on May 16, 2008 (gmt 0)

Hi All,

Do I have a problem with this file. I think the second line might be causing more restrictions than it should. Should it be /*basket.ashx*
And will the /*?* and /*? cause bots to stop indexing page with ? in the url.

Please advice.

User-agent: *
Disallow: /basket.ashx*
Disallow: /*?*
Disallow: /*?

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

# Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /

# digg mirror
User-agent: duggmirror
Disallow: /

 

goodroi




msg:3652473
 10:39 am on May 17, 2008 (gmt 0)

Welcome to WebmasterWorld!

I assume you are trying to use wildcards aka pattern matching.

The line "Disallow: /*?" will block googlebot from access ing all URLs that include a question mark in them.

To block all urls that end with basket.ashx use this line: "Disallow: /*basket.ashx$"

ps dont include the quotation marks :)

Please remember that wildcards also known as pattern matching are not officially part of robots.txt. It is something extra that Google, Yahoo and MSN allows. Each engine handles it slightly differently. Most of the smaller bots will not be able to handle the wildcards. Good luck and make sure to monitor how spiders index the site after you make changes to your robots.txt.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved