homepage Welcome to WebmasterWorld Guest from 54.145.209.42
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Disallow: /*? - is it ok ?
nex99

5+ Year Member



 
Msg#: 3761096 posted 7:51 am on Oct 8, 2008 (gmt 0)

i wants to block all affiliate links, so i create my robots.txt as...

User-agent: *
Disallow: /*?

is it ok for google and other search engines ? if not, please guide me the correct one.

 

jeffposaka

5+ Year Member



 
Msg#: 3761096 posted 3:53 pm on Oct 8, 2008 (gmt 0)

This will block all urls with the ? in the url. Is that what you want to do?

I have found this to be helpful:

[google.com...]

nex99

5+ Year Member



 
Msg#: 3761096 posted 1:14 pm on Oct 9, 2008 (gmt 0)

Yes. and thanks for the link.

jimbeetle

WebmasterWorld Senior Member jimbeetle us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3761096 posted 3:13 pm on Oct 9, 2008 (gmt 0)

Just be sure you know exactly what you want to do here. Robots.txt is used to block spiders from URLs on your site. They will still follow outgoing links unless the pages on your site on which the links appear are blocked.

Receptional Andy



 
Msg#: 3761096 posted 3:16 pm on Oct 9, 2008 (gmt 0)

I've had no issues with that syntax. Note that the safest approach is to only disallow those bots known to understand the wildcard, which I believe is googlebot, msnbot and slurp of the majors.

Unfortunately, the robots standard has developed into not much of a standard at all. I believe the below should work:


User-agent: googlebot
User-agent: slurp
User-agent: msnbot
Disallow: /*?

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3761096 posted 1:22 pm on Oct 21, 2008 (gmt 0)

Don't forget that you need at least one blank line after the last record too.

User-agent: googlebot
User-agent: slurp
User-agent: msnbot
Disallow: /*?


Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved