homepage Welcome to WebmasterWorld Guest from 54.166.113.249
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Using wild card in robots
Can you add * to ip?
walrus

10+ Year Member



 
Msg#: 363 posted 5:28 pm on Apr 13, 2004 (gmt 0)

Disallow:
User-agent:63.148.99.*
Can you do this to block all combinations of the last part of an ip?

 

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 363 posted 5:37 pm on Apr 13, 2004 (gmt 0)

walrus,

I'm afraid you have confused robots.txt with .htaccess and other methods of access control.
The objects to be disallowed in robots.txt are robot names and resources -- such as pages -- on your site.

User-agent: Scooter
Disallow: /igi-bin/
Disallow: /private/

Try a search on WebmasterWorld for "mod_rewrite user_agent" or ".htaccess Deny", or if you're on an IIS server, "ISAPI rewrite".

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved