Welcome to WebmasterWorld Guest from 22.214.171.124
joined:Apr 3, 2002
This is a quick question because I am only asking about a robot that can be disallowed in robots.txt
joined:July 10, 2002
We do have a crawler for images and other formats, it is using the agent string "FAST/WebCrawler-3.x Multimedia".
User-agent: FAST/WebCrawler-3.x Multimedia Disallow: /
Or do I have to write this some other way?
joined:Mar 27, 2001
I had not noticed you around here before. Welcome.
Become a Pro Member