stigsb
msg:223096 | 6:45 am on Jul 10, 2002 (gmt 0) |
Hi, this is Stig from FAST. We do have a crawler for images and other formats, it is using the agent string "FAST/WebCrawler-3.x Multimedia". - Stig
|
misosoph
msg:223097 | 7:41 am on Jul 10, 2002 (gmt 0) |
Thank you. I think this is what I am looking for. But will this work? User-agent: FAST/WebCrawler-3.x Multimedia
Disallow: / Or do I have to write this some other way? Thanks again.
|
jimmykav
msg:223098 | 8:16 am on Jul 10, 2002 (gmt 0) |
Hi Stig I had not noticed you around here before. Welcome.
|
stigsb
msg:223099 | 9:46 pm on Jul 10, 2002 (gmt 0) |
Misosoph: yes, that robot.txt file should work. Please note that your robots.txt file changes won't apply until we re-fetch that file. This should not take longer than three to four days. - Stig
|
|
