Welcome to WebmasterWorld Guest from 50.16.24.12

Forum Moderators: goodroi

I am writing my 1st robots.txt :) - never bothered before

Anyone know any crawlers I should add ??

   
3:45 am on Jan 12, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I am writing my 1st robots.txt :)

Never really had a need to before.

Anyone know any crawlers I should add ??

Thanks !!

Dazz

Woz

4:06 am on Jan 12, 2002 (gmt 0)

WebmasterWorld Senior Member woz is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Dazz,

Brett put one together sometime ago. Use the search feature and you should be able to find it.

Onya
Woz

4:11 am on Jan 12, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Will do cheers.


Dazz

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month