Welcome to WebmasterWorld Guest from 54.226.62.251

Forum Moderators: goodroi

Message Too Old, No Replies

I am writing my 1st robots.txt :) - never bothered before

Anyone know any crawlers I should add ??

     

diddlydazz

3:45 am on Jan 12, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I am writing my 1st robots.txt :)

Never really had a need to before.

Anyone know any crawlers I should add ??

Thanks !!

Dazz

Woz

4:06 am on Jan 12, 2002 (gmt 0)

WebmasterWorld Senior Member woz is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Dazz,

Brett put one together sometime ago. Use the search feature and you should be able to find it.

Onya
Woz

diddlydazz

4:11 am on Jan 12, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Will do cheers.


Dazz

 

Featured Threads

Hot Threads This Week

Hot Threads This Month