Welcome to WebmasterWorld Guest from 54.234.153.208

Forum Moderators: goodroi

Message Too Old, No Replies

I am writing my 1st robots.txt :) - never bothered before

Anyone know any crawlers I should add ??

     
3:45 am on Jan 12, 2002 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Nov 27, 2001
posts:1159
votes: 2


I am writing my 1st robots.txt :)

Never really had a need to before.

Anyone know any crawlers I should add ??

Thanks !!

Dazz

Woz

4:06 am on Jan 12, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member woz is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 13, 2000
posts:4823
votes: 0


Dazz,

Brett put one together sometime ago. Use the search feature and you should be able to find it.

Onya
Woz

4:11 am on Jan 12, 2002 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Nov 27, 2001
posts:1159
votes: 2


Will do cheers.


Dazz

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members