Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Should I be using a Robots.txt file?\n



11:13 pm on Jan 9, 2001 (gmt 0)

10+ Year Member

Can the se's pick up on a cloaking script from slurping .cgi's??? What's a good read on robot.txt files?


1:06 am on Jan 10, 2001 (gmt 0)

Hi BH!

I found these sites,

Their is info at w3.org
scroll down the page to "notes on helping search engines index your web site."

I also found this robots.txt checker:
This page also offers "botwatch". The key to the robots.txt file is to make sure the syntax is written exactly. Otherwise the robot can't read it because it's a text input file.

Does this help at all? I'm just learning and gathering resources at this time, as you probably know. This is so cool. :)


4:34 am on Jan 10, 2001 (gmt 0)

10+ Year Member

Good stuff Mouse mover..Keep up the good werk ¦;)/


12:31 pm on Jan 10, 2001 (gmt 0)

no problem BH. check this flash site out, m-axis.com. make sure you have your sound on!

wish someone would answer your cgi question soon. oh, i think i know. the robot doesn't get the cgi program of course, they get the html page you decided to serve them through the cgi script's logic. you know, the client sends out a message to the server and the server then sends them a message and a html file. so every message sent to the server is filtered through the cgi program first, so i highly doubt their is a way for them to tell, unless you tell them. hehe. is this dull?


5:44 pm on Jan 10, 2001 (gmt 0)

10+ Year Member

Wow ...you're moving right along if you juststarted messing with flash. Sound is a pain...I found this [killersound.com ] they gotz a "killer" tute on sound layering.


7:40 pm on Jan 10, 2001 (gmt 0)

no, that isn't mine! i shutter to think how long it would take me, but i think it's a good example. something to strive towards. one should probably 'begin' with a graphic art's degree as the graphics are outstanding, to say the least.

Thanks for the sound link! i have been thinking about sound but i'm still on 'day 2' since i became side tracked due to the search engine problem. i don't want to write a program that has to consider every single agent on the planet. i think i will just have a default solution after the most obvious has been considered, or something??


Featured Threads

Hot Threads This Week

Hot Threads This Month