homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Should I be using a Robots.txt file?\n

 11:13 pm on Jan 9, 2001 (gmt 0)

Can the se's pick up on a cloaking script from slurping .cgi's??? What's a good read on robot.txt files?



 1:06 am on Jan 10, 2001 (gmt 0)

Hi BH!

I found these sites,

Their is info at w3.org
scroll down the page to "notes on helping search engines index your web site."

I also found this robots.txt checker:
This page also offers "botwatch". The key to the robots.txt file is to make sure the syntax is written exactly. Otherwise the robot can't read it because it's a text input file.

Does this help at all? I'm just learning and gathering resources at this time, as you probably know. This is so cool. :)


 4:34 am on Jan 10, 2001 (gmt 0)

Good stuff Mouse mover..Keep up the good werk ¦;)/


 12:31 pm on Jan 10, 2001 (gmt 0)

no problem BH. check this flash site out, m-axis.com. make sure you have your sound on!

wish someone would answer your cgi question soon. oh, i think i know. the robot doesn't get the cgi program of course, they get the html page you decided to serve them through the cgi script's logic. you know, the client sends out a message to the server and the server then sends them a message and a html file. so every message sent to the server is filtered through the cgi program first, so i highly doubt their is a way for them to tell, unless you tell them. hehe. is this dull?


 5:44 pm on Jan 10, 2001 (gmt 0)

Wow ...you're moving right along if you juststarted messing with flash. Sound is a pain...I found this [killersound.com ] they gotz a "killer" tute on sound layering.


 7:40 pm on Jan 10, 2001 (gmt 0)

no, that isn't mine! i shutter to think how long it would take me, but i think it's a good example. something to strive towards. one should probably 'begin' with a graphic art's degree as the graphics are outstanding, to say the least.

Thanks for the sound link! i have been thinking about sound but i'm still on 'day 2' since i became side tracked due to the search engine problem. i don't want to write a program that has to consider every single agent on the planet. i think i will just have a default solution after the most obvious has been considered, or something??

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved