homepage Welcome to WebmasterWorld Guest from 107.21.187.131
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Should I be using a Robots.txt file?\n
BoneHeadicus




msg:1525572
 11:13 pm on Jan 9, 2001 (gmt 0)

Can the se's pick up on a cloaking script from slurping .cgi's??? What's a good read on robot.txt files?

 

mousemoves




msg:1525573
 1:06 am on Jan 10, 2001 (gmt 0)

Hi BH!

I found these sites,

Their is info at w3.org
[w3.org...]
scroll down the page to "notes on helping search engines index your web site."

I also found this robots.txt checker:
[tardis.ed.ac.uk...]
This page also offers "botwatch". The key to the robots.txt file is to make sure the syntax is written exactly. Otherwise the robot can't read it because it's a text input file.

Does this help at all? I'm just learning and gathering resources at this time, as you probably know. This is so cool. :)

BoneHeadicus




msg:1525574
 4:34 am on Jan 10, 2001 (gmt 0)

Good stuff Mouse mover..Keep up the good werk ¦;)/

mousemoves




msg:1525575
 12:31 pm on Jan 10, 2001 (gmt 0)

no problem BH. check this flash site out, m-axis.com. make sure you have your sound on!

wish someone would answer your cgi question soon. oh, i think i know. the robot doesn't get the cgi program of course, they get the html page you decided to serve them through the cgi script's logic. you know, the client sends out a message to the server and the server then sends them a message and a html file. so every message sent to the server is filtered through the cgi program first, so i highly doubt their is a way for them to tell, unless you tell them. hehe. is this dull?

BoneHeadicus




msg:1525576
 5:44 pm on Jan 10, 2001 (gmt 0)

Wow ...you're moving right along if you juststarted messing with flash. Sound is a pain...I found this [killersound.com ] they gotz a "killer" tute on sound layering.

mousemoves




msg:1525577
 7:40 pm on Jan 10, 2001 (gmt 0)

no, that isn't mine! i shutter to think how long it would take me, but i think it's a good example. something to strive towards. one should probably 'begin' with a graphic art's degree as the graphics are outstanding, to say the least.

Thanks for the sound link! i have been thinking about sound but i'm still on 'day 2' since i became side tracked due to the search engine problem. i don't want to write a program that has to consider every single agent on the planet. i think i will just have a default solution after the most obvious has been considered, or something??

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved