homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

New to Robots
New to Robots

 4:52 am on Feb 22, 2004 (gmt 0)

I am very new to robots and am not 100% sure what I need to do to make a very effective robot. This being not only the robot itself which I have read much on but also the metdescriptions on the index page. Now I am not sure because allot of the information I have read does not even speak of including a metadescription telling the index to follow the robot. Could someone clarify this for me and explain a little more as to what I should place in the index header for the bot. Also I was curious will all bots be allowed if I run a wildcard * instead of telling it to allow only specific bot types....

Thank you,



 4:56 am on Feb 22, 2004 (gmt 0)

Welcome to WebmasterWorld. I'm not sure about something - are you trying to create a robot or create a robots.txt file?


 5:25 am on Feb 22, 2004 (gmt 0)

I am trying to make the most effective way to have the engines run the robots.txt file... Im not sure what u mean by a robot, LOL....
I was also wondering if there are any resources to actually test the effectiveness fo your robots.txt file...


 5:31 am on Feb 22, 2004 (gmt 0)

Also I was wondering if I am using PHP and not html would I run it as a noindex or still as an index?


 5:48 am on Feb 22, 2004 (gmt 0)

Another ting I am curious about is can a robots.txt file be set to search www. and http:// seperately..........
Because when I run a spider test for instance on
[giftmallz.com...] it spiders nothing but when I run it as www.giftmallz.com it spiders the site..... Is there a way around this so it will spider either or?


 5:38 pm on Feb 22, 2004 (gmt 0)
Hi gotmallz
you are a very confuse man about what robots.txt is. Like Mardi_Gras said, that's what you actually mean?
I just post same info about robots and the metatag(only 1) that you put on your index.
Read this forum http://www.webmasterworld.com/forum93/279.htm

 8:32 pm on Feb 22, 2004 (gmt 0)

No I have read and also put the header tags in, the main problem I am having now is witht he fact that some sites will ony spider with [whatever.com...] and some with only www.whatever.com....... This is creating total confusion. As the spider sessions should grab either or correct, but this is not happening.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved