New to Robots

[gotmallz]

Hello,
I am very new to robots and am not 100% sure what I need to do to make a very effective robot. This being not only the robot itself which I have read much on but also the metdescriptions on the index page. Now I am not sure because allot of the information I have read does not even speak of including a metadescription telling the index to follow the robot. Could someone clarify this for me and explain a little more as to what I should place in the index header for the bot. Also I was curious will all bots be allowed if I run a wildcard * instead of telling it to allow only specific bot types....

Thank you,

[Mardi_Gras]

Welcome to WebmasterWorld. I'm not sure about something - are you trying to create a robot or create a robots.txt file?

[gotmallz]

I am trying to make the most effective way to have the engines run the robots.txt file... Im not sure what u mean by a robot, LOL....
I was also wondering if there are any resources to actually test the effectiveness fo your robots.txt file...

[gotmallz]

Also I was wondering if I am using PHP and not html would I run it as a noindex or still as an index?

[gotmallz]

Another ting I am curious about is can a robots.txt file be set to search www. and http:// seperately..........
Because when I run a spider test for instance on
[giftmallz.com...] it spiders nothing but when I run it as www.giftmallz.com it spiders the site..... Is there a way around this so it will spider either or?

[Uzil]

Hi gotmallz
you are a very confuse man about what robots.txt is. Like Mardi_Gras said, that's what you actually mean?
I just post same info about robots and the metatag(only 1) that you put on your index.
Read this forum http://www.webmasterworld.com/forum93/279.htm

[gotmallz]

No I have read and also put the header tags in, the main problem I am having now is witht he fact that some sites will ony spider with [whatever.com...] and some with only www.whatever.com....... This is creating total confusion. As the spider sessions should grab either or correct, but this is not happening.