Welcome to WebmasterWorld Guest from 54.144.68.27

Forum Moderators: goodroi

Message Too Old, No Replies

New to Robots

New to Robots

     

gotmallz

4:52 am on Feb 22, 2004 (gmt 0)

10+ Year Member



Hello,
I am very new to robots and am not 100% sure what I need to do to make a very effective robot. This being not only the robot itself which I have read much on but also the metdescriptions on the index page. Now I am not sure because allot of the information I have read does not even speak of including a metadescription telling the index to follow the robot. Could someone clarify this for me and explain a little more as to what I should place in the index header for the bot. Also I was curious will all bots be allowed if I run a wildcard * instead of telling it to allow only specific bot types....

Thank you,

Mardi_Gras

4:56 am on Feb 22, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Welcome to WebmasterWorld. I'm not sure about something - are you trying to create a robot or create a robots.txt file?

gotmallz

5:25 am on Feb 22, 2004 (gmt 0)

10+ Year Member



I am trying to make the most effective way to have the engines run the robots.txt file... Im not sure what u mean by a robot, LOL....
I was also wondering if there are any resources to actually test the effectiveness fo your robots.txt file...

gotmallz

5:31 am on Feb 22, 2004 (gmt 0)

10+ Year Member



Also I was wondering if I am using PHP and not html would I run it as a noindex or still as an index?

gotmallz

5:48 am on Feb 22, 2004 (gmt 0)

10+ Year Member



Another ting I am curious about is can a robots.txt file be set to search www. and http:// seperately..........
Because when I run a spider test for instance on
[giftmallz.com...] it spiders nothing but when I run it as www.giftmallz.com it spiders the site..... Is there a way around this so it will spider either or?

Uzil

5:38 pm on Feb 22, 2004 (gmt 0)

10+ Year Member


Hi gotmallz
you are a very confuse man about what robots.txt is. Like Mardi_Gras said, that's what you actually mean?
I just post same info about robots and the metatag(only 1) that you put on your index.
Read this forum http://www.webmasterworld.com/forum93/279.htm

gotmallz

8:32 pm on Feb 22, 2004 (gmt 0)

10+ Year Member



No I have read and also put the header tags in, the main problem I am having now is witht he fact that some sites will ony spider with [whatever.com...] and some with only www.whatever.com....... This is creating total confusion. As the spider sessions should grab either or correct, but this is not happening.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month