Welcome to WebmasterWorld Guest from 54.198.147.221

Forum Moderators: goodroi

Message Too Old, No Replies

New to Robots

New to Robots

     
4:52 am on Feb 22, 2004 (gmt 0)

New User

10+ Year Member

joined:Feb 22, 2004
posts:6
votes: 0


Hello,
I am very new to robots and am not 100% sure what I need to do to make a very effective robot. This being not only the robot itself which I have read much on but also the metdescriptions on the index page. Now I am not sure because allot of the information I have read does not even speak of including a metadescription telling the index to follow the robot. Could someone clarify this for me and explain a little more as to what I should place in the index header for the bot. Also I was curious will all bots be allowed if I run a wildcard * instead of telling it to allow only specific bot types....

Thank you,

4:56 am on Feb 22, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 27, 2002
posts:1422
votes: 0


Welcome to WebmasterWorld. I'm not sure about something - are you trying to create a robot or create a robots.txt file?
5:25 am on Feb 22, 2004 (gmt 0)

New User

10+ Year Member

joined:Feb 22, 2004
posts:6
votes: 0


I am trying to make the most effective way to have the engines run the robots.txt file... Im not sure what u mean by a robot, LOL....
I was also wondering if there are any resources to actually test the effectiveness fo your robots.txt file...
5:31 am on Feb 22, 2004 (gmt 0)

New User

10+ Year Member

joined:Feb 22, 2004
posts:6
votes: 0


Also I was wondering if I am using PHP and not html would I run it as a noindex or still as an index?
5:48 am on Feb 22, 2004 (gmt 0)

New User

10+ Year Member

joined:Feb 22, 2004
posts:6
votes: 0


Another ting I am curious about is can a robots.txt file be set to search www. and http:// seperately..........
Because when I run a spider test for instance on
[giftmallz.com...] it spiders nothing but when I run it as www.giftmallz.com it spiders the site..... Is there a way around this so it will spider either or?
5:38 pm on Feb 22, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 20, 2004
posts:48
votes: 0

Hi gotmallz
you are a very confuse man about what robots.txt is. Like Mardi_Gras said, that's what you actually mean?
I just post same info about robots and the metatag(only 1) that you put on your index.
Read this forum http://www.webmasterworld.com/forum93/279.htm
8:32 pm on Feb 22, 2004 (gmt 0)

New User

10+ Year Member

joined:Feb 22, 2004
posts:6
votes: 0


No I have read and also put the header tags in, the main problem I am having now is witht he fact that some sites will ony spider with [whatever.com...] and some with only www.whatever.com....... This is creating total confusion. As the spider sessions should grab either or correct, but this is not happening.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members