Mardi_Gras
msg:1529127 | 4:56 am on Feb 22, 2004 (gmt 0) |
Welcome to WebmasterWorld. I'm not sure about something - are you trying to create a robot or create a robots.txt file?
|
gotmallz
msg:1529128 | 5:25 am on Feb 22, 2004 (gmt 0) |
I am trying to make the most effective way to have the engines run the robots.txt file... Im not sure what u mean by a robot, LOL....
I was also wondering if there are any resources to actually test the effectiveness fo your robots.txt file...
|
gotmallz
msg:1529129 | 5:31 am on Feb 22, 2004 (gmt 0) |
Also I was wondering if I am using PHP and not html would I run it as a noindex or still as an index?
|
gotmallz
msg:1529130 | 5:48 am on Feb 22, 2004 (gmt 0) |
Another ting I am curious about is can a robots.txt file be set to search www. and http:// seperately..........
Because when I run a spider test for instance on
[giftmallz.com...] it spiders nothing but when I run it as www.giftmallz.com it spiders the site..... Is there a way around this so it will spider either or?
|
Uzil
msg:1529131 | 5:38 pm on Feb 22, 2004 (gmt 0) |
Hi gotmallz
you are a very confuse man about what robots.txt is. Like Mardi_Gras said, that's what you actually mean?
I just post same info about robots and the metatag(only 1) that you put on your index.
Read this forum http://www.webmasterworld.com/forum93/279.htm
|
gotmallz
msg:1529132 | 8:32 pm on Feb 22, 2004 (gmt 0) |
No I have read and also put the header tags in, the main problem I am having now is witht he fact that some sites will ony spider with [whatever.com...] and some with only www.whatever.com....... This is creating total confusion. As the spider sessions should grab either or correct, but this is not happening.
|
|
