| creating robots.txt
|
artdog
msg:1526931 | 2:56 am on Jan 23, 2004 (gmt 0) |
The WYSIWYG program I'm using called NAM* is a simple, and easy to use, html creator. BUT it doesn't make or address robots.txt at all. Is there a program I could feed my pages into that would create robots.txt or fix my pages? Thanks (in over my head)
|
Birdman
msg:1526932 | 3:04 am on Jan 23, 2004 (gmt 0) |
You only need(not really) one robots.txt file. There should be no ill effects from NOT having robots.txt. [searchengineworld.com...]
[robotstxt.org...]
|
artdog
msg:1526933 | 4:06 am on Jan 23, 2004 (gmt 0) |
Birdman, I looked through all that stuff and I'm just not getting it. I ran my site through the Validator and got an error on every line. So I'm thinking my site is not indexable because of it. I have more content and keywords than competitors that are on the first page of Google but my site ranking is going nowhere. Thanks for the response.
|
artdog
msg:1526934 | 4:29 am on Jan 23, 2004 (gmt 0) |
I'm back again, I ran everyone on the first page of google through the Validator and they all have the same errors I have regardless of robots.txt evident in the source. So what's the point of the Validator?
|
Birdman
msg:1526935 | 12:21 pm on Jan 23, 2004 (gmt 0) |
Are you referring to the HTML validator? I think your are looking in the wrong place, concerning your lack of good SERP [webmasterworld.com]s. If I were you, I would forget about robots.txt and look at a couple other things: [ul] How many backlinks [webmasterworld.com] do you have?
Is the site spiderable [searchengineworld.com] Making your HTML validate could also be important because a spider [webmasterworld.com] could easily get confused if your HTML code is badly broken. If your site is relatively new, you should just concentrate on getting backlinks, adding new content, and fixing any validation errors. In time, you should see your site climbing in the SERPs. Good luck with it,
Birdman
|
|
|
