Welcome to WebmasterWorld Guest from 50.16.24.12

Forum Moderators: goodroi

creating robots.txt

   
2:56 am on Jan 23, 2004 (gmt 0)

10+ Year Member



The WYSIWYG program I'm using called NAM* is a simple, and easy to use, html creator. BUT it doesn't make or address robots.txt at all. Is there a program I could feed my pages into that would create robots.txt or fix my pages?

Thanks (in over my head)

3:04 am on Jan 23, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



You only need(not really) one robots.txt file. There should be no ill effects from NOT having robots.txt.

[searchengineworld.com...]
[robotstxt.org...]

4:06 am on Jan 23, 2004 (gmt 0)

10+ Year Member



Birdman,

I looked through all that stuff and I'm just not getting it. I ran my site through the Validator and got an error on every line. So I'm thinking my site is not indexable because of it. I have more content and keywords than competitors that are on the first page of Google but my site ranking is going nowhere.

Thanks for the response.

4:29 am on Jan 23, 2004 (gmt 0)

10+ Year Member



I'm back again, I ran everyone on the first page of google through the Validator and they all have the same errors I have regardless of robots.txt evident in the source. So what's the point of the Validator?
12:21 pm on Jan 23, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Are you referring to the HTML validator? I think your are looking in the wrong place, concerning your lack of good SERP [webmasterworld.com]s.

If I were you, I would forget about robots.txt and look at a couple other things:

[ul]

  • How many backlinks [webmasterworld.com] do you have?
  • Is the site spiderable [searchengineworld.com]

    Making your HTML validate could also be important because a spider [webmasterworld.com] could easily get confused if your HTML code is badly broken.

    If your site is relatively new, you should just concentrate on getting backlinks, adding new content, and fixing any validation errors. In time, you should see your site climbing in the SERPs.

    Good luck with it,
    Birdman

  •  

    Featured Threads

    My Threads

    Hot Threads This Week

    Hot Threads This Month