homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

creating robots.txt

 2:56 am on Jan 23, 2004 (gmt 0)

The WYSIWYG program I'm using called NAM* is a simple, and easy to use, html creator. BUT it doesn't make or address robots.txt at all. Is there a program I could feed my pages into that would create robots.txt or fix my pages?

Thanks (in over my head)



 3:04 am on Jan 23, 2004 (gmt 0)

You only need(not really) one robots.txt file. There should be no ill effects from NOT having robots.txt.



 4:06 am on Jan 23, 2004 (gmt 0)


I looked through all that stuff and I'm just not getting it. I ran my site through the Validator and got an error on every line. So I'm thinking my site is not indexable because of it. I have more content and keywords than competitors that are on the first page of Google but my site ranking is going nowhere.

Thanks for the response.


 4:29 am on Jan 23, 2004 (gmt 0)

I'm back again, I ran everyone on the first page of google through the Validator and they all have the same errors I have regardless of robots.txt evident in the source. So what's the point of the Validator?


 12:21 pm on Jan 23, 2004 (gmt 0)

Are you referring to the HTML validator? I think your are looking in the wrong place, concerning your lack of good SERP [webmasterworld.com]s.

If I were you, I would forget about robots.txt and look at a couple other things:


  • How many backlinks [webmasterworld.com] do you have?
  • Is the site spiderable [searchengineworld.com]

    Making your HTML validate could also be important because a spider [webmasterworld.com] could easily get confused if your HTML code is badly broken.

    If your site is relatively new, you should just concentrate on getting backlinks, adding new content, and fixing any validation errors. In time, you should see your site climbing in the SERPs.

    Good luck with it,

  • Global Options:
     top home search open messages active posts  

    Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
    rss feed

    All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
    Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
    WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
    © Webmaster World 1996-2014 all rights reserved