homepage Welcome to WebmasterWorld Guest from 54.197.183.230
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
creating robots.txt
artdog

10+ Year Member



 
Msg#: 247 posted 2:56 am on Jan 23, 2004 (gmt 0)

The WYSIWYG program I'm using called NAM* is a simple, and easy to use, html creator. BUT it doesn't make or address robots.txt at all. Is there a program I could feed my pages into that would create robots.txt or fix my pages?

Thanks (in over my head)

 

Birdman

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 247 posted 3:04 am on Jan 23, 2004 (gmt 0)

You only need(not really) one robots.txt file. There should be no ill effects from NOT having robots.txt.

[searchengineworld.com...]
[robotstxt.org...]

artdog

10+ Year Member



 
Msg#: 247 posted 4:06 am on Jan 23, 2004 (gmt 0)

Birdman,

I looked through all that stuff and I'm just not getting it. I ran my site through the Validator and got an error on every line. So I'm thinking my site is not indexable because of it. I have more content and keywords than competitors that are on the first page of Google but my site ranking is going nowhere.

Thanks for the response.

artdog

10+ Year Member



 
Msg#: 247 posted 4:29 am on Jan 23, 2004 (gmt 0)

I'm back again, I ran everyone on the first page of google through the Validator and they all have the same errors I have regardless of robots.txt evident in the source. So what's the point of the Validator?

Birdman

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 247 posted 12:21 pm on Jan 23, 2004 (gmt 0)

Are you referring to the HTML validator? I think your are looking in the wrong place, concerning your lack of good SERP [webmasterworld.com]s.

If I were you, I would forget about robots.txt and look at a couple other things:

[ul]

  • How many backlinks [webmasterworld.com] do you have?
  • Is the site spiderable [searchengineworld.com]

    Making your HTML validate could also be important because a spider [webmasterworld.com] could easily get confused if your HTML code is badly broken.

    If your site is relatively new, you should just concentrate on getting backlinks, adding new content, and fixing any validation errors. In time, you should see your site climbing in the SERPs.

    Good luck with it,
    Birdman

  • Global Options:
     top home search open messages active posts  
     

    Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
    rss feed

    All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
    Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
    WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
    © Webmaster World 1996-2014 all rights reserved