homepage Welcome to WebmasterWorld Guest from 54.196.62.132
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / WebmasterWorld / New To Web Development
Forum Library, Charter, Moderators: brotherhood of lan & mack

New To Web Development Forum

    
robots.txt
top page not found errors
pearlshoppe




msg:3765009
 12:29 am on Oct 14, 2008 (gmt 0)

I am not only new to this site (just joined today and I've been reading all day :-) but I have new website. My problem is when I type in my address it shows "site may not exist" I hit name again and my site appears. In the "top page not found" section under administration I have 408 robots.txt errors. Is this something web designers left out? I'm clueless and know I have a mountain of learning ahead of me. Thanks to someone else in pearl business who recommended this site and said to read, read, read I hope to learn from all of you. Brett Tabke's thread on "Successful Site in 12 Months with Google Alone" has already taught me more than the fishing I've done for the last month. Thank you and I appreciate any help.

[edited by: jatar_k at 11:15 pm (utc) on Oct. 14, 2008]
[edit reason] removed specifics [/edit]

 

jdMorgan




msg:3765049
 12:57 am on Oct 14, 2008 (gmt 0)

Hi pearlshoppe, and welcome to WebmasterWorld!

How long has this "Site may not exist" problem been going on? If your site has been on-line for more than two days and this problem persists, then you may have a serious problem that bears further investigation. If the site recently just went on-line, it may simply be a matter of incomplete DNS propagation which should clear up within 48 hours at most.

Request the URL http://www.example.com/robots.txt -- substituting your domain name for "www.example.com". If you get a 404 Not Found response, then your site does not have a robots.txt file, and this is the source of the errors you see.

robots.txt is meant to keep search engine robots out of your admin and stats sections, and also keep them out of any other areas you don't want to appear in search results. For example, you might restrict them from publishing links to your Terms of Service or Privacy Policy pages, since these are not likely where you would want a searcher to land on your site after doing a search.

You do not have to have a robots.txt file, but search engines will continue to ask for it forever, and they will 'litter' your error log file with the resulting 404 errors. This simply makes the error log file a lot bigger and a lot less useful/usable for finding legitimate erros, and it wastes the server's time logging these errors. So, even if you don't necessarily want or need to restrict robots' access to your pages, you might consider putting up at least a blank robots.txt file to stop all these 404 errors.

Simply create a blank file using a plain-text editor such as Notepad on Windows, and upload that to the root directory of your server at www.example.com/robots.txt, naming it exactly "robots.txt" -- "robots" being plural, and all-lowercase.

[added] We have a robots.txt-specific forum here [webmasterworld.com]. [/added]

Jim

[edited by: jdMorgan at 12:58 am (utc) on Oct. 14, 2008]

pearlshoppe




msg:3765060
 1:20 am on Oct 14, 2008 (gmt 0)

Jim,
Thank you (I think). My site has been active for 3 months. I didn't know if there was problem with site or not. I hired a web company last October. They said it would take two months to do, and eight months later I was still waiting. When I finally got management to email me the guy doing my site had been "busy", but website was up within days only to find out he had copied a competitors privacy/password page which I have redone. I haven't paid final bill and he won't answer email on why can't get to website easily. If it is caused because I don't have enough "content" I can write and solve that problem, but I'm an idiot with this stuff (which is why I hired someone to do it) Do all websites have trouble showing up on searches in the beginning?

jdMorgan




msg:3765068
 1:48 am on Oct 14, 2008 (gmt 0)

If you are "tied" to this company in any way -- for example, if *they* registered your domain name, or if they host your Web site, then the first order of business is to get control/ownership of these.

I'm not sure in what context you see this "Site may not exist" message. If you are talking about searching for your domain name or company name when you see this message, then it may simply be that your site has too few incoming links to make the search engines aware of it. So the third order of business may be to seek incoming links from other good sites that are closely-related to your business, as well as to continue reading here until you get comfortable with basic SEO -- Don't make the mistake of rushing off to change everything before you start to feel confident that you understand the 'big picture.'

Generally, Web sites which have few incoming links and little or no optimization will always rank poorly. If you do a search for "<your product> <your city>" and your site does not come up on the first page of results, then you have a lot of work to do. And if you have no brick-and-mortar store to anchor you to a location, then you've got even more work to do, because the search phrase "<your product>" is thousands of times more competitive than the phrase "<your product> <your city>" -- In essence, people adding a location to their product search reduces the competition dramatically.

So read, understand, plan, ask here, then implement. Many if not most of the threads here have to do with "damage control" of poorly-researched, poorly-planned, or poorly-implemented strategies. The common theme is that most people (including full-time Webmasters) think that things are simpler than they really are. So caution and a very thorough approach are indicated. I think an appropriate statement would be, "If it seems simple and straightforward, then you're probably forgetting something."

However, your current strategy of digging through the forum libraries here is a winner. Look not only for interesting library subjects, but look also for the subjects that seem to recur often in the forums. Those constitute the "common and frequent problems" that you should be aware of before embarking on any improvement projects.

Best,
Jim

pearlshoppe




msg:3765072
 2:05 am on Oct 14, 2008 (gmt 0)

Wow, you're good! I registered domain name and did webhosting, so I'm ok there. And I'm sure I don't have any incoming links, still learning what it is and how to do it. I'm sure I have ALOT of work to do, but I will heed your advice and go slowly and learn and ask questions here. Easier to do it right first time than fix it later. Thank you again, I'm sure this won't be the last time I need your advice or input :-)

Julie

jdMorgan




msg:3765082
 2:30 am on Oct 14, 2008 (gmt 0)

I forgot to mention above that your statement about writing more content is also right on the money. You need incoming links to get traffic and to get the search engines' attention. But you also need content to keep people on your site (to make it 'sticky'), and if that content is unique and genuinely useful, then it also serves as "link bait" -- People will link to your content from forums and blogs (and even a few from their own Web sites). So content is very important, and the only reason I didn't mention it above is that you indicated that you were already aware of it.

So keep in mind that the content serves not only to keep people on your site (reading), but also as link-bait. You want that content to be useful whether these visitors actually buy or not, so don't feel yourself limited to churning out sales pitches -- Tell us about your widgets, where they come from, why some are blue while others are green, some fuzzy and some not, some affordable and some exorbitantly expensive etc.

How are these widgets produced? -- Tell about the people who 'make' or harvest them. If there are some really notable examples of these widgets, make a page about them too... Do certain celebrities prefer certain types of your widgets? There's another page.

Try to write about subjects that are genuinely-interesting and informative. Then make the sales pitch in a very low-key way -- This works for sales, site stickyness, and link-bait.

Jim

pearlshoppe




msg:3765440
 3:43 pm on Oct 14, 2008 (gmt 0)

Jim, quick question, please. When writing content how do you get underline words to light up blue so you can go somewhere else? Only ones that light up for me are website addresses. If it's too difficult to answer easily, just say so and I will hunt in forum for it. Thanks

Julie

pearlshoppe




msg:3765715
 10:05 pm on Oct 14, 2008 (gmt 0)

Jim,Sorry for last post. Two senior members answered question for me Thanks
Julie

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / New To Web Development
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved