Forum Moderators: mack
[edited by: jatar_k at 11:15 pm (utc) on Oct. 14, 2008]
[edit reason] removed specifics [/edit]
How long has this "Site may not exist" problem been going on? If your site has been on-line for more than two days and this problem persists, then you may have a serious problem that bears further investigation. If the site recently just went on-line, it may simply be a matter of incomplete DNS propagation which should clear up within 48 hours at most.
Request the URL http://www.example.com/robots.txt -- substituting your domain name for "www.example.com". If you get a 404 Not Found response, then your site does not have a robots.txt file, and this is the source of the errors you see.
robots.txt is meant to keep search engine robots out of your admin and stats sections, and also keep them out of any other areas you don't want to appear in search results. For example, you might restrict them from publishing links to your Terms of Service or Privacy Policy pages, since these are not likely where you would want a searcher to land on your site after doing a search.
You do not have to have a robots.txt file, but search engines will continue to ask for it forever, and they will 'litter' your error log file with the resulting 404 errors. This simply makes the error log file a lot bigger and a lot less useful/usable for finding legitimate erros, and it wastes the server's time logging these errors. So, even if you don't necessarily want or need to restrict robots' access to your pages, you might consider putting up at least a blank robots.txt file to stop all these 404 errors.
Simply create a blank file using a plain-text editor such as Notepad on Windows, and upload that to the root directory of your server at www.example.com/robots.txt, naming it exactly "robots.txt" -- "robots" being plural, and all-lowercase.
[added] We have a robots.txt-specific forum here [webmasterworld.com]. [/added]
Jim
[edited by: jdMorgan at 12:58 am (utc) on Oct. 14, 2008]
I'm not sure in what context you see this "Site may not exist" message. If you are talking about searching for your domain name or company name when you see this message, then it may simply be that your site has too few incoming links to make the search engines aware of it. So the third order of business may be to seek incoming links from other good sites that are closely-related to your business, as well as to continue reading here until you get comfortable with basic SEO -- Don't make the mistake of rushing off to change everything before you start to feel confident that you understand the 'big picture.'
Generally, Web sites which have few incoming links and little or no optimization will always rank poorly. If you do a search for "<your product> <your city>" and your site does not come up on the first page of results, then you have a lot of work to do. And if you have no brick-and-mortar store to anchor you to a location, then you've got even more work to do, because the search phrase "<your product>" is thousands of times more competitive than the phrase "<your product> <your city>" -- In essence, people adding a location to their product search reduces the competition dramatically.
So read, understand, plan, ask here, then implement. Many if not most of the threads here have to do with "damage control" of poorly-researched, poorly-planned, or poorly-implemented strategies. The common theme is that most people (including full-time Webmasters) think that things are simpler than they really are. So caution and a very thorough approach are indicated. I think an appropriate statement would be, "If it seems simple and straightforward, then you're probably forgetting something."
However, your current strategy of digging through the forum libraries here is a winner. Look not only for interesting library subjects, but look also for the subjects that seem to recur often in the forums. Those constitute the "common and frequent problems" that you should be aware of before embarking on any improvement projects.
Best,
Jim
Julie
So keep in mind that the content serves not only to keep people on your site (reading), but also as link-bait. You want that content to be useful whether these visitors actually buy or not, so don't feel yourself limited to churning out sales pitches -- Tell us about your widgets, where they come from, why some are blue while others are green, some fuzzy and some not, some affordable and some exorbitantly expensive etc.
How are these widgets produced? -- Tell about the people who 'make' or harvest them. If there are some really notable examples of these widgets, make a page about them too... Do certain celebrities prefer certain types of your widgets? There's another page.
Try to write about subjects that are genuinely-interesting and informative. Then make the sales pitch in a very low-key way -- This works for sales, site stickyness, and link-bait.
Jim
Julie
