homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Please help me with robots.txt

 11:30 am on Jan 31, 2005 (gmt 0)

I have a site:

My robots.txt contains:
User agent:*

I put this txt file in my main directory

But when i type the name of my site i acn see in browsersomething like this:

where i must put the robots.txt in main directory or in /en directory?
10x in advance



 12:01 pm on Jan 31, 2005 (gmt 0)

Hi Karantiq,
Welcome to Webmasterworld.

The proper format is as follows

The following allows all robots to visit all files because the wildcard "*" specifies all robots.

User-agent: *

This one keeps all robots out.

User-agent: *
Disallow: /

You want to place the robots.txt where yout html files are.


 12:57 pm on Jan 31, 2005 (gmt 0)

10x for this quick reply


 4:32 pm on Jan 31, 2005 (gmt 0)

One more question. Can I have multiple robots.txt files in my site - I mean in multiple directories (I have html files in different folders)? Would this affect the indexing?


 4:45 pm on Jan 31, 2005 (gmt 0)

No you dont need multiple robots.txt files, spiders will only request the robots.txt file in the root directory of your site.


 5:15 pm on Jan 31, 2005 (gmt 0)

The website is already designed and it's not possible to move the html files to the folder where robots.txt is. Is there a way to make the search engines index my site anyway?


 5:40 pm on Jan 31, 2005 (gmt 0)

jbinbpt coment was not quite correct, he should of said you should place the robots.txt file in the root directory of your website which is the same place as your index.html file, you can have html files in other directories.

A robots.txt file will not help to get your site indexed in a search engine, this file is only used to disallow robots not to allow, and thats only if the spider or robot what may visit is a friendly robot, if its not it will just ignore the file and spider your site anyway

See this link below on how to build a successful site


 1:18 pm on Feb 8, 2005 (gmt 0)

Hello ncw164x
The problem with my site is that, it is finished now and the major index.html file is not in the root directory. I started my site with good google position 4-th place from 512000 sites. But after that i renew information on my site - meta tags, title etc...I use in one technique in my site- when i upload pictures in it, the script automatically give to the alt tags of the pictures good keywords. can it be the problem with indexing by Googlebot. When my index.html is not in root directory, how i must submit in the search engines-with the full path to the index.html or only to the root directory. Do you know someone who will help me. Even i can pay for supporting of my site and for increasing its rank popularity
10x in advance
ps. can you give me an E-mail of the google support centre to write to them, because it is really important for me and my work

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved