Welcome to WebmasterWorld Guest from 54.144.243.34

Forum Moderators: goodroi

Please help me with robots.txt

   
11:30 am on Jan 31, 2005 (gmt 0)

10+ Year Member



I have a site:

www.mydomain.com
My robots.txt contains:
User agent:*
Dissalow

I put this txt file in my main directory

But when i type the name of my site i acn see in browsersomething like this:

www.mydomain.com/en/.....
where i must put the robots.txt in main directory or in /en directory?
10x in advance

12:01 pm on Jan 31, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi Karantiq,
Welcome to Webmasterworld.

The proper format is as follows

The following allows all robots to visit all files because the wildcard "*" specifies all robots.

User-agent: *
Disallow:

This one keeps all robots out.

User-agent: *
Disallow: /

You want to place the robots.txt where yout html files are.

12:57 pm on Jan 31, 2005 (gmt 0)

10+ Year Member



10x for this quick reply
4:32 pm on Jan 31, 2005 (gmt 0)

10+ Year Member



One more question. Can I have multiple robots.txt files in my site - I mean in multiple directories (I have html files in different folders)? Would this affect the indexing?
4:45 pm on Jan 31, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



No you dont need multiple robots.txt files, spiders will only request the robots.txt file in the root directory of your site.
5:15 pm on Jan 31, 2005 (gmt 0)

10+ Year Member



The website is already designed and it's not possible to move the html files to the folder where robots.txt is. Is there a way to make the search engines index my site anyway?
5:40 pm on Jan 31, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



jbinbpt coment was not quite correct, he should of said you should place the robots.txt file in the root directory of your website which is the same place as your index.html file, you can have html files in other directories.

A robots.txt file will not help to get your site indexed in a search engine, this file is only used to disallow robots not to allow, and thats only if the spider or robot what may visit is a friendly robot, if its not it will just ignore the file and spider your site anyway

See this link below on how to build a successful site
[searchengineworld.com...]

1:18 pm on Feb 8, 2005 (gmt 0)

10+ Year Member



Hello ncw164x
The problem with my site is that, it is finished now and the major index.html file is not in the root directory. I started my site with good google position 4-th place from 512000 sites. But after that i renew information on my site - meta tags, title etc...I use in one technique in my site- when i upload pictures in it, the script automatically give to the alt tags of the pictures good keywords. can it be the problem with indexing by Googlebot. When my index.html is not in root directory, how i must submit in the search engines-with the full path to the index.html or only to the root directory. Do you know someone who will help me. Even i can pay for supporting of my site and for increasing its rank popularity
10x in advance
ps. can you give me an E-mail of the google support centre to write to them, because it is really important for me and my work
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month