Welcome to WebmasterWorld Guest from 54.197.187.2

Forum Moderators: goodroi

Message Too Old, No Replies

Please help me with robots.txt

     
11:30 am on Jan 31, 2005 (gmt 0)

New User

10+ Year Member

joined:Jan 31, 2005
posts:6
votes: 0


I have a site:

www.mydomain.com
My robots.txt contains:
User agent:*
Dissalow

I put this txt file in my main directory

But when i type the name of my site i acn see in browsersomething like this:

www.mydomain.com/en/.....
where i must put the robots.txt in main directory or in /en directory?
10x in advance

12:01 pm on Jan 31, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 27, 2002
posts:1187
votes: 0


Hi Karantiq,
Welcome to Webmasterworld.

The proper format is as follows

The following allows all robots to visit all files because the wildcard "*" specifies all robots.

User-agent: *
Disallow:

This one keeps all robots out.

User-agent: *
Disallow: /

You want to place the robots.txt where yout html files are.

12:57 pm on Jan 31, 2005 (gmt 0)

New User

10+ Year Member

joined:Jan 31, 2005
posts:6
votes: 0


10x for this quick reply
4:32 pm on Jan 31, 2005 (gmt 0)

New User

10+ Year Member

joined:Jan 31, 2005
posts:6
votes: 0


One more question. Can I have multiple robots.txt files in my site - I mean in multiple directories (I have html files in different folders)? Would this affect the indexing?
4:45 pm on Jan 31, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 7, 2003
posts:1179
votes: 0


No you dont need multiple robots.txt files, spiders will only request the robots.txt file in the root directory of your site.
5:15 pm on Jan 31, 2005 (gmt 0)

New User

10+ Year Member

joined:Jan 31, 2005
posts:6
votes: 0


The website is already designed and it's not possible to move the html files to the folder where robots.txt is. Is there a way to make the search engines index my site anyway?
5:40 pm on Jan 31, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 7, 2003
posts:1179
votes: 0


jbinbpt coment was not quite correct, he should of said you should place the robots.txt file in the root directory of your website which is the same place as your index.html file, you can have html files in other directories.

A robots.txt file will not help to get your site indexed in a search engine, this file is only used to disallow robots not to allow, and thats only if the spider or robot what may visit is a friendly robot, if its not it will just ignore the file and spider your site anyway

See this link below on how to build a successful site
[searchengineworld.com...]

1:18 pm on Feb 8, 2005 (gmt 0)

New User

10+ Year Member

joined:Jan 31, 2005
posts:6
votes: 0


Hello ncw164x
The problem with my site is that, it is finished now and the major index.html file is not in the root directory. I started my site with good google position 4-th place from 512000 sites. But after that i renew information on my site - meta tags, title etc...I use in one technique in my site- when i upload pictures in it, the script automatically give to the alt tags of the pictures good keywords. can it be the problem with indexing by Googlebot. When my index.html is not in root directory, how i must submit in the search engines-with the full path to the index.html or only to the root directory. Do you know someone who will help me. Even i can pay for supporting of my site and for increasing its rank popularity
10x in advance
ps. can you give me an E-mail of the google support centre to write to them, because it is really important for me and my work
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members