How do i add robot.txt?

[genny]

Hi,

Please help me as how to add robot.txt to my site example.com. Where do i add the file? please help.

[edited by: ThomasB at 9:36 am (utc) on June 6, 2005]
[edit reason] examplified [/edit]

[ThomasB]

Genny, just upload a file called "robots.txt" to the root directory of your domain. example.com/robots.txt

You can find a good tutorial here: [searchengineworld.com...]

[pigeontechs]

robot.txt is used only for not wanting to list your pages in google.. you need to use robot.txt if you need to hide some pages.. im i wrong?

[Dijkgraaf]

robots.txt is for any search engine bot not just google.
It doesn't actually hide the page, all it does is tell the bots which pages they aren't allowed to request.
Sometimes the URL's will still get listed in a search engine without any title or text.
If you really want to stop bots finding the URL, you will have to password protect access or similar.

[ElvisFan]

[just upload a file called "robots.txt" to the root directory of your domain]

Okay... please help... I know it is simple... but please bare with me

Created a robots.txt file in notepad... sitting on my harddrive... (do I need to include my url at the top?)

# For domain: http://www.example.com

User-agent: *
Disallow:

Now here is the problem I can't seem to get my head around.. How do I drop it into Frontpage... at the root directory of my domain? Please walk me through this step... thanks

[edited by: ThomasB at 11:32 pm (utc) on July 3, 2005]

[larryhatch]

Elvis: Can you find your index.html or other main page in your host's directory?
You need to upload robots.txt to that very same directory,
right along with your index and other main pages.
There is no need to name your site, that's known to the crawlers already. -Larry

[ElvisFan]

Thanks Larry... so what you are saying is I need to go to my host's (HN) site into "plesk" and do something there? And do I really need to create a robots.txt to get SE to spider my site...

Oh... while here... please maybe because I'm from DownUnder and a blonde... but where and how do I know when my site has been spidered... where do I see that data....

Just one more thing... I did a popularity thingy and could this be right... it seems so odd

yahoo 582
google 49
altavista 599

why is google so low same site same amount of links... am I missing something... thanks....

[Dijkgraaf]

No, you don't need to have a robots.txt file to have a search engine spider your site.
You only need to upload one if you want to disallow certain bots or pages, or if you just want to stop 404 errors for when it is requested.

If you have access to your web logs or stats generated from them you can see what visitors you have, including bots. Otherwise, do a search at a search engine and see if you come up.

Google only serves up a sampling of links, it won't list all the links it knows. People have speculated that this is to stop people trying reverse engineer Page Rank and trying to beat the system.

[larryhatch]

Elvis: Ask yourself:
" How did I upload my index.html page? "

You upload your robots.txt file exactly the same way, and into exactly
the same directory. I FTP all my files up, and never heard of 'PLESK' -Larry

[Reid]

I added robots.txt to a FP site easy. It shows up as a notepad document and when you edit it (right in FP) it edits via notepad.

The location of robots.txt is always http://www.example.com/robots.txt
this location is called the 'root directory'
the homepage is also in the 'root directory'

In FP just open FP and open mycomputer folder containing robots.txt in another window. Drag the robots.txt file out of the folder and drop it right in FP (right beside index.htm or default.htm). now you can double-click it anytime and edit robots.txt