homepage Welcome to WebmasterWorld Guest from 54.167.173.250
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
I am new to all of this about Spiders...Robot.txt
Help Message
lightwriter




msg:1526675
 2:39 am on Jul 11, 2001 (gmt 0)

Hi,

I am new to using the robot.txt with my site and though my questions may seem dumb to some of you, I'd greatly appreciate any assistance.

Do I have to list all of the different spiders in the robots.txt file?

Does anyone have a sample of what it should appear as?

What are the most effective approaches towards using robots.txt and spiders?

Thank you all for your assistance in this matter, I do appreciate it.

Sincerely,

Erik

 

grnidone




msg:1526676
 3:17 am on Jul 11, 2001 (gmt 0)

The only "dumb" question around here is the one that is not asked. Do not ever feel bad about asking a question.

Robots.txt tutorial [searchengineworld.com]

There are some excellent links at the bottom of that page as well.

The only spiders you need to list in your robots.txt file are those you *don't* want to come to your site: like email collectors.

-G

Cronin




msg:1526677
 9:06 pm on Jul 22, 2001 (gmt 0)

Hi all.
This is my first posting, could you show me the text i would have to put in a robot.txt to stop e-mail spiders. thanks maddog
Thanks iv found it

highman




msg:1526678
 9:29 pm on Jul 22, 2001 (gmt 0)

Could you post it please

Cronin




msg:1526679
 9:43 pm on Jul 22, 2001 (gmt 0)

I have just had a look for my robot.txt on my site and gould not find it, gould some let me know where i would find this file, Thanks

highman




msg:1526680
 9:44 pm on Jul 22, 2001 (gmt 0)

It should be in the root like:
[yoursite.com...]

Cronin




msg:1526681
 9:55 pm on Jul 22, 2001 (gmt 0)

I have had a look at a Robots.txt Syntax Checking
on [tardis.ed.ac.uk...]

An Error occurred whilst fetching the robots.txt file

Not found

This is what i get every day on my log files.

Can you help

grnidone




msg:1526682
 12:42 am on Jul 23, 2001 (gmt 0)

Cronin,

I checked the robots.txt file on the website in your profile with this robots.txt validator:

[searchengineworld.com...]

and it said it was correct. Is that the site you are talking about?

-G

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved