Welcome to WebmasterWorld Guest from 54.145.39.186

Forum Moderators: open

Message Too Old, No Replies

Spiders

do i need one

     

Andrew Thomas

3:15 pm on Mar 12, 2002 (gmt 0)

10+ Year Member



Im totally confused on Spiders? Ive looked in the Forums, All im concerned about is getting as many SE to look at my site...

So do I need a robots.txt file, if so, is there a basic working template i can use...

thanks from someone who is totally confused

agerhart

3:19 pm on Mar 12, 2002 (gmt 0)

WebmasterWorld Senior Member agerhart is a WebmasterWorld Top Contributor of All Time 10+ Year Member



[robotstxt.org...]

There is an example that you can use about half way down the page.....you need this for your site.

Andrew Thomas

3:27 pm on Mar 12, 2002 (gmt 0)

10+ Year Member



thanks, but why would i want to Disallow, spiders from looking at my files, when I thought the idea was to promote sites as much as possable?

agerhart

3:31 pm on Mar 12, 2002 (gmt 0)

WebmasterWorld Senior Member agerhart is a WebmasterWorld Top Contributor of All Time 10+ Year Member



you want to disallow certain spiders from certain section.....for instance, you don't want people to be able to access your stats, right? Use the "disallow" to keep spiders out of certain folders that you want to keep private.

Andrew Thomas

3:35 pm on Mar 12, 2002 (gmt 0)

10+ Year Member



Oh thanks, sorry for sounding dumb... But i was looking at the forums and could only see people putting disalow on all folders and couldnt see why!

Apart from stats there isnt much else i want to protect.. All my images are in one folder, but with meaningfull names which hopefully SE may pick up as keywords, so should I leave this open for spiders to search though?

Ove

3:42 pm on Mar 12, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Ok Andrew

You dont wanna share your stats thats right.

But that folder who cares about that part does not be in the www folder in my case i got a folder called bin and it is ÷zldfjkgnao÷

i try again when i get in my ftp i have that bin folder and from there i have to go to the www do you mean i have to set a robot.txt to protect the bin folder.

Hope you get what i mean
/Ove

 

Featured Threads

Hot Threads This Week

Hot Threads This Month