Spiders

do i need one

[Andrew Thomas]

Im totally confused on Spiders? Ive looked in the Forums, All im concerned about is getting as many SE to look at my site...

So do I need a robots.txt file, if so, is there a basic working template i can use...

thanks from someone who is totally confused

[agerhart]

[robotstxt.org...]

There is an example that you can use about half way down the page.....you need this for your site.

[Andrew Thomas]

thanks, but why would i want to Disallow, spiders from looking at my files, when I thought the idea was to promote sites as much as possable?

[agerhart]

you want to disallow certain spiders from certain section.....for instance, you don't want people to be able to access your stats, right? Use the "disallow" to keep spiders out of certain folders that you want to keep private.

[Andrew Thomas]

Oh thanks, sorry for sounding dumb... But i was looking at the forums and could only see people putting disalow on all folders and couldnt see why!

Apart from stats there isnt much else i want to protect.. All my images are in one folder, but with meaningfull names which hopefully SE may pick up as keywords, so should I leave this open for spiders to search though?

[Ove]

Ok Andrew

You dont wanna share your stats thats right.

But that folder who cares about that part does not be in the www folder in my case i got a folder called bin and it is özldfjkgnaoö

i try again when i get in my ftp i have that bin folder and from there i have to go to the www do you mean i have to set a robot.txt to protect the bin folder.

Hope you get what i mean
/Ove