agerhart
msg:703105 | 3:19 pm on Mar 12, 2002 (gmt 0) |
[robotstxt.org...] There is an example that you can use about half way down the page.....you need this for your site.
|
Andrew Thomas
msg:703106 | 3:27 pm on Mar 12, 2002 (gmt 0) |
thanks, but why would i want to Disallow, spiders from looking at my files, when I thought the idea was to promote sites as much as possable?
|
agerhart
msg:703107 | 3:31 pm on Mar 12, 2002 (gmt 0) |
you want to disallow certain spiders from certain section.....for instance, you don't want people to be able to access your stats, right? Use the "disallow" to keep spiders out of certain folders that you want to keep private.
|
Andrew Thomas
msg:703108 | 3:35 pm on Mar 12, 2002 (gmt 0) |
Oh thanks, sorry for sounding dumb... But i was looking at the forums and could only see people putting disalow on all folders and couldnt see why! Apart from stats there isnt much else i want to protect.. All my images are in one folder, but with meaningfull names which hopefully SE may pick up as keywords, so should I leave this open for spiders to search though?
|
Ove
msg:703109 | 3:42 pm on Mar 12, 2002 (gmt 0) |
Ok Andrew You dont wanna share your stats thats right. But that folder who cares about that part does not be in the www folder in my case i got a folder called bin and it is özldfjkgnaoö i try again when i get in my ftp i have that bin folder and from there i have to go to the www do you mean i have to set a robot.txt to protect the bin folder. Hope you get what i mean
/Ove
|
|
