Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: goodroi
2. make a list of everything on your site you DON'T want robots/spiders to visit, and list in in robots.txt like so:
The * after disallow means NO spider is supposed to visit the files & directories in this section.
If you only want to ban specific robots from certain files, add a second section like so (replace the * with the user-agent of the spider you want to ban):
However, I have seen it take about 3 months after a spider visting and reading robots.txt before they think about removing the disallowed pages from their index. Of course, in most cases, it is entirely in the search engines interest to remove disallowed content, because often a link to this content is useless.
I have read about it somewhere, but I forgot it :-(
Could anyone help me?
Seems to work fine.
In your case, you can write a separate disallow line for each file in that directory you don't want spidered, but still allow access to the directory itself and therefore the subdirectories:
It may result in a long robots.txt file, but it should do the trick.
This was what I suspected to be right.
I know that writing separate lines for each files would also work, but there is nearly one hundred of them. Maybe I'm too lazy :-)
Anyway it could be a warning for those who have same site structure. I really didn't want to spam SEs, these (asp) files contain one single line (response redirect), but it might led to Google's dropping me off completely! I don't know for sure, but...SEs are getting to be more and more clever.
Thank you again
The only time I use second level subdirectories is to hold administrative or data/graphics files:
I disallow everything from my image and admin directories anyway, so that keeps everything else nice and clean.... either I allow the spiders into the directory or not.
I'm lazy too... I hate having to redesign and reorganize things once I build them! A lesson for the lazy: it's worth spending the extra time to plan out a site structure at first that will allow you to be lazy later! ;)