Forum Moderators: open
Does google follow cgi-bin directory links at all?
Placing the cgi-bin as a forbidden directory on the robot.txt file could do anyting good/bad for my website except for that google won't crawl that dir?
(lower pr or anyting like that)?
Also how does my robot.txt should look like if i just want that all the robots won't crawl the cgi-bin directory but would crawl everyting else?
Thanks.
putting
User-agent: *
Disallow: /cgi-bin/
in robots.txt should do the trick, even if I recently found that Google sometimes seems to ignore my own robots.txt.
It will definitely NOT hurt your PR.
