Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: martinibuster
If your page is publically accessible by a browser (and has people actually visiting it) then expect the big search engines to find it evntually.
This does have worrying issues for managing a site, looks at though banning robots and other measures are required for pages we previously didn't think would end up in the SERPS.
At SES Chicago I asked a Yahoo rep is this would be a problem as far as duplicate content penalties. He didn't seem to understand my concern and only questioned how Yahoo would know about these URLs. I said I guessed it was the toolbar.
More recently, I asked the question on Yahoo's search blog because I am concerned about a duplicate content penalty (the URLs show up in SiteExplorer, too). No answer.
[edited by: MLHmptn at 2:08 pm (utc) on Jan. 21, 2007]
Keeping Ad Tracking and Dead URLs out of Yahoo! Search [ysearchblog.com]
If you don't want Yahoo! Slurp, our Web crawler to index these URLs you can use wildcards in robots.txt. For example, if you are using the parameter 'ref' to track ad sources, you can use a rule like the one below to keep your tracking URLs from being Slurped:
User-Agent: Yahoo! Slurp