I have just noticed that my robots.txt was removed for some time and now my search pages are indexed making it look spammy - I have put back robots.txt with the following in my robots.txt
User-agent: *
Disallow: /search
Disallow: /?q=search/
But the page urls are like this:
http ://mysite.com/search?page= (then the number)
http ://mysite.com/search?page= (then the number)
http ://www.mysite.com/Search/page.html?= (then the page number)
So now I put that in my robots.txt does that mean that anything with the word /search area should get spidered? The first line makes this applicable to all search engines.
[edited by: goodroi at 3:56 pm (utc) on Oct 14, 2017]
[edit reason] delinked example domains [/edit]