Welcome to WebmasterWorld Guest from 34.238.194.166

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Search being indexed

     
3:41 am on Oct 14, 2017 (gmt 0)

Junior Member

Top Contributors Of The Month

joined:May 3, 2017
posts:112
votes: 6


I have just noticed that my robots.txt was removed for some time and now my search pages are indexed making it look spammy - I have put back robots.txt with the following in my robots.txt

User-agent: *
Disallow: /search
Disallow: /?q=search/

But the page urls are like this:

http ://mysite.com/search?page= (then the number)
http ://mysite.com/search?page= (then the number)
http ://www.mysite.com/Search/page.html?= (then the page number)

So now I put that in my robots.txt does that mean that anything with the word /search area should get spidered? The first line makes this applicable to all search engines.

[edited by: goodroi at 3:56 pm (utc) on Oct 14, 2017]
[edit reason] delinked example domains [/edit]

9:46 pm on Oct 14, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


You can also set that parameter to be ignored in GSC > Crawl > URL Parameters
10:44 pm on Oct 14, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15757
votes: 828


Is
example.com/search
an actual page that physically exists? If so, an alternative approach is to slap a "noindex" meta on it.

If a given URL has already been indexed, disallowing it in robots.txt will not immediately remove it from the index, though it will disappear eventually. In the meantime, you can use the Remove feature in GSC.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members