What is the consequence of blocking a website through robots.txt ?
guyaftab001
5:41 am on Sep 11, 2022 (gmt 0)
What is the consequence of blocking a website through robots.txt . And then restoring it 6 months later. So what happens if I block a site from google for six months.
Suraj
11:02 am on Sep 15, 2022 (gmt 0)
Robot.txt files prevent search engine spiders from accessing a URL. your Web page URL will still appear on SERP but there won't be any images, descriptions, audio files, or videos. This won't, however, fully prevent a website from showing up in search results on Google. So, if you want to protect a website from Googlebots, you may use the noindex meta tag or you can also password-protect the page. There is no time limit on how long you can use this file; you are free to do so.
tangor
4:29 pm on Sep 15, 2022 (gmt 0)
So what happens if I block a site from google for six months.
Most likely your site will continue to appear in the serps, but will have something to the effect:
"We'd like to show you what this site is about but access is denied by robots.txt"
How potential visitors react is the question. :)
Just curious, what is the reasoning to denying the largest search engine in the world?
martinibuster
5:10 pm on Sep 15, 2022 (gmt 0)
....what is the reasoning to denying the largest search engine in the world?
Page could be thin Section could be under developed Page/sections could be duplicate and in need of improvement
Many reasons for blocking crawlers. ;)
lucy24
5:26 pm on Sep 15, 2022 (gmt 0)
:: quick run to logs to check something ::
One thing that will not happen within 6 months is G###--or any other search engine--forgetting your site. If you drop the Disallow (“block” is the wrong word for robots.txt) they will notice within a few days at most, after which they may well re-crawl the entire site.
But always think carefully about which of two mutually exclusive options is best for your site, or for specific pages thereon: either deny crawling, or set it to "noindex".
