Welcome to WebmasterWorld Guest from 184.108.40.206
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|“google sent me”
||5||lucy24||7:27 pm Jan 12, 2018|
|New "Well-Known Uniform Resource Identifiers"
Well-Known Uniform Resource Identifiers are defined in RFC-5785
|5||Andova_Begarin||6:19 pm Dec 8, 2017|
|move to https and sitemaps
||9||fatpeter||2:03 pm Dec 6, 2017|
|robots.txt: prevent new bots en masse
configuring robots.txt to black all bots but the ones you want
|6||Andova_Begarin||6:46 am Nov 27, 2017|
|Static site plus blog on root subfolder – separate site maps or not?
||3||jason2004||12:02 pm Nov 6, 2017|
|robots.txt and HTTPS
||8||lucy24||11:50 am Oct 19, 2017|
|Is there a way to block thumbnails in robots.txt?
||7||virtualreality||11:06 pm Sep 14, 2017|
|The Rise & Fall of Robots.txt
Perspective on the Robots Text Standard
|2||keyplyr||4:18 am Sep 8, 2017|
|Excluding image urls from being indexed by robots.
||13||born2run||12:43 am Sep 5, 2017|
|My site gets edited every 4-5 hours, the revisit after metatag value?
||9||born2run||12:42 am Aug 27, 2017|
|Am I going to be punished for NOT having a sitemap?
||5||csdude55||11:48 pm Jun 20, 2017|
|My website inner pages are not crawling and indexed?
||5||Movierola||10:17 am May 18, 2017|
|Strange 404's showing up in GSC
 ( 1 2 )
|40||NickMNS||1:40 pm Apr 24, 2017|
|Sudden drop in traffic from blog on subdomain
figuring out the reason in traffic from blog
|1||Sachet||2:33 pm Mar 17, 2017|
|Site not indexed?
||12||kingbottler||10:11 am Mar 14, 2017|
|Which tags to keep or toss
||6||csdude55||8:25 am Feb 7, 2017|
|Disallow images with suffix
||4||terrybarnes||12:28 pm Jan 13, 2017|
|webmaster SiteLink option
I can't find search appearance > sitelink option
|2||Shihab||12:43 pm Jan 10, 2017|
|Sitemaps File Size up to 50Mb
||4||engine||11:27 pm Dec 12, 2016|
|Blog Tag and Author page blocking with robots.txt
||5||seojustin||6:27 am Dec 3, 2016|
|Robots - Disallow All Except Images, JS & CSS
||2||almo136||11:02 pm Oct 5, 2016|
|Please explain what I do?
||6||williamgeff||6:59 am Sep 19, 2016|
|Is it normal for a browser to request filelist.xml and editdata.mso?
what or why is filelist.xlm and editdata.mso being requested?
|1||SumGuy||3:01 am Sep 8, 2016|
|no indexing only certain pages in wordpress
||2||romerome||7:26 pm Sep 5, 2016|
must be an old WebmasterWorld suggestion for robots.txt
|3||AussieWebmaster||11:48 pm Jul 20, 2016|