Welcome to WebmasterWorld Guest from 126.96.36.199
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|User-agent: Mediapartners-Google Disallow
||4||Tom1961||9:07 am Sep 21, 2018|
|robots.txt now deletes archive.org wayback machine
||18||tangor||5:59 pm Sep 19, 2018|
|robots.txt using host: and config for cdn
||10||Milchan||7:25 pm Aug 29, 2018|
|Robots.txt code format.
||9||born2run||3:45 am Jul 11, 2018|
|Disallowing a specific Bot using Robots.txt
||10||lucy24||7:21 pm May 16, 2018|
|mobile friendly checker finding phantom robots.txt
||3||kvetina||4:23 pm May 3, 2018|
|What Makes a Good robots.txt File?
||14||Broaster||9:20 pm Apr 23, 2018|
|Meta Tags in SERPs
||7||John134||10:30 am Apr 11, 2018|
|Google News indexing issues
Google News indexing issues
|3||Robbo55||2:26 pm Mar 26, 2018|
|Regarding question on sitemaps and http to https
||3||Boulos||2:51 am Mar 22, 2018|
|Structured Data Mark Up
Need to know which mark up data for the website
|7||Irfan_Ansari||1:14 pm Feb 20, 2018|
Need to Correct Answer or Solution for My Query
|11||Irfan_Ansari||7:24 am Feb 20, 2018|
|Deleted old articles so now have to update sitemap?
||3||born2run||4:01 am Feb 18, 2018|
|move to https and sitemaps
||10||fatpeter||11:18 am Jan 23, 2018|
|“google sent me”
||5||lucy24||7:27 pm Jan 12, 2018|
|New "Well-Known Uniform Resource Identifiers"
Well-Known Uniform Resource Identifiers are defined in RFC-5785
|5||Andova_Begarin||6:19 pm Dec 8, 2017|
|robots.txt: prevent new bots en masse
configuring robots.txt to black all bots but the ones you want
|6||Andova_Begarin||6:46 am Nov 27, 2017|
|Static site plus blog on root subfolder – separate site maps or not?
||3||jason2004||12:02 pm Nov 6, 2017|
|robots.txt and HTTPS
||8||lucy24||11:50 am Oct 19, 2017|
|Is there a way to block thumbnails in robots.txt?
||7||virtualreality||11:06 pm Sep 14, 2017|
|The Rise & Fall of Robots.txt
Perspective on the Robots Text Standard
|2||keyplyr||4:18 am Sep 8, 2017|
|Excluding image urls from being indexed by robots.
||13||born2run||12:43 am Sep 5, 2017|
|My site gets edited every 4-5 hours, the revisit after metatag value?
||9||born2run||12:42 am Aug 27, 2017|
|Am I going to be punished for NOT having a sitemap?
||5||csdude55||11:48 pm Jun 20, 2017|
|My website inner pages are not crawling and indexed?
||5||Movierola||10:17 am May 18, 2017|