Welcome to WebmasterWorld Guest from 54.226.25.74

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Topmenu in JS or html
Is JS an option at all
11 dennisjensen 7:00 am Jan 10, 2019
Canonical for HTTPS site pointing toward HTTP
Is this a problem?
5 TomSnow 1:11 am Jan 9, 2019
Which is loaded first, robots.txt or htaccess?
Hiding a test site from bots, can both be deployed?
13 dennisjensen 9:18 am Nov 20, 2018
Canonical of Site Root
Lighthouse Error?
6 IanTurner 8:49 am Nov 10, 2018
Robots.txt code format.
13 born2run 7:24 pm Oct 10, 2018
User-agent: Mediapartners-Google Disallow
4 Tom1961 9:07 am Sep 21, 2018
robots.txt now deletes archive.org wayback machine
18 tangor 5:59 pm Sep 19, 2018
robots.txt using host: and config for cdn
10 Milchan 7:25 pm Aug 29, 2018
Disallowing a specific Bot using Robots.txt
10 lucy24 7:21 pm May 16, 2018
mobile friendly checker finding phantom robots.txt
3 kvetina 4:23 pm May 3, 2018
What Makes a Good robots.txt File?
14 Broaster 9:20 pm Apr 23, 2018
Meta Tags in SERPs
7 John134 10:30 am Apr 11, 2018
Google News indexing issues
Google News indexing issues
3 Robbo55 2:26 pm Mar 26, 2018
Regarding question on sitemaps and http to https
3 Boulos 2:51 am Mar 22, 2018
Structured Data Mark Up
Need to know which mark up data for the website
7 Irfan_Ansari 1:14 pm Feb 20, 2018
Robots.txt Guidance
Need to Correct Answer or Solution for My Query
11 Irfan_Ansari 7:24 am Feb 20, 2018
Deleted old articles so now have to update sitemap?
3 born2run 4:01 am Feb 18, 2018
move to https and sitemaps
10 fatpeter 11:18 am Jan 23, 2018
“google sent me”
5 lucy24 7:27 pm Jan 12, 2018
New "Well-Known Uniform Resource Identifiers"
Well-Known Uniform Resource Identifiers are defined in RFC-5785
5 Andova_Begarin 6:19 pm Dec 8, 2017
robots.txt: prevent new bots en masse
configuring robots.txt to black all bots but the ones you want
6 Andova_Begarin 6:46 am Nov 27, 2017
Static site plus blog on root subfolder – separate site maps or not?
3 jason2004 12:02 pm Nov 6, 2017
robots.txt and HTTPS
8 lucy24 11:50 am Oct 19, 2017
Is there a way to block thumbnails in robots.txt?
7 virtualreality 11:06 pm Sep 14, 2017
The Rise & Fall of Robots.txt
Perspective on the Robots Text Standard
2 keyplyr 4:18 am Sep 8, 2017