Welcome to WebmasterWorld Guest from 50.16.47.14

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Robots INclusions standard?
13 mikomido 12:28 am Sep 9, 2007
blocking the hostsite but not the hosted sites
3 messemaeker 12:12 pm Sep 7, 2007
Blocking Windows Live robot/spider.
2 Tidal2 10:56 am Sep 7, 2007
Do we have to use robots.txt on every website?
for robots.txt issues
5 masood82 8:42 am Sep 6, 2007
How to block tags?
hundreds of undesired links !
3 rashe18 4:33 pm Sep 4, 2007
Blocked through robots.txt . will the page be crawled ever
5 getxb 6:02 am Aug 31, 2007
URI Only Listing
Need a refresher course on this...
9 pageoneresults 12:20 pm Aug 24, 2007
robot.txt Security
Can be read publically
4 kkonline 10:08 am Aug 24, 2007
robots.txt="noindex,follow"?
2 davidcamus 9:55 am Aug 24, 2007
how to block a page
on a particular location
7 experienced 9:47 am Aug 24, 2007
Multiple "Sitemap: " lines in robots.txt?
Several TLDs use same site code / robots.txt.
2 geoffyp 1:33 pm Aug 10, 2007
Baiduspider and multiple user-agent text file.
4 Vimes 12:04 pm Aug 7, 2007
How can you Disallow a Sub-Domain?
2 zammo 10:11 am Jul 31, 2007
Google Exclude .swf Files
How do i exclude swf files from indexing by google
5 nxfx 5:26 pm Jul 30, 2007
How to disallow entire content, except a particular location
2 brakkar 12:07 pm Jul 30, 2007
disallowing url midway
2 defanjos 5:20 pm Jul 29, 2007
Block sub directory
5 maxbear 5:24 am Jul 28, 2007
Does this work?
disallow multiple bots at once...
3 Devilfear 6:48 am Jul 27, 2007
Using multiple hash marks
Does that conform to protocol?
5 Reno 7:17 pm Jul 25, 2007
Disallowing Iframed pages
2 Ozz13 1:29 pm Jul 23, 2007
Proper format for disallow of URL
4 Ozz13 1:14 pm Jul 23, 2007
What the dilly?
3 tonynoriega 5:39 pm Jul 18, 2007
Re-instate pages previously blocked
sorry for this noob question
4 dibbern2 11:23 pm Jul 16, 2007
Broken Links on my web site
I have too many broken links
2 erdsah88 12:47 pm Jul 16, 2007
"Disallow: //" - error or meaningful?
robots.txt crawler error
5 sitetruth 11:58 pm Jul 11, 2007