homepage Welcome to WebmasterWorld Guest from 54.234.128.25
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconrobots.txt and subdomains
  how do I block SE's from my subdomains
2onebaldguy 5:51 am on Jun 6, 2002 (gmt 0)
newpost thread iconrobots.txt question
 
2coolshop 2:21 pm on May 31, 2002 (gmt 0)
newpost thread iconUser agent in robots.txt and browscap.ini...
  Why is there a format different?
6Pushycat 11:39 pm on May 30, 2002 (gmt 0)
newpost thread iconrobots.txt -- block one user-agent, allow all others
 
2luma 1:44 am on May 23, 2002 (gmt 0)
newpost thread iconRobot.txt file help
  How to create this file
7beasscr 6:30 pm on May 16, 2002 (gmt 0)
newpost thread iconrobots.txt
  Generates a 404 error in my logs
9zeus 1:52 pm on May 3, 2002 (gmt 0)
newpost thread iconDoes robots.txt override meta tag?
 
8kira 7:12 pm on May 2, 2002 (gmt 0)
newpost thread iconRobots.txt validator updated.
 
7Brett_Tabke 6:48 pm on Apr 19, 2002 (gmt 0)
newpost thread iconWhat is robots.txt syntax
 
22Alec_Doggone 9:07 am on Apr 19, 2002 (gmt 0)
newpost thread iconSubmitting robots.txt to google
  Would google look sooner for it?
17Yidaki 8:19 am on Apr 19, 2002 (gmt 0)
newpost thread iconRobots.txt Pattern Matching
  How Well Does It Work?
5WebGuerrilla 1:15 pm on Mar 15, 2002 (gmt 0)
newpost thread iconRobots.txt Question?
  How often does Ink check for the file??
6MarkHutch 5:11 pm on Mar 11, 2002 (gmt 0)
newpost thread iconDo I need a robots.txt file in my directory?
  Google Bot - Whats Up with it?
10jady 9:58 pm on Feb 24, 2002 (gmt 0)
newpost thread iconHow important is the Robots.txt file now?
  ....in 2002
29ScottM 1:16 am on Jan 28, 2002 (gmt 0)
newpost thread iconRobots.txt Question
 
7mud 3:21 am on Jan 17, 2002 (gmt 0)
newpost thread iconI am writing my 1st robots.txt :) - never bothered before
  Anyone know any crawlers I should add ??
3diddlydazz 4:11 am on Jan 12, 2002 (gmt 0)
newpost thread iconMSIECrawler not obeying my robots.txt :(
  any ideas ?
5diddlydazz 3:09 am on Jan 12, 2002 (gmt 0)
newpost thread iconRobots.txt standard
 
4count_zer0 11:21 am on Dec 7, 2001 (gmt 0)
newpost thread iconrobots.txt
 
7Kenny B 1:02 pm on Sep 25, 2001 (gmt 0)
newpost thread iconRobots.txt specific query
 
8IanTurner 7:19 pm on Sep 24, 2001 (gmt 0)
newpost thread iconbasic robots.txt question
 
5kapow 4:31 pm on Aug 2, 2001 (gmt 0)
newpost thread iconI am new to all of this about Spiders...Robot.txt
  Help Message
8lightwriter 12:42 am on Jul 23, 2001 (gmt 0)
newpost thread iconrobots.txt
  Banning dodgy spiders - do they take any notice?
4themoff 7:35 pm on Jul 13, 2001 (gmt 0)
newpost thread iconRobots.txt
  Does Google honour it....
12Rusky 5:58 am on Jul 5, 2001 (gmt 0)
newpost thread iconFear and Loathing of Robots.txt
  Coming Clean - Confessions of a a robo'phobic
12Brett_Tabke 1:23 am on Jun 25, 2001 (gmt 0)
Displaying Page 70 of 71<< Back 1 ... 60 61 62 63 64 65 66 67 68 69 [ 70 ] 71 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved