homepage Welcome to WebmasterWorld Guest from 54.197.65.82
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconI am writing my 1st robots.txt :) - never bothered before
  Anyone know any crawlers I should add ??
3diddlydazz 4:11 am on Jan 12, 2002 (gmt 0)
newpost thread iconMSIECrawler not obeying my robots.txt :(
  any ideas ?
5diddlydazz 3:09 am on Jan 12, 2002 (gmt 0)
newpost thread iconRobots.txt standard
 
4count_zer0 11:21 am on Dec 7, 2001 (gmt 0)
newpost thread iconrobots.txt
 
7Kenny B 1:02 pm on Sep 25, 2001 (gmt 0)
newpost thread iconRobots.txt specific query
 
8IanTurner 7:19 pm on Sep 24, 2001 (gmt 0)
newpost thread iconbasic robots.txt question
 
5kapow 4:31 pm on Aug 2, 2001 (gmt 0)
newpost thread iconI am new to all of this about Spiders...Robot.txt
  Help Message
8lightwriter 12:42 am on Jul 23, 2001 (gmt 0)
newpost thread iconrobots.txt
  Banning dodgy spiders - do they take any notice?
4themoff 7:35 pm on Jul 13, 2001 (gmt 0)
newpost thread iconRobots.txt
  Does Google honour it....
12Rusky 5:58 am on Jul 5, 2001 (gmt 0)
newpost thread iconFear and Loathing of Robots.txt
  Coming Clean - Confessions of a a robo'phobic
12Brett_Tabke 1:23 am on Jun 25, 2001 (gmt 0)
newpost thread iconBad style in robots.txt file. Why?
 
5billyblu 10:25 am on Jun 1, 2001 (gmt 0)
newpost thread iconRobots.txt Validator
 
4Froggyman 9:17 am on May 26, 2001 (gmt 0)
newpost thread iconRobots.txt
  Is there such a thing as Allow: *
4merlin78 2:38 am on May 24, 2001 (gmt 0)
newpost thread iconRobots.txt file
  How do I do this...
2lizzie 8:27 pm on May 19, 2001 (gmt 0)
newpost thread iconRobots.txt
  How effective is it anyway
6circuitjump 2:35 pm on May 11, 2001 (gmt 0)
newpost thread iconRobozilla and Robots.txt
 
4Brett_Tabke 8:11 am on May 5, 2001 (gmt 0)
newpost thread iconreplacing robots.txt with a CGI
 
7theperlyking 7:42 pm on Apr 25, 2001 (gmt 0)
newpost thread iconrobots.txt
  warning; robots.txt containing DOS or Mac line
5roma 10:35 am on Apr 25, 2001 (gmt 0)
newpost thread iconRobots.txt
  Do you use it if you want to all all robots?
7WebSeeker 8:30 am on Apr 16, 2001 (gmt 0)
newpost thread iconLooking for complicated borderline Robots.txt
 
4Brett_Tabke 10:14 am on Apr 9, 2001 (gmt 0)
newpost thread iconGoogle Link points and Robots.txt[2] ( 1 2 )
  Google points algo and side discussion of robots.txt format
34tmw 2:53 pm on Mar 27, 2001 (gmt 0)
newpost thread iconDoorway / Hallway pages Vs. Robots.txt
  Does one need to build "many" sites under one roof?
5Garyh 10:45 pm on Mar 20, 2001 (gmt 0)
newpost thread iconYahoo and Robots.txt
 
2Brett_Tabke 9:16 pm on Feb 28, 2001 (gmt 0)
newpost thread iconHow does robot.txt work?
 
4Acternaweb 2:59 am on Feb 28, 2001 (gmt 0)
newpost thread iconMr. Roboto
  How do I set up a Robots.txt file?
6CaveToad 3:57 pm on Jan 31, 2001 (gmt 0)
Displaying Page 71 of 72<< Back 1 ... 61 62 63 64 65 66 67 68 69 70 [ 71 ] 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved