homepage Welcome to WebmasterWorld Guest from 54.166.100.8
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconrobots.txt with? in URL
 
16jozomannen 11:17 am on Jul 6, 2005 (gmt 0)
newpost thread iconHow do i add robot.txt?
 
10genny 4:38 am on Jul 6, 2005 (gmt 0)
newpost thread iconInclude files, spiders, and robots.txt
 
4rottsnroses 11:24 pm on Jul 3, 2005 (gmt 0)
newpost thread iconHow to Stop BecomeBot?
 
12shilmy 3:09 pm on Jun 29, 2005 (gmt 0)
newpost thread iconGoogle ignoring my robots.txt?
 
9homesby 12:17 am on Jun 29, 2005 (gmt 0)
newpost thread iconRobots meta tag *CORRECT* syntax
  Space* or no space for "index,*follow", and case sensitive?
6Clint 12:56 pm on Jun 28, 2005 (gmt 0)
newpost thread iconQuestion about Robots with multiple similar sites
  Question about Robots with multiple similar sites
2ccash 9:54 pm on Jun 27, 2005 (gmt 0)
newpost thread iconRobots/Slurp, etc - what does this mean
  I am a newbie - not sure what this means
4ramnyc 3:01 pm on Jun 22, 2005 (gmt 0)
newpost thread iconDoes adding robot text help your site get crawled more often?
  Advantages of robot text?
6moneyangel13 8:04 am on Jun 22, 2005 (gmt 0)
newpost thread iconwildcards in page names
  Disallow: /store/scripts/emailFriend.asp*
5jimsthoughts 3:54 pm on Jun 21, 2005 (gmt 0)
newpost thread iconHow can I allow robots to access my pages
  inktomi is following links it should not
4expat123 10:08 am on Jun 21, 2005 (gmt 0)
newpost thread iconUpdating robots.txt and Crawlers behaviour
  How to tell bots to come back when you changed robots.txt banning them
6silverbytes 6:57 pm on Jun 20, 2005 (gmt 0)
newpost thread iconslurp ignoring disallow?
  -or have i got it wrong?
4tallis 12:09 pm on Jun 20, 2005 (gmt 0)
newpost thread iconQuestion about blocking 1 robot, but allow all others
 
8capsmaster 3:26 pm on Jun 18, 2005 (gmt 0)
newpost thread iconFrequencey of GET /robots.txt
  How often should a crawler/spider/robot ask for robots.txt?
9Dijkgraaf 5:08 am on Jun 18, 2005 (gmt 0)
newpost thread iconNeed Clarification On Directory Exclusions
 
10jonrichd 5:00 am on Jun 18, 2005 (gmt 0)
newpost thread iconRobots.txt and Mod Rewrite
  Which url does the robot read?
2eflouret 4:52 am on Jun 18, 2005 (gmt 0)
newpost thread iconhow to ban Google from indexing ALL .php files
  I'm using rewrite and don't want a dupe penalty
16walkman 5:12 pm on Jun 15, 2005 (gmt 0)
newpost thread iconboitho.com bot violating robots.txt[4] ( 1 2 3 4 )
  Specifically requested only forbidden files
111jazzguy 8:30 pm on Jun 14, 2005 (gmt 0)
newpost thread iconrobots.txt and Google on intranet
  robots.txt not working
4John488 8:11 pm on Jun 13, 2005 (gmt 0)
newpost thread iconSpecific Spiders do need more bandwidth?
  Leaving Gbot or Msn does other robots take more bandwidth when crawling?
2bnvrc 12:39 pm on Jun 13, 2005 (gmt 0)
newpost thread iconrobots.text ..please help
  help with robots.text file
2dmiller63 12:35 pm on Jun 13, 2005 (gmt 0)
newpost thread iconRobot,txt Help
 
8branmh 6:34 pm on Jun 12, 2005 (gmt 0)
newpost thread iconGoogle Robots.txt Wildcard Feature
  Please take a look at my robot.txt
4seunosewa 7:02 am on Jun 11, 2005 (gmt 0)
newpost thread iconSeparate robots.txt file for secure site
  How do i block one and not the other?
3Vimes 3:34 am on Jun 10, 2005 (gmt 0)
Displaying Page 46 of 71<< Back 1 ... 36 37 38 39 40 41 42 43 44 45 [ 46 ] 47 48 49 50 51 52 53 54 55 ... 71 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved