homepage Welcome to WebmasterWorld Guest from 54.243.23.129
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconRobots/Slurp, etc - what does this mean
  I am a newbie - not sure what this means
4ramnyc 3:01 pm on Jun 22, 2005 (gmt 0)
newpost thread iconDoes adding robot text help your site get crawled more often?
  Advantages of robot text?
6moneyangel13 8:04 am on Jun 22, 2005 (gmt 0)
newpost thread iconwildcards in page names
  Disallow: /store/scripts/emailFriend.asp*
5jimsthoughts 3:54 pm on Jun 21, 2005 (gmt 0)
newpost thread iconHow can I allow robots to access my pages
  inktomi is following links it should not
4expat123 10:08 am on Jun 21, 2005 (gmt 0)
newpost thread iconUpdating robots.txt and Crawlers behaviour
  How to tell bots to come back when you changed robots.txt banning them
6silverbytes 6:57 pm on Jun 20, 2005 (gmt 0)
newpost thread iconslurp ignoring disallow?
  -or have i got it wrong?
4tallis 12:09 pm on Jun 20, 2005 (gmt 0)
newpost thread iconQuestion about blocking 1 robot, but allow all others
 
8capsmaster 3:26 pm on Jun 18, 2005 (gmt 0)
newpost thread iconFrequencey of GET /robots.txt
  How often should a crawler/spider/robot ask for robots.txt?
9Dijkgraaf 5:08 am on Jun 18, 2005 (gmt 0)
newpost thread iconNeed Clarification On Directory Exclusions
 
10jonrichd 5:00 am on Jun 18, 2005 (gmt 0)
newpost thread iconRobots.txt and Mod Rewrite
  Which url does the robot read?
2eflouret 4:52 am on Jun 18, 2005 (gmt 0)
newpost thread iconhow to ban Google from indexing ALL .php files
  I'm using rewrite and don't want a dupe penalty
16walkman 5:12 pm on Jun 15, 2005 (gmt 0)
newpost thread iconboitho.com bot violating robots.txt[4] ( 1 2 3 4 )
  Specifically requested only forbidden files
111jazzguy 8:30 pm on Jun 14, 2005 (gmt 0)
newpost thread iconrobots.txt and Google on intranet
  robots.txt not working
4John488 8:11 pm on Jun 13, 2005 (gmt 0)
newpost thread iconSpecific Spiders do need more bandwidth?
  Leaving Gbot or Msn does other robots take more bandwidth when crawling?
2bnvrc 12:39 pm on Jun 13, 2005 (gmt 0)
newpost thread iconrobots.text ..please help
  help with robots.text file
2dmiller63 12:35 pm on Jun 13, 2005 (gmt 0)
newpost thread iconRobot,txt Help
 
8branmh 6:34 pm on Jun 12, 2005 (gmt 0)
newpost thread iconGoogle Robots.txt Wildcard Feature
  Please take a look at my robot.txt
4seunosewa 7:02 am on Jun 11, 2005 (gmt 0)
newpost thread iconSeparate robots.txt file for secure site
  How do i block one and not the other?
3Vimes 3:34 am on Jun 10, 2005 (gmt 0)
newpost thread iconBad bots
 
15ownerrim 4:25 am on Jun 8, 2005 (gmt 0)
newpost thread iconIf I disallow /filename.php ...
  will google still index /filename.php?whatever
7Trisha 7:05 am on Jun 7, 2005 (gmt 0)
newpost thread iconAgent Name for MSN
  Where can I find this information.
8Pat1975 5:55 am on Jun 3, 2005 (gmt 0)
newpost thread iconGeneral questions
  General robots.txt queries
7makzan 11:49 pm on Jun 2, 2005 (gmt 0)
newpost thread iconDisallow /file.php same as Diallow file.php?
 
6One_on_One 3:43 am on Jun 2, 2005 (gmt 0)
newpost thread iconStopping spiders
  from Competitors Analysis Tools...
4ambiorix_rb 1:21 pm on Jun 1, 2005 (gmt 0)
newpost thread iconSEO / robots.txt question
  unintentional cross-linking penalty
1wolfgang 6:27 pm on May 31, 2005 (gmt 0)
Displaying Page 46 of 71<< Back 1 ... 36 37 38 39 40 41 42 43 44 45 [ 46 ] 47 48 49 50 51 52 53 54 55 ... 71 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved