homepage Welcome to WebmasterWorld Guest from 54.163.139.36
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconselectively including folders
  that are beneath a folder excluded
2stu2 5:20 am on Jul 17, 2005 (gmt 0)
newpost thread iconWould this work? Robots exclusion or considered cloaking?
 
5dvldvl3 5:11 am on Jul 17, 2005 (gmt 0)
newpost thread iconPreviously Indexed Files-Want To Exclude Now
  RobotsTxt Change
3buview1 12:23 pm on Jul 16, 2005 (gmt 0)
newpost thread iconGoogle only hits robots and index
  Google only hits robots and index
10rostom 3:51 pm on Jul 15, 2005 (gmt 0)
newpost thread iconDisallowing & in url
 
2Wolvereness 2:54 pm on Jul 12, 2005 (gmt 0)
newpost thread iconrobots.txt verses sitemap.html verses OPML file
 
2Neal 8:39 pm on Jul 11, 2005 (gmt 0)
newpost thread iconJust added robots.txt file
 
4soccrt 4:41 am on Jul 11, 2005 (gmt 0)
newpost thread iconblock all except
 
2Radial 9:31 pm on Jul 7, 2005 (gmt 0)
newpost thread iconrobots.txt with? in URL
 
16jozomannen 11:17 am on Jul 6, 2005 (gmt 0)
newpost thread iconHow do i add robot.txt?
 
10genny 4:38 am on Jul 6, 2005 (gmt 0)
newpost thread iconInclude files, spiders, and robots.txt
 
4rottsnroses 11:24 pm on Jul 3, 2005 (gmt 0)
newpost thread iconHow to Stop BecomeBot?
 
12shilmy 3:09 pm on Jun 29, 2005 (gmt 0)
newpost thread iconGoogle ignoring my robots.txt?
 
9homesby 12:17 am on Jun 29, 2005 (gmt 0)
newpost thread iconRobots meta tag *CORRECT* syntax
  Space* or no space for "index,*follow", and case sensitive?
6Clint 12:56 pm on Jun 28, 2005 (gmt 0)
newpost thread iconQuestion about Robots with multiple similar sites
  Question about Robots with multiple similar sites
2ccash 9:54 pm on Jun 27, 2005 (gmt 0)
newpost thread iconRobots/Slurp, etc - what does this mean
  I am a newbie - not sure what this means
4ramnyc 3:01 pm on Jun 22, 2005 (gmt 0)
newpost thread iconDoes adding robot text help your site get crawled more often?
  Advantages of robot text?
6moneyangel13 8:04 am on Jun 22, 2005 (gmt 0)
newpost thread iconwildcards in page names
  Disallow: /store/scripts/emailFriend.asp*
5jimsthoughts 3:54 pm on Jun 21, 2005 (gmt 0)
newpost thread iconHow can I allow robots to access my pages
  inktomi is following links it should not
4expat123 10:08 am on Jun 21, 2005 (gmt 0)
newpost thread iconUpdating robots.txt and Crawlers behaviour
  How to tell bots to come back when you changed robots.txt banning them
6silverbytes 6:57 pm on Jun 20, 2005 (gmt 0)
newpost thread iconslurp ignoring disallow?
  -or have i got it wrong?
4tallis 12:09 pm on Jun 20, 2005 (gmt 0)
newpost thread iconQuestion about blocking 1 robot, but allow all others
 
8capsmaster 3:26 pm on Jun 18, 2005 (gmt 0)
newpost thread iconFrequencey of GET /robots.txt
  How often should a crawler/spider/robot ask for robots.txt?
9Dijkgraaf 5:08 am on Jun 18, 2005 (gmt 0)
newpost thread iconNeed Clarification On Directory Exclusions
 
10jonrichd 5:00 am on Jun 18, 2005 (gmt 0)
newpost thread iconRobots.txt and Mod Rewrite
  Which url does the robot read?
2eflouret 4:52 am on Jun 18, 2005 (gmt 0)
Displaying Page 46 of 72<< Back 1 ... 36 37 38 39 40 41 42 43 44 45 [ 46 ] 47 48 49 50 51 52 53 54 55 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved