homepage Welcome to WebmasterWorld Guest from 54.166.84.82
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconDoes this work?
  Disallow: /somepath/*/file/*
6DoppyNL 5:36 pm on Mar 13, 2004 (gmt 0)
newpost thread iconCCGCrawl -- custom crawler/harvester software
  New crawler software potential for abuse
2privacyman 1:38 pm on Mar 13, 2004 (gmt 0)
newpost thread iconHow to exclude bots from specific domains if all point to same place?
  We have foreign domains that all point to the same folder
2alpine 6:06 am on Mar 13, 2004 (gmt 0)
newpost thread iconYahoo Images
 
2RammsteinNicCage 5:56 am on Mar 13, 2004 (gmt 0)
newpost thread iconWho is NaverBot-1.0 (NHN Corp.
  Just wondering what bot this is....
2RobbieD 11:51 am on Mar 12, 2004 (gmt 0)
newpost thread iconmeta-robots tag; wil link be followed or not?
 
2DoppyNL 5:30 am on Mar 12, 2004 (gmt 0)
newpost thread iconrobots.txt not working
  Disallows are still being indexed
3jchance 5:21 am on Mar 12, 2004 (gmt 0)
newpost thread iconCan't seem to block Xenu
 
6WebBender 12:55 am on Mar 12, 2004 (gmt 0)
newpost thread iconphp and 'include' files
 
3charlie12345 3:19 am on Mar 11, 2004 (gmt 0)
newpost thread iconWeird robot... Google?
 
2motorhaven 2:14 am on Mar 11, 2004 (gmt 0)
newpost thread iconnewbie question
 
2maxidrom11 2:10 am on Mar 11, 2004 (gmt 0)
newpost thread iconHow To Tell Robots to Index My Site Everyday
  I need it to come everyday and update my website content everday
2dhaliwal 9:01 am on Mar 9, 2004 (gmt 0)
newpost thread iconSome robot keeps calling a 301 page
  about 10000 entries till I banned the IP
1walkman 9:25 pm on Mar 8, 2004 (gmt 0)
newpost thread iconHow to Disallow Jeeves?
 
5skuba 7:03 pm on Mar 5, 2004 (gmt 0)
newpost thread iconIs this done correctly
 
2mikemcs 3:00 pm on Mar 3, 2004 (gmt 0)
newpost thread iconNo Robots.txt page
  what's the harm?
7dauction 4:11 am on Mar 2, 2004 (gmt 0)
newpost thread iconRobots.txt files
  Who to Disallow
6Uniterra 5:08 pm on Mar 1, 2004 (gmt 0)
newpost thread iconHow to disallow doorway pages from indexing?
 
4Voyteck 2:30 pm on Mar 1, 2004 (gmt 0)
newpost thread iconDirectory patterns in query strings
  Will robots.txt block them?
2Wail 8:13 pm on Feb 29, 2004 (gmt 0)
newpost thread iconlist of bots to ban
  it's getting ridiculous but don't want to ban any useful ones
2walkman 4:48 pm on Feb 29, 2004 (gmt 0)
newpost thread iconshould the file .htaccess visible
 
7newbies 3:44 am on Feb 28, 2004 (gmt 0)
newpost thread iconCan I disallow an entire directory but one file?
  too many files to list, plus I'd be advertising them
6walkman 4:13 am on Feb 26, 2004 (gmt 0)
newpost thread iconRobots.txt and Index,nofollow
  do they have to match
4graywolf 10:38 pm on Feb 22, 2004 (gmt 0)
newpost thread iconNew to Robots
  New to Robots
7gotmallz 8:32 pm on Feb 22, 2004 (gmt 0)
newpost thread iconhmmm confused
  this makes no sense
1gotmallz 6:15 am on Feb 22, 2004 (gmt 0)
Displaying Page 60 of 72<< Back 1 ... 50 51 52 53 54 55 56 57 58 59 [ 60 ] 61 62 63 64 65 66 67 68 69 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved