homepage Welcome to WebmasterWorld Guest from 23.20.33.176
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe and Support WebmasterWorld
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconNeed quick help
  Prevent robots from clicking on PPC links
2hostlead 11:11 am on Jan 27, 2004 (gmt 0)
newpost thread iconhow to let robot crawl the links at a page but not index this page?
  index, robot
2johnlim 11:39 pm on Jan 26, 2004 (gmt 0)
newpost thread iconWildcard use in robots.txt
  Need to disallow spidering of graphics files
2Probably 4:58 am on Jan 26, 2004 (gmt 0)
newpost thread iconrobots.txt and htaccess
  .htaccess circumvents to site root
4stefan009 5:35 pm on Jan 24, 2004 (gmt 0)
newpost thread iconAltavista Scooter eats up too much trafic
 
2maxidrom11 5:28 am on Jan 24, 2004 (gmt 0)
newpost thread iconDoes every site need a robots.txt?
  I have one, but I'm not sure I need it
2MatthewHSE 4:00 pm on Jan 23, 2004 (gmt 0)
newpost thread iconcreating robots.txt
 
5artdog 12:21 pm on Jan 23, 2004 (gmt 0)
newpost thread iconMy page has never been indexed
 
2kknusa 6:45 am on Jan 23, 2004 (gmt 0)
newpost thread iconSubdomains
 
2BBB_Guy 6:38 am on Jan 23, 2004 (gmt 0)
newpost thread iconIndexed pages that are disalowed by robots.txt
  is this normal?
10WebJoe 7:03 pm on Jan 20, 2004 (gmt 0)
newpost thread iconGoogle ignoring robots.txt
  No real surprise, but a real pain
9voice220 4:12 pm on Jan 19, 2004 (gmt 0)
newpost thread iconIs there any bots to block?
  Else than the infamous "Web Content International"?
3Daemon 11:56 pm on Jan 16, 2004 (gmt 0)
newpost thread iconRobots.txt help
 
5mikemcs 1:04 pm on Jan 16, 2004 (gmt 0)
newpost thread iconWho are these spiders and what do they want?
 
7obsos 4:51 pm on Jan 14, 2004 (gmt 0)
newpost thread iconRevisit instruction for robots
  I want to specify different revisit instruction different sections
3adityamodi 5:13 am on Jan 10, 2004 (gmt 0)
newpost thread iconHave I been blocking Altavista spider from spidering my site?
 
6irock 6:10 pm on Jan 9, 2004 (gmt 0)
newpost thread iconrobot.txt to remove an alias
  Afraid of duplicated content
2Ulkari 5:48 pm on Jan 8, 2004 (gmt 0)
newpost thread iconSpider Names
  I need the names of some SEs usernames
3The_Young_One 2:47 am on Jan 8, 2004 (gmt 0)
newpost thread iconBlocking Includes (SSI)
  blocking ssi
3direktor1 4:03 pm on Jan 7, 2004 (gmt 0)
newpost thread iconSubdomains
  How to exclude
4gosman 2:54 pm on Jan 6, 2004 (gmt 0)
newpost thread iconDoes the leading / make any difference?
  Sorry, I'm new to robots.txt
2AsleepATheWheel 1:07 am on Jan 6, 2004 (gmt 0)
newpost thread iconfaulty robots.txt?
 
5tenaka 2:13 pm on Jan 3, 2004 (gmt 0)
newpost thread iconOnly those on the list are allowed entrance
  All others OUT
7grnidone 5:29 pm on Jan 2, 2004 (gmt 0)
newpost thread icontxt text for website
  txt
4krikri 5:11 pm on Jan 2, 2004 (gmt 0)
newpost thread iconCheck my robots.txt please
  Don't want to block adsense bot
3conroy 2:10 pm on Jan 2, 2004 (gmt 0)
Displaying Page 62 of 71<< Back 1 ... 52 53 54 55 56 57 58 59 60 61 [ 62 ] 63 64 65 66 67 68 69 70 71 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved