homepage Welcome to WebmasterWorld Guest from 54.211.231.221
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe and Support WebmasterWorld
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread icon*/? */index.*? is a must in robots.txt
  I see /?widgetid=45 indexed
3AjiNIMC 8:17 am on Jan 30, 2008 (gmt 0)
newpost thread iconRobots.txt (/index.jsp?pg name=ccpm/lob/sublob/pgname&site=a1)
  Pattern on parameters
2Surendra 12:20 am on Jan 29, 2008 (gmt 0)
newpost thread iconDenying access to everything but the home page?
  Using robots.txt to only allow robot access to the home page
3monkeylytics 8:00 pm on Jan 28, 2008 (gmt 0)
newpost thread iconControl Robots with Meta or robots.txt
 
3driller41 12:21 pm on Jan 15, 2008 (gmt 0)
newpost thread iconRobots.txt disallowed.
 
12g1smd 12:18 am on Jan 15, 2008 (gmt 0)
newpost thread iconrobots.txt with wildcard?
 
3tranceformer 12:39 am on Jan 13, 2008 (gmt 0)
newpost thread iconDisallowing with robots.txt
  robots.txt, disallow certain urls
3jim_knopf 12:36 am on Jan 13, 2008 (gmt 0)
newpost thread iconGetting correct user agent
  getting from access log the correct user agent for robots.txt
2hessodreamy 9:03 pm on Jan 11, 2008 (gmt 0)
newpost thread iconScrapers checking robots.txt
  Spiders arrived on heels of robots.txt deployment
4timster 1:29 am on Jan 4, 2008 (gmt 0)
newpost thread icon404 Errors in Webmaster Console
  What to do with these pages?
2rowtc2 11:35 pm on Jan 3, 2008 (gmt 0)
newpost thread icondirectory and subdirectory access
  allowing access to subdirectory but not main directory
3Mr_Cool_Dude 6:11 pm on Jan 2, 2008 (gmt 0)
newpost thread iconrobots.txt question
  robots.txt examples
2jim_knopf 5:43 pm on Jan 2, 2008 (gmt 0)
newpost thread iconReference robots.txt file?
  Does anyone have one?
3mmiller 11:30 pm on Dec 25, 2007 (gmt 0)
newpost thread iconhow to manage cname in robot
  don't was to be indexed
4experienced 2:10 pm on Dec 21, 2007 (gmt 0)
newpost thread icondisallow vs ban in .htaccess
 
12joergnw10 11:10 am on Dec 16, 2007 (gmt 0)
newpost thread iconHow do I prevent the SEs from crawling any links to my site w/HTTPS
  I don't want the crawlers to index anything w/ HTTPS protocol
3ZydoSEO 8:37 pm on Dec 9, 2007 (gmt 0)
newpost thread iconyahoo google msn etc.
  other searches
1ComputerHermit 5:56 pm on Dec 7, 2007 (gmt 0)
newpost thread iconProposal for robots.txt To Have Greater Flexibility
 
12engine 4:58 pm on Nov 30, 2007 (gmt 0)
newpost thread iconset crawl rate permanently to faster
 
1agushalim 5:39 pm on Nov 28, 2007 (gmt 0)
newpost thread iconWildcards on robots.txt disallows
 
11MindTwist 5:59 pm on Nov 27, 2007 (gmt 0)
newpost thread iconSite title and description missing in search results
  I had made some changes in robots.txt file
2ecmedia 2:12 pm on Nov 27, 2007 (gmt 0)
newpost thread iconperfect robot text files
  good robot text files
2likwidholdings 12:57 am on Nov 27, 2007 (gmt 0)
newpost thread iconGoogle showing site despite blocking in robots.txt
  The page is not cached though
3ecmedia 4:28 pm on Nov 20, 2007 (gmt 0)
newpost thread iconDisallow all except adsense
 
6thegreatpretender 4:40 pm on Nov 19, 2007 (gmt 0)
newpost thread iconHow to avoid double indexing
 
3mavi 4:43 pm on Nov 13, 2007 (gmt 0)
Displaying Page 25 of 71<< Back 1 ... 15 16 17 18 19 20 21 22 23 24 [ 25 ] 26 27 28 29 30 31 32 33 34 ... 71 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved