Welcome to WebmasterWorld Guest from 54.144.7.239

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
robots.txt verses sitemap.html verses OPML file
2 Neal 8:39 pm Jul 11, 2005
Just added robots.txt file
4 soccrt 4:41 am Jul 11, 2005
block all except
2 Radial 9:31 pm Jul 7, 2005
robots.txt with? in URL
16 jozomannen 11:17 am Jul 6, 2005
How do i add robot.txt?
10 genny 4:38 am Jul 6, 2005
Include files, spiders, and robots.txt
4 rottsnroses 11:24 pm Jul 3, 2005
How to Stop BecomeBot?
12 shilmy 3:09 pm Jun 29, 2005
Google ignoring my robots.txt?
9 homesby 12:17 am Jun 29, 2005
Robots meta tag *CORRECT* syntax
Space* or no space for "index,*follow", and case sensitive?
6 Clint 12:56 pm Jun 28, 2005
Question about Robots with multiple similar sites
Question about Robots with multiple similar sites
2 ccash 9:54 pm Jun 27, 2005
Robots/Slurp, etc - what does this mean
I am a newbie - not sure what this means
4 ramnyc 3:01 pm Jun 22, 2005
Does adding robot text help your site get crawled more often?
Advantages of robot text?
6 moneyangel13 8:04 am Jun 22, 2005
wildcards in page names
Disallow: /store/scripts/emailFriend.asp*
5 jimsthoughts 3:54 pm Jun 21, 2005
How can I allow robots to access my pages
inktomi is following links it should not
4 expat123 10:08 am Jun 21, 2005
Updating robots.txt and Crawlers behaviour
How to tell bots to come back when you changed robots.txt banning them
6 silverbytes 6:57 pm Jun 20, 2005
slurp ignoring disallow?
-or have i got it wrong?
4 tallis 12:09 pm Jun 20, 2005
Question about blocking 1 robot, but allow all others
8 capsmaster 3:26 pm Jun 18, 2005
Frequencey of GET /robots.txt
How often should a crawler/spider/robot ask for robots.txt?
9 Dijkgraaf 5:08 am Jun 18, 2005
Need Clarification On Directory Exclusions
10 jonrichd 5:00 am Jun 18, 2005
Robots.txt and Mod Rewrite
Which url does the robot read?
2 eflouret 4:52 am Jun 18, 2005
how to ban Google from indexing ALL .php files
I'm using rewrite and don't want a dupe penalty
16 walkman 5:12 pm Jun 15, 2005
boitho.com bot violating robots.txt
Specifically requested only forbidden files[4] ( 1 2 3 4 )
111 jazzguy 8:30 pm Jun 14, 2005
robots.txt and Google on intranet
robots.txt not working
4 John488 8:11 pm Jun 13, 2005
Specific Spiders do need more bandwidth?
Leaving Gbot or Msn does other robots take more bandwidth when crawling?
2 bnvrc 12:39 pm Jun 13, 2005
robots.text ..please help
help with robots.text file
2 dmiller63 12:35 pm Jun 13, 2005

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members