Welcome to WebmasterWorld Guest from 54.166.224.46

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
can you have 2 robots.txt files on site?
one in main and one in subdomain
8 stiksandstones 12:10 am Aug 19, 2005
What bots ignore robots.txt?
8 Jon12345 10:56 pm Aug 18, 2005
Validator Questions
Need Help Deciphering Validator Output
7 Jioffri 6:07 am Aug 18, 2005
Quick question on robots.txt
Can you separate the agent and the allows/disallows?
2 cmendla 11:19 pm Aug 17, 2005
Google's Nonstandard robots.txt is Not Valid
Googles robots.txt Does not validate
6 forak 1:59 pm Aug 17, 2005
Search engines picking up our backend?
Developement site picked up
3 financialhost 12:46 pm Aug 11, 2005
Still no PR for site
3 gnuman 8:39 pm Aug 10, 2005
Can excluding with one word also exclude phrases?
3 Marcia 10:47 pm Aug 7, 2005
New algo about Robots.txt
Is there any new algo introduced regarding robots.txt?
4 Graham_Hans 2:31 am Aug 7, 2005
Dmoz robots.txt
Is the dmoz.org/world disallowed?
4 angiolo 8:05 am Aug 6, 2005
How to exclude Oracle Portal Help pages using robots.txt
Need to exclude Oracle Portal Help Pages from search engines
10 atul_iiit 9:12 am Aug 5, 2005
http://<hostname.domainname>/robots.txt is access denied can robots
Can robots read robots.txt if I get access denied for this file
4 atul_iiit 4:25 pm Aug 4, 2005
What's Robot.txt?
What's the importance of Robot.txt?
3 Lowkei 9:02 am Jul 31, 2005
Don't want to index dynamic pages
But would it be a mistake to block them?
5 chopin2256 2:12 am Jul 30, 2005
Could 744 permissions on robots.txt cause problems
all other files are set to 644
5 dbar 7:41 pm Jul 29, 2005
Robots.txt viewable
viewing the robots.txt file
2 eleap 3:27 pm Jul 27, 2005
How to exclude a few pages?
2 jack38 10:25 am Jul 27, 2005
Blocking a file extension with robots.txt
Inherited a site with a problem
5 abbeyvet 5:45 pm Jul 26, 2005
How to avoid getting print-pages indexed?
What's the best way for Google?
1 simczak 3:42 pm Jul 24, 2005
Robots.txt stops automated link software
2 jdhuk 11:01 pm Jul 18, 2005
Which editor?
Notepad or editor in UNIX
3 Bali_Freak 12:15 pm Jul 18, 2005
selectively including folders
that are beneath a folder excluded
2 stu2 5:20 am Jul 17, 2005
Would this work? Robots exclusion or considered cloaking?
5 dvldvl3 5:11 am Jul 17, 2005
Previously Indexed Files-Want To Exclude Now
RobotsTxt Change
3 buview1 12:23 pm Jul 16, 2005
Google only hits robots and index
Google only hits robots and index
10 rostom 3:51 pm Jul 15, 2005

Featured Threads

Hot Threads This Week

Hot Threads This Month