Welcome to WebmasterWorld Guest from 54.167.0.111

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
robots.txt use and hosting provider problem
robots.txt use and hosting provider problem
2 david_cool 6:55 pm Dec 16, 2003
robots.txt on phpbb forum
2 RammsteinNicCage 4:06 pm Dec 12, 2003
Spiders get / and /robots.txt but no more!
Is my robots.txt killing me?
6 dwilson 8:55 pm Dec 11, 2003
[404] HTTP Page robots.txt not found 404 error
4 Strike 12:44 pm Dec 10, 2003
changing robots.txt
replacing robots.txt
2 darex 8:24 pm Dec 9, 2003
Can we prevent a form link from being spidered?
form href prevent spider indexing
3 Frannie 2:32 pm Dec 9, 2003
Robots and directory hierachies
4 finlay 5:57 pm Dec 8, 2003
wildcard disallow for pages with varibles
would this work for pages ending with?keep= : Disallow: *keep=*?
2 The_Rabbit 5:25 pm Dec 8, 2003
disallowing based on part of a query string
3 jibbajabba 4:39 pm Dec 2, 2003
Whois.sc Probe/Bot
Prevent Snapshot Cache
1 CustomFit 3:41 am Dec 2, 2003
Hiding robots.txt
21 keyplyr 1:45 am Dec 2, 2003
Did you re-test?
3 hybridx 2:51 pm Nov 28, 2003
Weird Code Return from Googlebot when robots.txt was spidered
3 5x54u 9:14 pm Nov 25, 2003
Problem with robots.txt
4 Nisse 8:08 am Nov 25, 2003
robots and dynamic url
How to ban robots from a dynamic url
3 Ledfish 8:09 pm Nov 23, 2003
Googlebot Wildcard
5 payday 11:26 am Nov 22, 2003
Importance of Hotbot
5 slobizman 9:08 am Nov 20, 2003
How long until robots.txt works?
11 slobizman 3:23 pm Nov 18, 2003
Unknown robots
crawl, spider, and robot.
2 synergy 4:10 am Nov 16, 2003
Would crawlers crawl this?
Usiing the correct robots.txt technique
4 ulysee 3:02 pm Nov 15, 2003
internetseer
billing for requested data
14 plumsauce 7:47 pm Nov 14, 2003
Pros and cons of having
a robots.txt file?
3 ideavirus 5:32 pm Nov 13, 2003
Does anyone know what this is?
I am thinking virus infected machine but unsure
3 Crowbar 4:28 pm Nov 12, 2003
Blocking .log file with robots.txt
Is it necessary, and syntax...?
7 Robert_Charlton 6:59 am Nov 11, 2003
How to make a page *inaccessible* to searches
without using password protection?
12 halfandhalf 8:04 pm Nov 10, 2003

Featured Threads

Hot Threads This Week

Hot Threads This Month