Welcome to WebmasterWorld Guest from 188.8.131.52
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Getting 403 Forbidden for robots.txt
Can anyone help
|11||GodLikeLotus||4:59 am Dec 18, 2003|
|robots.txt use and hosting provider problem
robots.txt use and hosting provider problem
|2||david_cool||6:55 pm Dec 16, 2003|
|robots.txt on phpbb forum
||2||RammsteinNicCage||4:06 pm Dec 12, 2003|
|Spiders get / and /robots.txt but no more!
Is my robots.txt killing me?
|6||dwilson||8:55 pm Dec 11, 2003|
| HTTP Page robots.txt not found 404 error
||4||Strike||12:44 pm Dec 10, 2003|
|2||darex||8:24 pm Dec 9, 2003|
|Can we prevent a form link from being spidered?
form href prevent spider indexing
|3||Frannie||2:32 pm Dec 9, 2003|
|Robots and directory hierachies
||4||finlay||5:57 pm Dec 8, 2003|
|wildcard disallow for pages with varibles
would this work for pages ending with?keep= : Disallow: *keep=*?
|2||The_Rabbit||5:25 pm Dec 8, 2003|
|disallowing based on part of a query string
||3||jibbajabba||4:39 pm Dec 2, 2003|
Prevent Snapshot Cache
|1||CustomFit||3:41 am Dec 2, 2003|
||21||keyplyr||1:45 am Dec 2, 2003|
|Did you re-test?
||3||hybridx||2:51 pm Nov 28, 2003|
|Weird Code Return from Googlebot when robots.txt was spidered
||3||5x54u||9:14 pm Nov 25, 2003|
|Problem with robots.txt
||4||Nisse||8:08 am Nov 25, 2003|
|robots and dynamic url
How to ban robots from a dynamic url
|3||Ledfish||8:09 pm Nov 23, 2003|
||5||payday||11:26 am Nov 22, 2003|
|Importance of Hotbot
||5||slobizman||9:08 am Nov 20, 2003|
|How long until robots.txt works?
||11||slobizman||3:23 pm Nov 18, 2003|
crawl, spider, and robot.
|2||synergy||4:10 am Nov 16, 2003|
|Would crawlers crawl this?
Usiing the correct robots.txt technique
|4||ulysee||3:02 pm Nov 15, 2003|
billing for requested data
|14||plumsauce||7:47 pm Nov 14, 2003|
|Pros and cons of having
a robots.txt file?
|3||ideavirus||5:32 pm Nov 13, 2003|
|Does anyone know what this is?
I am thinking virus infected machine but unsure
|3||Crowbar||4:28 pm Nov 12, 2003|
|Blocking .log file with robots.txt
Is it necessary, and syntax...?
|7||Robert_Charlton||6:59 am Nov 11, 2003|