| 8:12 pm on Feb 25, 2004 (gmt 0)|
robots.txt is usually used for restricting robots from indexing parts of a site.
| 12:50 pm on Mar 15, 2004 (gmt 0)|
If you have an META tag instruction for robots, do you need a robots.txt?
| 12:55 pm on Mar 15, 2004 (gmt 0)|
>>If you have an META tag instruction for robots, do you need a robots.txt?
Most SE's will/should obey a robots.txt file while there is a lesser amount will read the META tag.
Am open for correction on this, but I think that is what i have came to understand.
| 5:38 pm on Mar 16, 2004 (gmt 0)|
|If you have an META tag instruction for robots, do you need a robots.txt? |
At the very least, you should put an empty robots.txt to cut down on 404s on your site.
Using META tags allows you to control how robots index or follow links on your pages.
Using robots.txt restricts access to files (text and binary), as well as directories.
| 5:46 pm on Mar 16, 2004 (gmt 0)|
I'm probably opening up a can of worms here, but here we go.
| 8:52 pm on Mar 16, 2004 (gmt 0)|
To tell you the truth, I don't think it will be an issue. It just isn't something the average user (aka politician) understands or even knows about. Heck, I think a survey of websites will show that 50% of webmasters don't know about them.
If nobody knows, nobody will make a law about it.