Page is a not externally linkable
- Local
-- Foo
---- Attack of the Robots, Spiders, Crawlers.etc


2by4 - 7:53 pm on Nov 29, 2005 (gmt 0)


while brett may or may not be interested in the advice in this thread, I definitely am, there's some really solid stuff in here, have to bookmark it and come back to some of these posters at a later date, if it's still an interesting problem that is.

If I was running a site this big I wouldn't be using redhat, and I wouldn't be doing the server admin myself, I'd be talking to a hardcore server freak who can make this stuff dance around his little finger, and it would probably be running on 64 bit systems.

"There are also issues of scrapping, copyright, and liability"

Such as? Interesting, but the disclaimer on the bottom of the page says posters own their postings. All forums have this problem, and all big ones have a big version of it I assume. There's nothing unique about webmasterworld except its size, and how good its software is.


Thread source:: http://www.webmasterworld.com/foo/9618.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com