Malicious hits to your site, and other questions?

damage control

[skyshock21]

Since I'm new to the whole world of website management... I'm curious as to what these "bots" are capable of doing when they crawl your site. Are there any that are particularly malicious? I've heard of Denial of Service attacks, is this what people are trying to avoid with the robots.txt file?

I've noticed several large sites such as CNN and others have extensive robots.txt files which restrict certain bots access to their site. However, many large sites don't have these files.

My site doesn't have secure areas at the moment that I'm worried about. It's simply an informative page promoting a band I play in.

I guess my question is, should I be worried that I don't have a robots.txt file?

Also, I put the site up in Mid-August. Should it be showing up on search engines yet?

[edited by: heini at 9:44 pm (utc) on Sep. 26, 2003]
[edit reason] please don't use urls, thanks. [/edit]

[nancyb]

Welcome to WebmasterWorld [webmasterworld.com] skyshock21

You might also want to use the site search for "robots.txt" and "bad bots".

There is a wealth of information here and if you still have questions after digging into some of these threads, please don't hesitate to ask more specific questions.

(not having a robots.txt isn't necessarily bad, depends on if you want to keep some bots out, but then if they are bad bots, they won't bother with robots.txt anyway )