agodar

[Mokita]

A new bot that lobbed in and took the home page without checking robots.txt.

It came from an Italian IP, and Google shows they have three web sites, .com, .net and .biz but all are "under construction" and reveal no info about what they intend to do with the data they are collecting.

403s are their diet from now on.

[horsefern]

I am a babe in the wood when it comes to spiders and bots.

My site is currently being nibbled on by agodar and it is highly unnerving to see the hit numbers at 296 for March, down from 20,573 and change in February. And Firefox and IE are locked out! What the #*$!x?

I need help and quickly! Any suggestions as to how I can regain control of my site would be welcome.

I will send you the 'stats' link if it will help.

Thanks for reading my post.

Horsefern

[edited by: encyclo at 10:35 pm (utc) on Mar. 17, 2008]
[edit reason] no personal URLs please [/edit]

[jdMorgan]

The robots.txt file has nothing to do with "locking out" Firefox or Internet Explorer. These are browsers, which do not read robots.txt unless you type that in as the URL you wish to be displayed on your screen, and they certainly do not interpret or act on its contents.

It sounds like either your host has made a configuration change that broke your site, or perhaps your site has been hacked. In either case, you may wish to re-evaluate the hosting company.

You might also want to post in our Webmaster General forum [webmasterworld.com], and tell us as much as you can about your site (except for mentioning the domain name, which is not allowed here for your benefit and for ours). Instead, we all host our Web sites at "www.example.com" and we all sell "widgets" here. :)

When you post in Webmaster General, answering basic questions like these will help:

Have you contacted your host's support department?

If so, what did they say? (Please paraphrase, no e-mail quotes permitted here.)

Is your site static, or do you use script (or scripts to) generate pages?

Do you have a blog, or is the whole site (just) a blog?

If so, is it WordPress? Other?

Are you hosted on an Apache server or a Microsoft IIS server?

Shared, virtual private server, or dedicated server?

Jim