File Not Found Errors

Only happens with certain User-Agent Strings

[dmje]

I have been having a problem with what I call search bots not finding a file on my website, its very hard to explain just what is happening so bear with me:

I have a java script menu on my website that has code for the menu on every page of my site, lately I have been getting a lot of errors, file does not exist, for the following file:

99.246.**.*** File does not exist: /home/best/public_html/"+awmMenuPath+awmLibraryPath+"/awmlib"+scriptNo+".js

the line above is part of the menu code and works fine as far as I have been able to determine. What I have noticed is that every client that throws the error has the following line in my lastest vistors part of my CPanel

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)

I can look in the error log and find the page it says is in error and I can go to it and everything loads just fine, it is only those that have the line above that show the error. Other vistors can load the same page and it will show the line of code as working fine whereas, the others will return a 404.

I am getting lots of these errors and dont have a solution...can anyone advise as to what to do if anything?

[Samizdata]

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)

[webmasterworld.com...]

...

[dmje]

I read most of that thread and other threads off shooting from it, Am I to understand that there is nothing to worry about and nothing I can do about it?

Bandwidth for me is not an issue, I just dont like to see all those errors in my Cpanel error log or my log files, and I hope that I am not losing vistors in any way.

I did have AVG 8 installed without the toolbar, but didnt like the linkscanner so I uninstalled it and went with something else.

As you can see from below, I have had over 1700 hits all by the same user agent requesting that js file

/\"+awmMenuPath+awmLibraryPath+\"/awmlib\"+scriptNo+\".js 1006

/metal/\"+awmMenuPath+awmLibraryPath+\"/awmlib\"+scriptNo+\".js 616

/cham/\"+awmMenuPath+awmLibraryPath+\"/awmlib\"+scriptNo+\".js 72

/cart/\"+awmMenuPath+awmLibraryPath+\"/awmlib\"+scriptNo+\".js 50

/%22+awmMenuPath+awmLibraryPath+%22/awmlib%22+scriptNo+%22.js 21

So do I just not worry about it or what? None of the threads I have read offered a solution or really said anything about what to do about it.

[Samizdata]

I have had over 1700 hits all by the same user agent requesting that js file

That is because AVG LinkScanner is a useless product - it is not your fault.

Am I to understand that there is nothing to worry about and nothing I can do about it?

I appreciate that there have been many threads about AVG LinkScanner in the past three months.

In those threads you will find various solutions for the various problems it causes - my own preference is to serve LinkScanner a very small dummy file, but exactly what you do and how you do it will depend to some extent on your server setup.

You might also try contacting AVG's head of communications Pat Bitton, who has requested feedback from webmasters (though noticeably not on WebmasterWorld) about the problems that LinkScanner causes.

I cannot post his email address here but he gives it in the comments to the original story in The Register (link on the thread I mentioned), and I am sure he would be keen to hear from you.

...

[dmje]

How would I go about serving up the dummy file? That sounds like a good idea to me

[Samizdata]

It will depend on your server. Apache example here:

[webmasterworld.com...]

Be aware that the people who make LinkScanner introduced another fake user-agent at the weekend and are in a complete mess over this, so things may change.

These are the conditions currently required to detect this piece of garbage.

RewriteCond %{HTTP_USER_AGENT} ;1813\)$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^User

The dummy file must not be empty. A link to your homepage is enough.

No human visitors will be affected.

...

[dmje]

I am sorry but I do not see an example at the link given. I am somewhat familiar with htaccess but I would still need to see some kind of example.

Thanks

[Samizdata]

Apologies, the link works for me but your settings may be different.

The example given was:

RewriteCond %{HTTP_USER_AGENT} ;1813\)$
RewriteRule !^a-very-small-page\.html$ /a-very-small-page.html [L]

I would generally not advise copy/pasting .htaccess you do not understand.

Please see the WebmasterWorld Apache forum for further .htaccess information.

For more information on LinkScanner please see:
[webmasterworld.com...]

If you want to fully understand this problem you will have to do a lot of reading.

...

[dmje]

Am I to understand that linkscanner has trouble reading java script? Is that why it cannot find the file mentioned above?

I am just trying to find an answer as to why this file continues to show up as non existant in my error logs when I know that it is there and works properly.

Linkscanner or whatever it is causing tons of 404 errors on my site by not finding this file

[g1smd]

May be related: [webmasterworld.com...]

[dmje]

I am up to 1944 404's for the following files so far this month..

/\"+awmMenuPath+awmLibraryPath+\"/awmlib\"+scriptNo+\".js
/metal/\"+awmMenuPath+awmLibraryPath+\"/awmlib\"+scriptNo+\".js
/cham/\"+awmMenuPath+awmLibraryPath+\"/awmlib\"+scriptNo+\".js
/cart/\"+awmMenuPath+awmLibraryPath+\"/awmlib\"+scriptNo+\".js

All these files exist and work properly except when the 1813 and now the others call for them.