homepage Welcome to WebmasterWorld Guest from 54.211.157.103
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 44 message thread spans 2 pages: 44 ( [1] 2 > >     
apache pb2.gif
why do bots ask for this
dupres01




msg:4692299
 5:26 pm on Aug 1, 2014 (gmt 0)

After some scum bots eat a bunch of 403's, they then ask for:

GET /icons/apache_pb2.gif HTTP/1.1

Which gets a 200.
Why do they do this and is there a way (or reason) to prevent it?

 

incrediBILL




msg:4692323
 5:56 pm on Aug 1, 2014 (gmt 0)

Wow.

I had to try that myself and it works like a charm on any Apache site. The question is whether or not they're targeting Apache vs IIS, or a specific version of Apache.

Removing the image from your server would be the obvious simple fix.

I'd open a discussion about this in the Apache server and see if the gurus have any ideas as this is nasty IMO.

wilderness




msg:4692328
 6:19 pm on Aug 1, 2014 (gmt 0)

Simply add an empty index.html file to the icons folder (and others if you please).

I'm on shared hosting, and some while back had bots grabbing server images above my root folder. Generally speaking the hosts do not allow editing of files above root folder, however an addition of an empty index file was a success for me.

incrediBILL




msg:4692347
 7:22 pm on Aug 1, 2014 (gmt 0)

You could have to add an icons folder too, correct?

I still can't believe I've messed with apache all this time and never noticed you could access files above your folder. That's a fundamental flaw IMO and now they're using it.

lucy24




msg:4692350
 7:34 pm on Aug 1, 2014 (gmt 0)

I had to try that myself and it works like a charm on any Apache site.

Not mine. (You know where I live. Feel free to try.) I get the expected 404. Maybe it depends on physical file structure? My host uses the "userspace" setup, where all domains are parallel, rather than the "primary/addon" structure.

not2easy




msg:4692387
 10:00 pm on Aug 1, 2014 (gmt 0)

It does not work on my sites either. (whew!)

Angonasec




msg:4692407
 11:41 pm on Aug 1, 2014 (gmt 0)

Gulp! It works on my shared hosting site.

I'll point them to this thread.

Angonasec




msg:4692408
 11:56 pm on Aug 1, 2014 (gmt 0)

Quick test:
Neither putting an empty index.html file, nor creating an icons folder with an empty index.html file prevented the default Apache image being displayed at my shared host.

lucy24




msg:4692418
 2:39 am on Aug 2, 2014 (gmt 0)

Follow-up: It DOES, however, work on MAMP, which nobody but me has ever touched-- and where there is clearly and unambiguously no /icons/ directory. That means I can pore over the config file in search of enlightenment-- but not right now :(

phranque




msg:4692419
 4:46 am on Aug 2, 2014 (gmt 0)

look for an Alias or AliasMatch directive in the server config file.

e.g.
Alias /icons/ /usr/local/apache/icons/


mod_alias:
http://httpd.apache.org/docs/current/mod/mod_alias.html [httpd.apache.org]

lucy24




msg:4692485
 5:06 pm on Aug 2, 2014 (gmt 0)

I just opened the MAMP directory and searched for /icons. Luckily Spotlight is OK with the leading slash (it ignores some punctuation).

# We include the /icons/ alias for FancyIndexed directory listings. If you
# do not use FancyIndexing, you may comment this out.

Alias /favicon.ico "/Applications/MAMP/bin/favicon.ico"

Alias /icons/ "/Applications/MAMP/Library/icons/"

<Directory "/Applications/MAMP/Library/icons">
Options Indexes MultiViews
AllowOverride None
Order allow,deny
Allow from all
</Directory>

If the FancyIndexing option is given with the IndexOptions directive, the column headers are links that control the order of the display.


The Alias directive itself can't be used in htaccess (other mod_alias directives of course can) so you can't change the setting if you're on shared hosting. But it seems like the kind of thing any halfway decent host would change on request.

The existence of this alias means that if you create a directory called /icons/, containing files of your own, in your ordinary filespace, it would not be recognized. (On my host, the same presumably applies to /stats/ since analog stats are stored in a different physical location, but accessed via example.com/stats/.)

You could comment-out the favicon alias to let your MAMP-or-equivalent site display your actual favicon from its default location (root directory). But to me it makes more sense to use theirs, as it's an instant way to tell whether I'm in MAMP or my live site.

Angonasec




msg:4692600
 9:12 am on Aug 3, 2014 (gmt 0)

My host has confirmed that the server config file Alias over-rides attempts by shared sites to block this via a local icons folder or blank index file.

As Lucy stated.

So what can such shared sites do to block this access method?

keyplyr




msg:4692605
 9:45 am on Aug 3, 2014 (gmt 0)



RewriteRule icons - [F]

Angonasec




msg:4692622
 10:10 am on Aug 3, 2014 (gmt 0)

I tried that local Rule, but still the gif displays.
Server config over-rides even that!

wilderness




msg:4692629
 10:27 am on Aug 3, 2014 (gmt 0)

My host has confirmed that the server config file Alias over-rides attempts by shared sites to block this via a local icons folder or blank index file.

As Lucy stated.

So what can such shared sites do to block this access method?


Angonasec,
I'm spread pretty thin these days and don't have much spare time.

Don't recall if it is/was my previous El Cheapo host or the current host where I was required to make this correction.

Simultaneously, I realized the host had custom 403s & 404s in place to advertise their hosting capabilities, thus I was required to make changes via CP to create my own custom Error Docs and eliminate their advertising.

The creation of my own Error Docs stopped the display of the hosts images, however did not stop crawl access to the previous paths, and required the blank index pages.

In the event your host does not provide a solution, than changing hosts might be a requirement.

Don

Angonasec




msg:4692633
 11:14 am on Aug 3, 2014 (gmt 0)

Thanx Don, I appreciate your input, but don't let me deflect you.

Actually, my own 403 is custom (nil bytes) and over-ridden, as is the blank index file method.

Changing hosts would be last resort, because they are well respected, and hopefully reading this thread :)

dstiles




msg:4692705
 8:45 pm on Aug 3, 2014 (gmt 0)

As an apache novice this took me a while to work out, but I finally nailed it. I know this isn't available to all apache users but if you have your own server...

Background: My own apache server under linux mint running a single web site for squirrelmail.

In the file /etc/apache2/mods-available/alias.conf (edited as Administrator), change "Allow from all" to "Deny from all". (This may be in a different place in other than mint.)

========
Alias /icons/ "/usr/share/apache2/icons/"

<Directory "/usr/share/apache2/icons">
Options FollowSymlinks
AllowOverride None
Order allow,deny
Deny from all
</Directory>
========

Remember to restart apache.

I assume an alternative would be to comment out the Alias line and the subsequent icons code but this was the first thing I tried and it worked.

lucy24




msg:4692748
 10:32 pm on Aug 3, 2014 (gmt 0)

RewriteRule icons - [F]

Sorry, no. Your htaccess file only affects requests that physically pass through it. Since the /icons/ directory is in a completely different location, requests for /icons/ will never see your htaccess file.

The same thing happened with that horrible, horrible robot from a few months ago that was assailing my /stats/ directory. Since the directory is aliased to an entirely different part of the server, my htaccess lockouts had no effect on it. The same thing also happened a few years ago when I had one directory aliased to my son's userspace (so he could upload game files directly). My htaccess had no effect on unwanted robots asking for files in this directory.

if you have your own server

... then you can comment-out anything you like :( or simply don't install the /icons/ directory at all.

Angonasec




msg:4692771
 12:53 am on Aug 4, 2014 (gmt 0)

And for the rest of us?

My shared host has given only cloth-eared responses to this thread, unable to perceive any cause for concern whatsoever.

Can you enlighten them please?

lucy24




msg:4692776
 1:35 am on Aug 4, 2014 (gmt 0)

:: pause to read back over this thread ::

Is there a cause for concern? It's hard to imagine what your unwanted Ukrainian visitors would do with pb2.gif-- or, for that matter, any of the other files living in the /icons/ directory. (MAMP has 78 pairs of gif/png duplicates, plus one animated gif, plus 32 more pairs in a /small/ subdirectory.)

I guess in theory the exact content of the icon tells the visitor what approximate Apache version your site is running (there's also an apache_pb.gif that doesn't include a number) but that's pretty far-fetched since there are more reliable ways of getting the same information in greater detail. Besides, any physical invasion is happening in some remote backwater of the server. It doesn't bring them any closer to your site.

I'd look more closely at what else your Ukrainians are asking for. Make sure they get their proper 403 whenever they set foot across your actual threshold.

dupres01




msg:4692779
 2:34 am on Aug 4, 2014 (gmt 0)

That was part of my original question (is there a reason to prevent it). As for why they do it, my only guess is that they want to know that they can access some part of the server- but I have no idea why. Then again, I am not well educated in such things, hence I ask.

Angonasec




msg:4692784
 3:17 am on Aug 4, 2014 (gmt 0)

I see, so 403'd hackers/nasty bots accessing your site via urls such as;

example.com/icons/apache_pb2.gif

...Receiving a 200, plus the server OS version, is perfectly safe, and a waste of their neurone power.

(As my host concludes.)

Or is that naive?

keyplyr




msg:4692808
 6:29 am on Aug 4, 2014 (gmt 0)

Sorry, no. Your htaccess file only affects requests that physically pass through it. Since the /icons/ directory is in a completely different location, requests for /icons/ will never see your htaccess file.

Sorry, no. I don't have a /icons/ directory in a completely different location. All requests pass through my local config.

lucy24




msg:4692809
 6:51 am on Aug 4, 2014 (gmt 0)

This thread is about the /icons/ directory that IS in a different location, following Apache defaults. If you have no such directory, and hence no such Alias directives, then this thread is not for you.

keyplyr




msg:4692810
 7:02 am on Aug 4, 2014 (gmt 0)



This thread is about the /icons/ directory that IS in a different location, following Apache defaults. If you have no such directory, and hence no such Alias directives, then this thread is not for you.



Not mine. (You know where I live. Feel free to try.) I get the expected 404. Maybe it depends on physical file structure? My host uses the "userspace" setup, where all domains are parallel, rather than the "primary/addon" structure.


LOL

dupres01




msg:4692811
 7:06 am on Aug 4, 2014 (gmt 0)

the directory structure under the var directory is:
cgi-bin
error
html
icons

all of which looks pretty standard to me. it is that icons directory that is being accessed. I do not have an icons directory under the html directory.

keyplyr




msg:4692827
 8:37 am on Aug 4, 2014 (gmt 0)


the directory structure under the var directory is:
cgi-bin
error
html
icons

For those of you that feel requests to files withing this /icons/ directory may be a threat, can you not just delete that directory with FTP (example: FileZilla?) Seems to me I've done this before.

phranque




msg:4692829
 8:50 am on Aug 4, 2014 (gmt 0)

For those of you that feel requests to files withing this /icons/ directory may be a threat, can you not just delete that directory


in shared hosting situations, most webmasters would not have ftp access to the /var subdirectories.

dupres01




msg:4692830
 8:50 am on Aug 4, 2014 (gmt 0)

sorry, left out the www part; the first line should be:

the directory structure under the var/www directory is:

and no, at least on the server i use, i can not delete the icons directory.

Angonasec




msg:4692841
 10:07 am on Aug 4, 2014 (gmt 0)

The server directives are fixed by the shared-hosting provider; cannot be changed by a tenant site, and over-ride all attempts to alter the behaviour using local htaccess.

Odds are many of you reading this are hosted this way too.

This 44 message thread spans 2 pages: 44 ( [1] 2 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved