Robots and private messages on a forum?

Robot searching through private messages send by forum users,

[Silver Dragon Breath]

Yesterday I logged in to the admin panel of my forum and I suddenly saw tons of users on the forum (no, not normal, I have a small forum). I tracked the ip and found it was a japanese software bank coop (couln't translate the page) and meanwhile I noticed that these users all where active on all the parts of my forum, including the private messaging files. The whole process took two minutes. What happened? Can a robot search throug private messages from users on a forum or is this some kind of hacking attempt?

[keyplyr]

Silver_Dragon_Breath - If you are hosted on an Apache server, there are things you can do in defense of unwanted robots and user agents. Use the site search for "htaccess" and read up.

Also threads in the robots.txt forum will be very helpful.

[Zaphod Beeblebrox]

Apart from that, your privacy protection seems to leave a lot to be desired. I would assume those are password protected areas.

[adamas]

Try entering some of the urls they were accessing in your browser and see whether the information is accessible or not.

[RammsteinNicCage]

I run a phpBB board and I think if a robot clicks on the inbox link, even though he obviously cannot logon, it still reads as checking his PMs. I could be wrong about this, I should try it myself one day....

Jennifer

[Silver Dragon Breath]

I run a phpBB forum too, and I think the protection is pretty good (did follow all the safety instructions after installing) so it could be indeed that the robot tries to login but can't. I use htaccess to protect images but I did not know it could also be used to protect other files, for instance pm. I'll check the treads here on this forum out. And for the robot.txt file, I've read about that file several times here on these forums but I just can't find it on my server (it's run on Apache).

[Silver Dragon Breath]

I found this to use as a robot.txt file:

User-agent: googlebot [OR]
User-agent: yandex [OR]
User-agent: slurp [OR]
User-agent: scooter

Disallow: /admin/
Disallow: /banners/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /main/
Disallow: /users/
User-agent: *
Disallow: /

What will it do? I don't understand the user-agent lines, does it allow thowe and does it disallow them to go through the admin, banners etc?
And does adding this to the httpdocs dir prevent privacy invading?

[jatar_k]

take a look at

Robots.txt Tutorial [searchengineworld.com]

also keep in mind that not all bots follow robots.txt. You may need to get into a little htaccess [webmasterworld.com] or bad bot trapping [webmasterworld.com]

[edited by: jatar_k at 8:34 pm (utc) on Nov. 18, 2003]

[keyplyr]

robots.txt tells robots what they are allowed to access. Be careful about using it prior to understanding what you are/aren't allowing.

Here's a good reference: [robotstxt.org...]

Start with creating and uploading a blank text file, named: robots.txt
That will at least serve to fulfill requests for it, and it's rumored that some spiders will not continue their indexing if they do not find it.

[Silver Dragon Breath]

Thanks a lot! I've read through the tutorial and other files and made a big robot.txt file now, validated it, it was ok and now I have it on my server (and I also disallowed that japanese software bank). Thanks for your help, the tutorials and extra's are wonderful.