Software to check competitors pricing

can you spot it in the logs?

[Rugles]

We found out through he grapevine that one of our very large competitors has software that checks various websites for pricing. Then they have the prices automatically adjusted on their site to keep them at the lowest price.

Does anyone have any idea if its off the shelf software? Is there a tell-tale User agent or patterns?

Because I am thinking that it would be really cool to feed them a "special" page to make sure they offer "special" pricing. Therefore reducing their margin and make them lose money on certain product lines.

Is anyone hip to what I am talking about?

[edited by: Rugles at 8:36 pm (utc) on May 9, 2008]

[lorax]

Unlikely to be an off the shelf package. Most likely going to be a custom script or a combination of off the shelf search engine with glue application or modified to suit. Not hard to do. Spot it? Next to impossible by signature. What you should be looking for are 1) short time frames between page requests - unlikely a human is going to request 10 pages in under 20 seconds, 2) odd browser identification strings or "unknown", and 3) patterns of any sort - examine the patterns to see what shape it takes and if it seems to follow natural human behavior or not.

[Morgenhund]

Hi Rugles,

if they are smart there is no way to catch them. It is easy to write a robot that will imitate human's behavioural patterns, and very hard to catch.

I'd recommend you finding out what IP range is assigned to their data center and try to grep logs for these IPs. If you find something, feel free to show different prices to any request originated from these IPs.

Of course, all above will work if they are dumb enough to crawle from same DC they use to host their pages.

[driller41]

Sounds like a fun idea but the chances of bankrupting yourself sounds quite high - enjoy.