How to prevent database data theft?
Discussion
Question for webdevs/coder types. I have a SQL database which I've built up with records over the past 20 years and have it available online for people to query the records once they've logged in. I've recently had interest from a company whose business model runs off such a dataset and they've made me a substantial offer to buy my database as is. This has apparently come about because they've been unable to hire staff with the prerequisite knowledge of the data algorithms and also a subpar attention to detail, leading to errors in the data which is losing them business and getting them bad press. They have been monitoring my database for some time and are impressed at the speed of updates which are typically done within 24-48 horus of release, whereas it's more like 4-5 weeks before updates are processed by their own database team.
Whilst their offer is attractive, it's not a life-changing amount and so as the data typically requires anywhere up to a couple of hundred updates per day I see more 'value' in offering an ongoing 'service' instead and keeping the database under my control. They are open to this as well but how could this work in practice? The service they provide produces a live feed of data which in peak periods would be referencing several hundred records from the database per second. It's only about 10 text fields per record, but that strikes me as being quite costly in both bandwidth and computing power vs the current ~500 look-ups per day the database sees at present.
I am fearful that once I allow access to the data for commercial use there'll be nothing to stop them running a bot script to extract every record should they decide to do so. Are there any safeguard methods for database owners to prevent this from happening please?
Whilst their offer is attractive, it's not a life-changing amount and so as the data typically requires anywhere up to a couple of hundred updates per day I see more 'value' in offering an ongoing 'service' instead and keeping the database under my control. They are open to this as well but how could this work in practice? The service they provide produces a live feed of data which in peak periods would be referencing several hundred records from the database per second. It's only about 10 text fields per record, but that strikes me as being quite costly in both bandwidth and computing power vs the current ~500 look-ups per day the database sees at present.
I am fearful that once I allow access to the data for commercial use there'll be nothing to stop them running a bot script to extract every record should they decide to do so. Are there any safeguard methods for database owners to prevent this from happening please?
You could write 'views' to limit what they can see but also licence them to only make specific queries rather than querying all the data in one go. They wouldn't be technically prevented from using the view repeatedly to get all the data but if it was part of the contract then you could charge them accordingly.
You can then audit this so you can satisfy yourself that they're not just dumping the entire table contents and then cancelling the service. This ought to be possible with most database products
You can then audit this so you can satisfy yourself that they're not just dumping the entire table contents and then cancelling the service. This ought to be possible with most database products
MagicalTrevor said:
You could write 'views' to limit what they can see but also licence them to only make specific queries rather than querying all the data in one go. They wouldn't be technically prevented from using the view repeatedly to get all the data but if it was part of the contract then you could charge them accordingly.
You can then audit this so you can satisfy yourself that they're not just dumping the entire table contents and then cancelling the service. This ought to be possible with most database products
With the exception of a single 'notes' column for my own reference, the content of all the other columns need to be displayed in their feed for the service to provide what it needs to provide to attract customers so a limit on which bits of the database they can and can't see wouldn't work in this case unfortunately.You can then audit this so you can satisfy yourself that they're not just dumping the entire table contents and then cancelling the service. This ought to be possible with most database products
Jakg said:
Offer a PAYG service with rate limiting?
They pay say 1p per lookup - and can only do 1 every second.
This is a common problem for services such as online map servers or address lookups.
Interesting idea. Needs more thought They pay say 1p per lookup - and can only do 1 every second.
This is a common problem for services such as online map servers or address lookups.
. The data stream is time critical so would need to process the requests as soon as they are received, ergo wouldn't to able to limit the requests in this case.Scope for an API-based approach? You could issue them with a key which permits both logging of requests/request types and allows you to either charge per-request, or bill them at incremental periods. That could/should offer you rate-limiting if you want, and provides an additional level of security and auditability in data access (IE which token was seen making which requests?)
Depending on what database platform you are running, there might already be solutions out there you could implement.
Depending on what database platform you are running, there might already be solutions out there you could implement.
Given that you run 200 updates a day what would be the point of them stealing the database? Presumably it would quickly be out of date again & hence they would need to come back to you for the updates.
What percentage of the database entries are updated per day? If it has 10,000 records & only 500 of these change then the above might not apply but if all 10,000 change over time then their stolen copy would quickly become stale.
I'd certainly make sure you can track who has made each access request & also pollute the database with delberate trap records containing errors so you know if someone has taken the whole thing. It's the same technique as mapping firms use - they put in fake roads & other features so if they see them on another map they know it's been nicked (the OS sued the AA for £20m using this sort of proof)
What percentage of the database entries are updated per day? If it has 10,000 records & only 500 of these change then the above might not apply but if all 10,000 change over time then their stolen copy would quickly become stale.
I'd certainly make sure you can track who has made each access request & also pollute the database with delberate trap records containing errors so you know if someone has taken the whole thing. It's the same technique as mapping firms use - they put in fake roads & other features so if they see them on another map they know it's been nicked (the OS sued the AA for £20m using this sort of proof)
Lemming Train said:
...they've been unable to hire staff with the prerequisite knowledge of the data algorithms and also a subpar attention to detail, leading to errors in the data which is losing them business and getting them bad press...
...I am fearful that once I allow access to the data for commercial use there'll be nothing to stop them running a bot script to extract every record should they decide to do so. Are there any safeguard methods for database owners to prevent this from happening please?
You're improving their business offering and saving them money (otherwise they wouldn't be willing to pay you). I always figure you should focus on the value you're offering rather than being paranoid that everyone else is out to rip you off....I am fearful that once I allow access to the data for commercial use there'll be nothing to stop them running a bot script to extract every record should they decide to do so. Are there any safeguard methods for database owners to prevent this from happening please?
Failing that log their access and use fictitious entries to detect copyright abuse.
Mr Pointy said:
Given that you run 200 updates a day what would be the point of them stealing the database? Presumably it would quickly be out of date again & hence they would need to come back to you for the updates.
What percentage of the database entries are updated per day? If it has 10,000 records & only 500 of these change then the above might not apply but if all 10,000 change over time then their stolen copy would quickly become stale.
I'd certainly make sure you can track who has made each access request & also pollute the database with delberate trap records containing errors so you know if someone has taken the whole thing. It's the same technique as mapping firms use - they put in fake roads & other features so if they see them on another map they know it's been nicked (the OS sued the AA for £20m using this sort of proof)
200 record updates per day would only be on occasions. Typically it's a few dozen a day. The record count currently stands at approx 1.9m. Although during email discussions they have offered me a job at their HQ (2000 miles away) to research, edit and maintain the database, my spidey senses detect that they really just want my database because theirs is in such a state it's beyond recovery, and then as mine is up-to-date they'll update it themselves as the change updates are available online if you know where to look.What percentage of the database entries are updated per day? If it has 10,000 records & only 500 of these change then the above might not apply but if all 10,000 change over time then their stolen copy would quickly become stale.
I'd certainly make sure you can track who has made each access request & also pollute the database with delberate trap records containing errors so you know if someone has taken the whole thing. It's the same technique as mapping firms use - they put in fake roads & other features so if they see them on another map they know it's been nicked (the OS sued the AA for £20m using this sort of proof)
I'd be able to see from the logs who was accessing it so that's not a problem. I just don't see any way in being able to prevent them from using my data to build their own database, either from scraping the results of the genuine requests and depositing them in a database of their own, or from running a bot script. If I added fake records they would never appear in their feed because the data isn't genuine and so they'd never receive it in the first place for querying. If I added fake data to genuine records then all that would achieve is making me look bad as it would show incorrect data in their feed.
Clearly without more info as to what your data actually is, you won't get the best advice.
Like with photos on the internet, the problem is if you give them access to view the data, it's almost impossible to stop them stealing it if they really want to. Sure, you can put some stuff in place, but it'll be almost impossible to reduce the risk to zero. Not only that, but you also say they are HQ'd 2,000 miles away so likely not in the same country (
) making it even harder to pursue them if they do steal your data.
It sounds to me that your best bet is to sell them the database in it's entirety and be done with it. Make it a one-off transaction and walk away. Not life-changing, but also lowers your risk.
Like with photos on the internet, the problem is if you give them access to view the data, it's almost impossible to stop them stealing it if they really want to. Sure, you can put some stuff in place, but it'll be almost impossible to reduce the risk to zero. Not only that, but you also say they are HQ'd 2,000 miles away so likely not in the same country (
) making it even harder to pursue them if they do steal your data.It sounds to me that your best bet is to sell them the database in it's entirety and be done with it. Make it a one-off transaction and walk away. Not life-changing, but also lowers your risk.
Lemming Train said:
200 record updates per day would only be on occasions. Typically it's a few dozen a day. The record count currently stands at approx 1.9m. Although during email discussions they have offered me a job at their HQ (2000 miles away) to research, edit and maintain the database, my spidey senses detect that they really just want my database because theirs is in such a state it's beyond recovery, and then as mine is up-to-date they'll update it themselves as the change updates are available online if you know where to look.
I'd be able to see from the logs who was accessing it so that's not a problem. I just don't see any way in being able to prevent them from using my data to build their own database, either from scraping the results of the genuine requests and depositing them in a database of their own, or from running a bot script. If I added fake records they would never appear in their feed because the data isn't genuine and so they'd never receive it in the first place for querying. If I added fake data to genuine records then all that would achieve is making me look bad as it would show incorrect data in their feed.
If you sold them a one-off snapshot of your database & they then ran with it & updated it would it trash your ongoing income stream from other clients? If not, then you could sell them a copy but make it clear you are going to continue in business with your version.I'd be able to see from the logs who was accessing it so that's not a problem. I just don't see any way in being able to prevent them from using my data to build their own database, either from scraping the results of the genuine requests and depositing them in a database of their own, or from running a bot script. If I added fake records they would never appear in their feed because the data isn't genuine and so they'd never receive it in the first place for querying. If I added fake data to genuine records then all that would achieve is making me look bad as it would show incorrect data in their feed.
If it would put paid to further income from elsewhere you need to set the price at a correspondingly high level.
It's intriguing to know what the data is though...
If you can't develop API functionality yourself, and it sounds like you can't (no offence) then weigh up the cost of you getting this developed vs their offer.
If only a handful (200 vs 1.9m is not even that) of records get updated then unless you get into the realm of pay-per-query etc then once you give them access they could just automate a download of everything they need.
From the sounds of it if they had the skills they could probably scrape the results into a database anyway. You're probably lucky that they haven't done this already if it's valuable to them.
If it were me I'd be weighing up their offer in the context of how long you can realistically maintain control of this data, how long it might be before someone else scrapes it from you, and how much it would cost you to build in the necessary functionality to prevent scraping, automated queries, etc.
There's no reason you couldn't sell them a snapshot of your data with a contract that restricts its use, forbids it to be resold, offered online etc. Would need a solicitor though.
If only a handful (200 vs 1.9m is not even that) of records get updated then unless you get into the realm of pay-per-query etc then once you give them access they could just automate a download of everything they need.
From the sounds of it if they had the skills they could probably scrape the results into a database anyway. You're probably lucky that they haven't done this already if it's valuable to them.
If it were me I'd be weighing up their offer in the context of how long you can realistically maintain control of this data, how long it might be before someone else scrapes it from you, and how much it would cost you to build in the necessary functionality to prevent scraping, automated queries, etc.
There's no reason you couldn't sell them a snapshot of your data with a contract that restricts its use, forbids it to be resold, offered online etc. Would need a solicitor though.
Durzel said:
If you can't develop API functionality yourself, and it sounds like you can't (no offence) then weigh up the cost of you getting this developed vs their offer.
If only a handful (200 vs 1.9m is not even that) of records get updated then unless you get into the realm of pay-per-query etc then once you give them access they could just automate a download of everything they need.
From the sounds of it if they had the skills they could probably scrape the results into a database anyway. You're probably lucky that they haven't done this already if it's valuable to them.
If it were me I'd be weighing up their offer in the context of how long you can realistically maintain control of this data, how long it might be before someone else scrapes it from you, and how much it would cost you to build in the necessary functionality to prevent scraping, automated queries, etc.
There's no reason you couldn't sell them a snapshot of your data with a contract that restricts its use, forbids it to be resold, offered online etc. Would need a solicitor though.
The database content gets bot attempts on it virtually every week. But it's had a user log-in put in front of it requiring a 'proper' email address for some time, then when that didn't do much to deter them I had it further battened down by adding Captcha and also flood control on the queries. I'm no techie but I'm well aware that none of these are infallible to a determined 'thief' and are fairly easily circumvented with a little know-how. For the most part it is working out okay though, probably because they can't be arsed with the hassle, but casual user queries are not time sensitive so while these hurdles are an annoyance they don't stop people looking up the data for their own personal queries. For a commercial 'service' that would need to query potentially a couple of hundred records per second, obviously Captchas and flood control isn't going to work If only a handful (200 vs 1.9m is not even that) of records get updated then unless you get into the realm of pay-per-query etc then once you give them access they could just automate a download of everything they need.
From the sounds of it if they had the skills they could probably scrape the results into a database anyway. You're probably lucky that they haven't done this already if it's valuable to them.
If it were me I'd be weighing up their offer in the context of how long you can realistically maintain control of this data, how long it might be before someone else scrapes it from you, and how much it would cost you to build in the necessary functionality to prevent scraping, automated queries, etc.
There's no reason you couldn't sell them a snapshot of your data with a contract that restricts its use, forbids it to be resold, offered online etc. Would need a solicitor though.
.I would be slightly more open to letting them have direct access to the database (either via an API 'push' or 'pull' arrangement, or just send them it like they want) if I could spin it so that they would have to also take a monthly sub for the updates with a long minimum term to ensure I make it worth my while financially. Or, if they only want the database then they'll need to up their offer to a life-changing amount so that I'll no longer care!
Gassing Station | Computers, Gadgets & Stuff | Top of Page | What's New | My Stuff