client agnostic system, is that a phrase?
client agnostic system, is that a phrase?
Author
Discussion

Du1point8

Original Poster:

22,982 posts

223 months

Tuesday 23rd October 2018
quotequote all
I have a system that is self-sufficient/client agnostic and does not know about modules that can be plugged into it, the system has no module code and the modules use an API to extract data from the main system.

What is the technical phrase for a system that doesnt know about or need to know subsequent 'child' modules?

plug and play is not the one Im thinking of.

sas62

5,929 posts

109 months

Tuesday 23rd October 2018
quotequote all
So without one installed module, the system does nothing?

Perhaps a framework?

Du1point8

Original Poster:

22,982 posts

223 months

Tuesday 23rd October 2018
quotequote all
sas62 said:
So without one installed module, the system does nothing?

Perhaps a framework?
Think of it as a data lake with an API that a client can use... Now think of a module as a specific Audit system that a client wants.

In the data lake we wouldnt provide an audit system as standard as not all clients want it, but when they do we plug the module in and its schema/tables become live in the module environment.

Now say I have an API in both as the client has no direct access to the data, one for DL and one for Audit, I have 2 users, one that can see everything, one that can only see the Audit API and what it sees.

So the DL knows nothing of the Audit and has no reference of the Audit internally and the Audit can reference what it wants, so the DL is....

Thats the terminology Im looking for.


Fas1975

1,808 posts

195 months

Tuesday 23rd October 2018
quotequote all
Sounds like "thin client" to me. A dumb UI that is driven buy the functionality being pushed to it behind the scenes. All the UI is, is a presentation layer of the underlying capabilities.

Du1point8

Original Poster:

22,982 posts

223 months

Tuesday 23rd October 2018
quotequote all
Fas1975 said:
Sounds like "thin client" to me. A dumb UI that is driven buy the functionality being pushed to it behind the scenes. All the UI is, is a presentation layer of the underlying capabilities.
No UI, this is API only to bring back data sets

TallTom

208 posts

190 months

Tuesday 23rd October 2018
quotequote all
So you have data stored in a Data Lake (DL), that has 'Interrogation' capability/front end that is driven through a standard API request.

Secondly, you have audit tools of various flavours/types, that all have an API capabilities built into each that can send API requests to the DL (providing you explain where to send it to)

Once you choose your audit tool that meets client requirements and 'initialise it' it will then send a request to the DL and create its relevant Schema/Tables.

Independent/decoupled system? But that's still not quite right given your use cases.

I have seen Vendor agnostic systems before, but that's not quite right for this either. Unless you could swap the vendor to the audit method and make it a bold statement like 'Automated Malleable Data Lake' or '**Audit X*** Automated Malleable Data Lake' given APIs are often used for some form of systems automation.

My attempt is definitely a starter for 10 there though! (but I see you are trying to combine two systems into one Service and then explain only part of it, in your initial attempt)


Du1point8

Original Poster:

22,982 posts

223 months

Tuesday 23rd October 2018
quotequote all
I think I can go down the separation of concerns or the single responsibility principle.

The DL has a list of API functions:

getName
getAllData
getAddress

Each function has a set of overrides and the data is collected when something calls the API, no overrides, add name, add name and surname, etc.

That is the extent of the DL, nothing more nothing less.

The Modules are on top of the DL and can only use the same 3 API functions (in the example) to get the data they need to function, the schema and tables in the DL has no relevance to the Modules, I could if I want change the whole DL Schema and all the API functions, the module will still work if the data that is extracted from the API function call doesnt change.

So in answer to your query its:

Automated Malleable Data Lake'
'**Audit X***' for AMDL

Note that if necessary '**Audit X***' for AMDL can have its own mini Audit DL that it uses by extracting and doing whatever '**Audit X***' for AMDL needs to do with the data from AMDL into Audit DL. However, Audit DL will never ever post data back to AMDL. '**Audit X***' for AMDL will never have an independent feed into it other than audit specific reference data any raw data is into AMDL first.

Make sense?

zippy3x

1,384 posts

298 months

Tuesday 23rd October 2018
quotequote all
Extensible, client agnostic (as you say), modular, plugin architecture.

Pick two of those?

Du1point8

Original Poster:

22,982 posts

223 months

Tuesday 23rd October 2018
quotequote all
anonymous said:
[redacted]
Cause Devs and users are in general complete morons and I dont want them in the DB.

Also say that in its raw form the API getALLData, gets all the data you ever need, so now I have eliminated why you need access to the DB. Now if I change table A/B/C into a super table ABC and it makes the DB performance increase 1000%... thats awesome.

Except in SQL you called tables A/B/C... so I cant do that upgrade as your code is now coupled to the DL, if however your SQL didnt call the DB, but called the API getALLData, you wouldnt know any difference as Im still supplying the same data, your code still works, but the DB is now working 1000% faster.

Best of all I can upgrade ALL instances of the DL in the world at the same time and everyone can benefit from the same performance increase.

Du1point8

Original Poster:

22,982 posts

223 months

Tuesday 23rd October 2018
quotequote all
anonymous said:
[redacted]
We have too much data that SQL cant cope and whilst we do a grab all data, its more a grab all data with constraints (date related).

In this one currently, Im moving several historical feeds, one being circa 1PB of data into the DL, so you may call it a medium size db, the servers that we use initially are running with 5-7TB RAM for in-memory data ingestion/manipulation.

Current feeds bring in tens of billion rows of data a day and can ingest that in say 60 mins or so, my only issue so far was the feed compression being 50x meaning the in-memory storage did get pushed when bringing in a 50GB binary.

BigTZ4M

237 posts

202 months

Tuesday 23rd October 2018
quotequote all
Sounds to me like you've designed a Service Oriented Architecture

https://en.wikipedia.org/wiki/Service-oriented_arc...

otolith

69,794 posts

235 months

Tuesday 23rd October 2018
quotequote all
Du1point8 said:
Cause Devs and users are in general complete morons and I dont want them in the DB.
I used to work for a company which allowed accountants to run arbitrary SQL queries against the database behind the accounting system.

They took the system down on a fairly regular basis.

zippy3x

1,384 posts

298 months

Tuesday 23rd October 2018
quotequote all
otolith said:
I used to work for a company which allowed accountants to run arbitrary SQL queries against the database behind the accounting system.

They took the system down on a fairly regular basis.
I've seen the Sage database structure, it probably wasn't their fault

wiggy001

7,404 posts

302 months

Tuesday 23rd October 2018
quotequote all
BigTZ4M said:
Sounds to me like you've designed a Service Oriented Architecture

https://en.wikipedia.org/wiki/Service-oriented_arc...
Pretty much what I was going to post until I saw you beat me to it.

Chris Type R

8,985 posts

280 months

Tuesday 23rd October 2018
quotequote all
zippy3x said:
I've seen the Sage database structure, it probably wasn't their fault
Virtual fist pump.

otolith

69,794 posts

235 months

Tuesday 23rd October 2018
quotequote all
zippy3x said:
I've seen the Sage database structure, it probably wasn't their fault
laugh

Wasn't Sage, but did have a very weird database structure. Basically an EAV. I never worked on it, but our DBA despised it passionately.

ging84

9,548 posts

177 months

Tuesday 23rd October 2018
quotequote all
sounds like you've created some sort of middle ware
your data lake is still a data lake

cobra kid

5,628 posts

271 months

Wednesday 24th October 2018
quotequote all
otolith said:
laugh

Wasn't Sage, but did have a very weird database structure. Basically an EAV. I never worked on it, but our DBA despised it passionately.
Ours was essentially an OBG which gradually became an AKJH. This was then an PPR