client agnostic system, is that a phrase?
Discussion
I have a system that is self-sufficient/client agnostic and does not know about modules that can be plugged into it, the system has no module code and the modules use an API to extract data from the main system.
What is the technical phrase for a system that doesnt know about or need to know subsequent 'child' modules?
plug and play is not the one Im thinking of.
What is the technical phrase for a system that doesnt know about or need to know subsequent 'child' modules?
plug and play is not the one Im thinking of.
sas62 said:
So without one installed module, the system does nothing?
Perhaps a framework?
Think of it as a data lake with an API that a client can use... Now think of a module as a specific Audit system that a client wants.Perhaps a framework?
In the data lake we wouldnt provide an audit system as standard as not all clients want it, but when they do we plug the module in and its schema/tables become live in the module environment.
Now say I have an API in both as the client has no direct access to the data, one for DL and one for Audit, I have 2 users, one that can see everything, one that can only see the Audit API and what it sees.
So the DL knows nothing of the Audit and has no reference of the Audit internally and the Audit can reference what it wants, so the DL is....
Thats the terminology Im looking for.
So you have data stored in a Data Lake (DL), that has 'Interrogation' capability/front end that is driven through a standard API request.
Secondly, you have audit tools of various flavours/types, that all have an API capabilities built into each that can send API requests to the DL (providing you explain where to send it to)
Once you choose your audit tool that meets client requirements and 'initialise it' it will then send a request to the DL and create its relevant Schema/Tables.
Independent/decoupled system? But that's still not quite right given your use cases.
I have seen Vendor agnostic systems before, but that's not quite right for this either. Unless you could swap the vendor to the audit method and make it a bold statement like 'Automated Malleable Data Lake' or '**Audit X*** Automated Malleable Data Lake' given APIs are often used for some form of systems automation.
My attempt is definitely a starter for 10 there though! (but I see you are trying to combine two systems into one Service and then explain only part of it, in your initial attempt)
Secondly, you have audit tools of various flavours/types, that all have an API capabilities built into each that can send API requests to the DL (providing you explain where to send it to)
Once you choose your audit tool that meets client requirements and 'initialise it' it will then send a request to the DL and create its relevant Schema/Tables.
Independent/decoupled system? But that's still not quite right given your use cases.
I have seen Vendor agnostic systems before, but that's not quite right for this either. Unless you could swap the vendor to the audit method and make it a bold statement like 'Automated Malleable Data Lake' or '**Audit X*** Automated Malleable Data Lake' given APIs are often used for some form of systems automation.
My attempt is definitely a starter for 10 there though! (but I see you are trying to combine two systems into one Service and then explain only part of it, in your initial attempt)
I think I can go down the separation of concerns or the single responsibility principle.
The DL has a list of API functions:
getName
getAllData
getAddress
Each function has a set of overrides and the data is collected when something calls the API, no overrides, add name, add name and surname, etc.
That is the extent of the DL, nothing more nothing less.
The Modules are on top of the DL and can only use the same 3 API functions (in the example) to get the data they need to function, the schema and tables in the DL has no relevance to the Modules, I could if I want change the whole DL Schema and all the API functions, the module will still work if the data that is extracted from the API function call doesnt change.
So in answer to your query its:
Automated Malleable Data Lake'
'**Audit X***' for AMDL
Note that if necessary '**Audit X***' for AMDL can have its own mini Audit DL that it uses by extracting and doing whatever '**Audit X***' for AMDL needs to do with the data from AMDL into Audit DL. However, Audit DL will never ever post data back to AMDL. '**Audit X***' for AMDL will never have an independent feed into it other than audit specific reference data any raw data is into AMDL first.
Make sense?
The DL has a list of API functions:
getName
getAllData
getAddress
Each function has a set of overrides and the data is collected when something calls the API, no overrides, add name, add name and surname, etc.
That is the extent of the DL, nothing more nothing less.
The Modules are on top of the DL and can only use the same 3 API functions (in the example) to get the data they need to function, the schema and tables in the DL has no relevance to the Modules, I could if I want change the whole DL Schema and all the API functions, the module will still work if the data that is extracted from the API function call doesnt change.
So in answer to your query its:
Automated Malleable Data Lake'
'**Audit X***' for AMDL
Note that if necessary '**Audit X***' for AMDL can have its own mini Audit DL that it uses by extracting and doing whatever '**Audit X***' for AMDL needs to do with the data from AMDL into Audit DL. However, Audit DL will never ever post data back to AMDL. '**Audit X***' for AMDL will never have an independent feed into it other than audit specific reference data any raw data is into AMDL first.
Make sense?
anonymous said:
[redacted]
Cause Devs and users are in general complete morons and I dont want them in the DB.Also say that in its raw form the API getALLData, gets all the data you ever need, so now I have eliminated why you need access to the DB. Now if I change table A/B/C into a super table ABC and it makes the DB performance increase 1000%... thats awesome.
Except in SQL you called tables A/B/C... so I cant do that upgrade as your code is now coupled to the DL, if however your SQL didnt call the DB, but called the API getALLData, you wouldnt know any difference as Im still supplying the same data, your code still works, but the DB is now working 1000% faster.
Best of all I can upgrade ALL instances of the DL in the world at the same time and everyone can benefit from the same performance increase.
anonymous said:
[redacted]
We have too much data that SQL cant cope and whilst we do a grab all data, its more a grab all data with constraints (date related).In this one currently, Im moving several historical feeds, one being circa 1PB of data into the DL, so you may call it a medium size db, the servers that we use initially are running with 5-7TB RAM for in-memory data ingestion/manipulation.
Current feeds bring in tens of billion rows of data a day and can ingest that in say 60 mins or so, my only issue so far was the feed compression being 50x meaning the in-memory storage did get pushed when bringing in a 50GB binary.
Sounds to me like you've designed a Service Oriented Architecture
https://en.wikipedia.org/wiki/Service-oriented_arc...
https://en.wikipedia.org/wiki/Service-oriented_arc...
Du1point8 said:
Cause Devs and users are in general complete morons and I dont want them in the DB.
I used to work for a company which allowed accountants to run arbitrary SQL queries against the database behind the accounting system.They took the system down on a fairly regular basis.
BigTZ4M said:
Sounds to me like you've designed a Service Oriented Architecture
https://en.wikipedia.org/wiki/Service-oriented_arc...
Pretty much what I was going to post until I saw you beat me to it.https://en.wikipedia.org/wiki/Service-oriented_arc...
Gassing Station | Computers, Gadgets & Stuff | Top of Page | What's New | My Stuff



