Beyond Turing

Chapter 10 · Part IV - The Enterprise

Data and Subject

Information systems engineering in the age of algorithmic profiling

Chapter cover: Data and Subject

A contemporary information system - an ERP integrated with analytics modules, an electronic health record connected to machine learning pipelines, a medical wearable transmitting biosignals to a cloud predictive model - produces, at steady state, millions of records per day. Each record is, formally, a datum. Yet each datum, the moment it refers to an identified or identifiable person, constitutes - in the terminology of Article 4(1) of Regulation (EU) 2016/679 - a personal datum: information relating to a subject.

The categorial shift is subtle but decisive. The datum is an engineering artefact - structure, schema, type, constraint. The subject is a legal and ontological category - bearer of rights, dignity, psycho-physical integrity. Contemporary information systems engineering lives entirely within this tension: every pipeline that transforms the first into the second - or the second into the first - is a design choice with measurable technical, legal, and ethical consequences.

This article traces the architecture of that tension from the standpoint of the engineer who designs, validates, and governs information systems with high personal-data intensity. It is not a legal commentary on the GDPR or the AI Act - it is an engineering reading of the non-functional requirements that such regulations impose, and of the reference architectures that follow from them.

§1. The datum as engineering artefact

In the classical Big Data literature - from Laney (2001) onwards - the datum is characterised by the so-called 5V: Volume, Velocity, Variety, Veracity, Value. These are extrinsic properties - they describe the datum as a flow to be governed. Modern data engineering stricto sensu articulates this governance along a canonical pipeline: ingestion (batch or streaming), cleansing and deduplication, transformation (ETL or ELT), storage (data warehouse, data lake, lakehouse), serving (query engine, feature store), analytical or algorithmic consumption.

Each stage of the pipeline has verifiable engineering metrics: end-to-end latency, throughput, error rate, completeness, referential consistency, drift relative to the training distribution. Consolidated disciplines - data quality management (ISO 8000), data governance (DAMA-DMBOK), MLOps - codify best practices to keep these metrics within defined thresholds.

So far the datum is, for all intents, a neutral engineering artefact: a string, a tensor, a row in a relational table, a JSON document. Its nature changes radically the moment it comes into contact with a direct or indirect identifier referable to a natural person. That contact - which in engineering language is a simple join between two tables - is, in legal language, the constitutive event of the processing of personal data (Art. 4(2) GDPR). Two different languages for the same technical operation. And it is precisely this asymmetric symmetry that the engineer must learn to manage.

§2. The subject as legal-technical category

The GDPR introduces the figure of the data subject - the natural person to whom the data refer. It is not a philosophical category: it is a class of engineering requirements. The regulation grants the subject rights (access, rectification, erasure, portability, objection, restriction) that become, in the design of an information system, mandatory features. A CRM non-compliant with the right to erasure is not a CRM defective in an optional feature: it is a system structurally non-deployable in European jurisdiction.

The technically most relevant distinction is that between anonymisation and pseudonymisation (Recital 26 and Art. 4(5) GDPR). The former renders the subject non-identifiable in an irreversible way - and the datum exits the scope of the regulation. The latter replaces direct identifiers with reversible tokens - and the datum remains personal, albeit protected. Confusing the two is a frequent and costly architectural error.

The privacy-preserving computation literature has produced techniques with measurable formal guarantees. k-anonymity (Sweeney, 2002) requires every record to be indistinguishable from at least k-1 others with respect to quasi-identifying attributes. Subsequent refinements - l-diversity, t-closeness - address known attack types. Differential privacy (Dwork, 2006) provides a stronger parametric guarantee: an ε-differentially private mechanism ensures that the presence or absence of a single record influences the system's output within a multiplicative factor of e^ε. The ε parameter is engineering-negotiable - smaller values mean more privacy and less utility - but the guarantee is mathematical, not heuristic.

Alongside these are architectural techniques: federated learning (the model moves to the data, not the data to the model), secure multi-party computation, homomorphic encryption. Each has precise computational costs and trade-offs. The methodological point is sharp: protection of the subject is a quantifiable design requirement, not a vague constraint to be satisfied downstream. It is, in the proper sense, a system constraint - inscribed in the specification from the requirements engineering phase.

§3. Algorithmic profiling and high-risk systems

The introduction of machine learning models into information systems - classifiers, recommenders, clinical decision support systems, credit scoring engines, AI-enabled medical devices - transforms the equation. Added to the processing of data is profiling (Art. 4(4) GDPR) and, potentially, automated decision-making (Art. 22). It is no longer just a question of storing and protecting: it is a question of inferring, predicting, classifying, deciding.

Regulation (EU) 2024/1689 - the so-called AI Act, in force since 1 August 2024 - introduces a risk-based stratification. Article 6 and Annex III identify high-risk systems in areas such as: medical devices and diagnostics (in combined reading with Regulation (EU) 2017/745 - MDR), education and student assessment, personnel selection, credit scoring, law enforcement and migration management applications, critical infrastructure. For such systems the regulation prescribes cogent technical requirements: risk management system (Art. 9), training data governance (Art. 10), technical documentation (Art. 11), event logging (Art. 12), transparency towards the deployer (Art. 13), human oversight (Art. 14), accuracy-robustness-cybersecurity (Art. 15).

The overlap with Regulation 2017/745 on medical devices is particularly dense: a Software as a Medical Device (SaMD) based on ML for, for example, the classification of diagnostic images, falls simultaneously under MDR (CE marking, risk class, clinical evaluation) and under the AI Act (high-risk system pursuant to Art. 6(1)). The system engineer must orchestrate multiple compliance: ISO 13485 for the quality management system, ISO 14971 for risk management, IEC 62304 for the medical device software lifecycle, and - specifically for AI - ISO/IEC 42001 (AI management system, 2023) and ISO/IEC 23894 (AI risk management, 2023).

These are not abstract frameworks. They are - literally - checklists of engineering deliverables: risk assessment documents, data management plans, model cards, system cards, post-market surveillance plans. Each document has a formal structure, an identified reviewer, an update cycle. The AI Act transforms AI engineering from experimental discipline into regulated engineering - with everything this entails in terms of process maturity, traceability, accountability.

§4. Human oversight architectures and explainability

Article 14 of the AI Act - human oversight - is perhaps the most architecturally relevant prescription. It requires that high-risk systems be designed and developed in such a way that they can be effectively overseen by natural persons during their use. Translated into engineering, the prescription distinguishes at least three topologies of human control.

Human-in-the-loop provides for a human operator who validates each decision before execution: it is the most stringent model, applicable where error has irreversible consequences (robot-assisted surgical interventions, decisions on personal freedom). Human-on-the-loop provides for an operator who monitors the system and can intervene - but does not validate every single output: it is the typical model of decision support systems at steady state. Human-in-command provides for human authority to be structural - rules of engagement, deactivation powers, definition of operational perimeters - even when there is no pointwise intervention.

The choice between the three topologies is not purely normative: it is system sizing. It affects admissible latency, sustainable throughput, operational cost, user interface, operator training, interruptibility specifications. A human-in-the-loop system requiring confirmation on 10,000 predictions per minute is a design error. A human-on-the-loop system issuing critical decisions without audit-grade logging is a compliance error.

Human oversight further requires that the model be explainable within the limits imposed by the task. Explainable AI (XAI) - as a research field and as a prescriptive requirement of the AI Act - is no longer optional. Mature techniques (SHAP, LIME, counterfactual explanations, attention visualisation) are now standard components of production ML pipelines. The engineering point is that explainability is not a property of the model, but a property of the system: it depends on the interface, on the target user, on the context of use, on the tolerable cognitive load. An accurate but unreadable explanation is as useless as a wrong one.

Hence the consolidated practice of model cards (Mitchell et al., 2019) and datasheets for datasets (Gebru et al., 2021): structured documents that describe, for each model or dataset, intended use conditions, performance metrics disaggregated by subpopulations, known limitations, residual risks. These are no longer academic exercises - AI Act Art. 13 implicitly invokes them among the mandatory instructions for use.

§5. The irreducible remainder

No oversight architecture, no differential guarantee, no model card exhausts the subject. This is not a limit of the design - it is the design itself. Every model has a coverage gap: a zone of the operational domain where its performance has not been measured or is not characterised with sufficient statistical confidence. Every subject, before a system, is - at least in part - out-of-distribution relative to the training dataset. The singularity of the individual case structurally exceeds the generalisation of the model.

Mature engineering does not try to close this gap: it declares it. It measures it - where possible - with uncertainty quantification techniques (Bayesian calibration, ensemble disagreement, conformal prediction). It flags it to the human operator as a risk zone. It inscribes it in fallback protocols: what does the system do when its confidence level falls below threshold? Return to the human operator, issue a warning, activate an alternative path. These are concrete architectural decisions, not statements of principle.

The methodological point can be formulated as follows: the dignity of the subject is a design invariant. It is not a property to be demonstrated a posteriori, not a constraint to be satisfied in the cheapest possible way, not a bureaucratic requirement. It is a condition on which the system must be built from the first commit - with the same rigour with which one designs high availability, transactional consistency, cryptographic security. In this sense, the GDPR speaks appropriately of privacy by design (Art. 25): subject protection is architecture, not configuration.

An information system designed with this awareness does not renounce analytical power - it makes it locatable. It knows where it operates well, where it operates poorly, where it must stop. It knows that a subject is not the sum of their records, just as a system is not the sum of its services. This engineering awareness of its own boundaries is not a limitation of the technique: it is the point at which the technique reaches its maturity.

The datum can be modelled. The subject can be served. They are not, and never will be, the same thing. The information system that recognises this distinction is not less powerful - it is better designed. And an engineering that does not forget this distinction is not less technical - it is more complete.

↑ Back to contents

by Antonio Fabbrizio · MMXXVI