Profiling and Automated Decision-Making (Article 22 GDPR)

#10.1 Preliminary Remarks

We live in an information society. Information (including to a large extent that concerning natural persons) is becoming more valuable than material goods (with value assessed not only quantitatively but also qualitatively). For this reason, it has long been accepted that what counts is not collecting the greatest number of data possible, but selecting them appropriately and having them available for use. To this end, profiling is conducted.

Put simply, profiling is a process of collecting unrelated data and extracting patterns and correlations from them using mathematical and statistical models. In the case of the profiling of personal data, the goal of the profiling is to place an individual the data concern in a specific category and to predict or evaluate:

1. 1)capabilities of a person,
2. 2)his or her interests,
3. 3)how he or she is likely to behave.

From a processing perspective, profiling is divided into three stages:

1. 1)text and data mining,
2. 2)profile building, and
3. 3)profile analysis in terms of decision-making in individual matters.

Because the creation of a profile requires the processing of very large amounts of data, the process involves the use of computer algorithms and new technologies that make it possible to both collect more data and build profiles effectively. During automated profiling, a human or an algorithm makes a specific decision on the basis of a profile prepared by another algorithm. The development of automated profiling is greatly affected by the development of techniques of big data analytics, machine learning, and of the Internet of Things, which provide different data on the people for whom the profile is being created.

In practice, profiling is used: to create better-tailored advertising (behavioural marketing); in the financial and banking industry where, on the basis of credit card data, bank account use and debt registers a credit rating is generated; in the insurance industry where, on the basis of data collected on insurance risk, total risk and optimal insurance premiums are calculated; in the transport and services industry where, on the basis of data on consumption and browsing history, dynamic tariff systems are constructed; and to analyse the movements of individuals in order to evaluate their purchasing habits, manner of using their car or the popularity of particular business locations, broken down by individual visitors (and their characteristics, such as age and wealth). For several years now, precise data obtained by profiling have been used, for example, to prepare election campaign promises better suited to voter preferences leading to the Cambridge Analytica scandal.

In the public sector profiling is used: to detect terrorism by analysing information on to and from where and how often a given person travels; to detect financial abuse by monitoring transactions and building account owner profiles; to identify cases of human trafficking; and to suit job offers to the unemployed.

Even though profiling makes it possible to enhance the security and effectiveness of an organisation, as well as to reduce costs, its automated nature also makes it very controversial. Alarms have gone out that automated profiling may constitute a universal threat to the rights and freedoms of individuals. The main reservations expressed about profiling include:

1. 1)a lack of transparency in the algorithms used to create profiles and to build relations between a profile and, e.g., a service offered;
2. 2)potential breaches of privacy (by collecting data without the data subject’s knowledge);
3. 3)a lack of understanding about what may result from profiling;
4. 4)the perpetuation of existing stereotypes and social segregation;
5. 5)the possibility of a person being permanently assigned to a specific category and having specific preferences attributed to him or her that may constitute a source of discrimination.

As pointed out by the Chairwoman of the American Federal Trade Commission, Edith Ramirez, with automated and autonomous profiling, ‘Individuals may be judged not because of what they’ve done, or what they will do in the future, but because inferences or correlations drawn by algorithms suggest they may behave in ways that make them poor credit or insurance risks, unsuitable candidates for employment or admission to schools or other institutions, or unlikely to carry out certain functions.’ Even more serious doubts are raised by automated decision-making systems. In the case of classic automated profile, one may have doubts as to how a profile was created, yet we can assume that even if certain important circumstances were omitted by the system (because of how the algorithm was built), these could still be taken into account when a human being makes a final decision. In totally automated systems based on algorithms, machine learning and artificial intelligence, decisions are not subject to any human control, meaning all we can do is hope that all important circumstances were taken into account by the system (and, as everyone knows, there is no perfect system).

In the face of such a threat, the EC decided to comprehensively regulate through the GDPR:

1. 1)when we can profile,
2. 2)what obligations we bear in connection with profiling,
3. 3)what the rights of profiled people are, and
4. 4)in what cases particularly sensitive automated decision-making systems can be applied, including systems that make decisions on the basis of earlier profiling.

Just how important it was for the EC to regulate profiling issues in detail is shown by the fact that the concept of profiling appears in the GDPR, in different contexts, more than twenty times.

#10.2 Definition of Profiling in the GDPR

For the purposes of the GDPR, profiling is defined as:

‘any form of automated processing of personal data consisting of the use of personal data to evaluate certain personal aspects relating to a natural person, in particular to analyse or predict aspects concerning that natural person’s performance at work, economic situation, health, personal preferences, interests, reliability, behaviour, location or movements’ (Art. 4(4) GDPR).

As emphasised, in order to qualify as profiling in the meaning of the GDPR, it is key that:

1. 1)the processing (profiling) is automated (not entirely performed by a human),
2. 2)the processing (profiling) is performed on personal data,
3. 3)the result of the processing (profiling) is an evaluation of personal aspects (traits) of a natural person for given needs of the processor.

Other forms of data processing (non-automated or performed on data other than personal data or involving the use of personal data for a purpose other than evaluating personal aspects) do not constitute profiling under the GDPR, while in certain conditions they may qualify as automated decision-making.

It is also important to consider Recital 75 GDPR, pursuant to which:

‘where personal aspects are evaluated, particularly if analyses or forecasts are made of aspects concerning job performance, economic condition, health, personal preferences or interests, reliability, behaviour, location or travel – in order to create or use personal profiles’, – a risk may arise of individual rights and freedoms being violated. The passage of the GDPR evidently concerns profiling, and there is no word on the subject of automation.

#10.2.1 Automation

For the needs of profiling under the GDPR, the concept of ‘automation’ should be understood as the use of computer algorithms or systems for profiling. Automation may take place at the stage of:

1. 1)text and data mining,
2. 2)profile building,
3. 3)profile analysis in terms of decision-making in individual matters.

In contrast to Article 22 GDPR, the definition of profiling does not require profiling to be completely automated; that is, it does not rule out the presence of the human element in the profiling process.

Profiling may be as simple as segregating a database in Excel according to the criteria contained in one column.

#10.2.2 Personal Data

Collecting data on a group of natural persons without the possibility of extracting individuals from the group is not profiling in the meaning of the GDPR. Nor is collecting data on a group of customers in shopping centres without extracting the movement of particular natural person profiling or preparing energy tariffs for particular user groups on the basis of power consumption in particular hours without verifying the power consumption of individual persons.

#10.2.3 Impact

For processing to be profiling, the impact of the processing must be in the form of an assessment of certain personal factors. As the provision indicates, this involves, in particular, analyses and prognoses (drawing conclusions about the future) regarding aspects of a person’s behaviour. Put simply, this means that personal data are used to assess or predict a given person’s behaviour and assign him or her to some kind of category (e.g., bad debtors with a zero credit rating, or persons who, encouraged by a dynamic tariff dependent on energy consumption, will remain loyal customers of a power company).

#10.2.4 Activities Treated as Profiling

Table 2.1 shows examples of profiling versus corresponding actions that are not profiling.

#Table 2.1 Examples of Profiling Based on Personal Data

Source: own study.

#10.3 Obligations of a Personal Data Controller in Connection with Profiling, Automated Decision-Making, or Decisions Made on the Basis of Profiling

GDPR foresees several important obligations related to three situations: profiling, automated decision-making, and making decisions on the basis of profiling:

1. 1)notification of an automated decision based on profiling – both decisionmaking based on profiling and automated decision-making require informing the data subject of how the process works and what it means for that person. Information should be provided when the data are collected, within a month from obtaining data indirectly (from other sources than the data subject) and when exercising a request for data access (Article 13(2)(f), Article 14(2)(g), Article 15(1)(h) GDPR). If profiling is made through artificial intelligence or more accurately machine learning mechanisms, it may be difficult to identify and apply deadlines for providing such information to a data subject;
2. 2)the right to human intervention – Article 22 GDPR introduces the right not to be subjected to a decision based solely on automated processing, including profiling, which produces legal or other similar effects. However, such a decision can be made solely automatically or solely on the basis of profiling if consent is expressly granted as necessary to the conclusion or performance of an agreement; in such a situation, though, there must be a complaint procedure in place that involves a competent, authorised person;
3. 3)impact assessment – Article 35 GDPR requires an assessment to be made of the impact on personal data if systematic, comprehensive profiling is conducted in order to make decisions that cause legal or similar effects.

#10.3.1 General Obligations of a Controller

#10.3.1.1 Notification of Automated Decision-Making and Decision-Making Based on Profiling

As we have shown, both making decisions based on profiling and automated decisionmaking require notification to the data subject of how the process works and what it means for that person. Notice should be sent when the data are collected and when exercising a request for data access (Article 13(2)(f), Article 14(2)(g), Article 15(1)(h) GDPR).

#10.3.1.2 How Often to Notify

Given that profiling is often conducted on the basis of many different data sources, the obligation to notify when profiling will often have to be fulfilled in two ways: once at the moment data are collected for profiling from the data subject the profiling concerns (at the moment data are obtained or when collection begins, e.g. at the commencement of using the service in which the data are used for profiling) and a second time if the profiling is based on ‘external’ data not obtained from the person being profiled, within a month from when the data are obtained.

#10.3.1.3 A Change of Purpose of Processing

Personal data controllers should remember that profiling is a purpose that differs from that of merely collecting and analysing data. If a data subject is not informed that his or her data will be profiled at the time when the data are collected, then that person must be promptly notified at the moment the profiling begins.

#10.3.2 The Right to Object

The only provision of the GDPR that refers to every form of profiling – regardless of what the impact of the profiling is to be (whether a decision based thereon is to be issued by a person or automatically) – is that concerning the right to object.

The right to object applicable in case profiling is conducted based on public interest task or official authority (Article 6(1)(e) GDPR) or legitimate interest of a controller or a third party (Article 6(1)(f) GDPR) encompasses:

1. 1)the right to demand that profiling be stopped for reasons related to the data subject’s particular situation (more on the conditions for submitting an objection and on a particular situation in the commentary to Article 21 GDPR); as we have indicated, good examples of a particular situation are when we do not fit a pattern (i.e., when the algorithm has made a mistake), when the profiling infringes our personal rights, or when it preys on our addictions;
2. 2)the unconditional right to demand that profiling be stopped if it is being conducted for the purposes of direct marketing (to the extent profiling is connected with such direct marketing).

#10.4 Profiling and Automated Decision-Making

As per Recital 71 GDPR, GDPR protects in particular persons who are subjected to automated decisions (e.g., electronic credit applications, electronic methods of recruiting without human intervention). This relates to the concerns discussed above over potential breaches of individual rights and freedoms, lack of transparency of profiling and automated decision-making as well as lack of accuracy and bias. The Regulation itself does not define directly what is to be understood by an automated decision. We can assume that it means all decisions made without human participation (as indicated by the use in Article 22 GDPR of the phrase ‘based solely on automated processing’). Introducing an apparent human decision in the decision-making process (which is limited, e.g., to a single click on an ‘accept’ button) does not alter the automated nature of such a decision. For a decision to qualify as non-automated, it must be made by a human who is competent and professionally authorised to do so.

The Regulation restricts automated decisions, including profiling, that have legal or similar effects on the persons they concern. The concept of having legal effects would seem to be clear. Interpreting it in accordance with the GDPR recitals, it means any impact whatsoever on the legal situation of the person the decision concerns (inc. granting some kind of entitlement, granting or blocking access, concluding or terminating an agreement). Doubts arise, however, when analysing the phrase ‘similarly significantly affects him or her’. The wording and sense of the provision indicate that this is a category of effects of decisions that, in fact, do not change the legal status of the individual (they do not affect his or her rights or obligations), yet which are significant enough to alter (mainly adversely) that person’s situation (e.g., by depriving the person of access to goods or services of a given type). Examples of a significant impact on a person that does not cause any legal effects are: an automatic refusal to conclude a loan agreement (e.g., because of a customer’s credit rating), or a customer being presented with an automated offer for services (e.g., to buy airline tickets) that, due to the algorithm employed, is at a level that makes it impossible to buy the ticket (e.g., in the case of a sudden blizzard to results in many flights being cancelled where the ticket price depends on the number of available flights and seats in planes). In commentaries, and especially in the Article 29 Working Party Guidelines, the example is also used of people who are particularly susceptible to the content of advertisements that urge them to purchase given goods or services (e.g., deliberately addressing online roulette ads to persons profiled as very susceptible to gambling read: addicted or children).

Automated decisions that do not involve profiling may be based on:

1. 1)data collected directly from data subjects (provided, e.g., by filling out a questionnaire);
2. 2)data collected from observations of a given person (e.g., data on travel recorded in an application or by a camera, monitoring location, monitoring physical activity);
3. 3)data resulting from a previously created profile (e.g., a credit rating). Examples of automated decisions that do not involve profiling are:
   1. 1)a bank system imposing a debt collection charge on a customer who is in arrears with loan repayment instalments;
   2. 2)a customer being sent an automated declaration of termination of an agreement due to the occurrence of circumstances set out in the agreement (e.g., a failure to pay bills);
   3. 3)discount coupons being granted through a phone app for shops in a shopping centre the customer will pass by, where the customer’s travels are analysed by an algorithm in a visual surveillance or geolocation system based on telephone tracking;
   4. 4)ticket from a speed camera.

#10.4.1 Security Measures

In accordance with the GDPR, a controller that uses systems for issuing automated decisions is obliged to employ:

1. 1)technical and organisational measures that ensure that irregularities that occur when issuing decisions are corrected and that the risk of errors is reduced as much as possible;
2. 2)measures for securing personal data in a way that takes account of risks to the interests and rights of the data subject and that prevents, among other potential problems, the effects of discrimination against natural persons due to their racial or ethnic origin, political, religious views or convictions, trade union membership, state of health or genetic condition, or sexual orientation.

Given the serious penalties imposed for infringements of the GDPR, until binding guidelines on the provisions guaranteeing security and avoiding discrimination in data processing are issued, theoretically it is important to be diligent in striving to eliminate even unlikely errors or cases of discrimination when issuing automated decisions. When running tests, a data controller of a processor may think of engaging a group of persons to provoke system errors and effects of discrimination. From this perspective, the complaints mechanism with ‘human intervention’ required by Article 22(3) GDPR should be treated as an extremely important feedback channel for optimising an automated process.

In practice automated profiling and decision-making is present for a long time in various sectors, and in particular in the financial and insurance sector. Bank and insurers are often criticised over lack of transparency of their decision-making processes. Fintechs rely heavily on profiling and extensive access to personal data. GDPR, if taken seriously, should enforce much more clarity as to why such entities take specific decisions and allow question such decisions more effectively.

#10.4.2 When Automated Decisions or Decisions Based Solely on Profiling Can Be Used?

Here we must cite the provision itself in order to clarify another editing inconsistency of the GDPR.

Pursuant to Article 22(1) GDPR:

The data subject shall have the right not to be subject to a decision based solely on [1] automated processing, including [2] profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.

Dissecting this sentence logically, we obtain two types of decisions: (i) decisions based solely on automated processing and (ii) decisions based solely on profiling. In the second case, a decision need not be automated at all, since profiling has automation built into it by definition. It is enough if the procedure obligates a person to comply with the results of profiling, or in practice there are no exceptions from the results of profiling.

Regulation of automated decision-making, including profiling, starts with the general prohibition but foresees three exceptions to it. The right to automate a decision-making limited to:

1. 1)‘need for a contract’, when automated decision-making is necessary in order to conclude or perform an agreement between the data subject and the controller;
2. 2)‘law’, when a provision of law can be applied under EU law or the domestic law to which the controller is subject, and which foresees due measures for protecting the rights, freedoms, and legitimate interests of the data subject; and
3. 3)‘express consent’ of the person subject to automated decision.

The criterion of usefulness for a contract should be understood in a reasonably narrow manner taking into consideration the principle of minimisation. Thus, data useful for issuing an automated decision only include those data that are adequate, relevant, and necessary.

In turn, the criterion of a legal basis should be understood as a clear basis expressed in a provision of EU law (a regulation) or a domestic provision of an EU Member State. Such automated processing can, for example, happen when an obliged institution or a Member State authority processes financial data for the purpose of preventing money laundering and financing terrorism.

The criterion of consent requires as usual that consent is given completely freely and without coercion. In respect of profiling, all cases of economic encouragements (bonuses, discounts) to grant consent should be examined carefully, as pricing model may be structured in such a way that not granting consent results in exorbitant prices of a particular service, etc. Then freedom of consent is a fiction.

#10.4.2.1 Right to Human Intervention

Where the issuance of an automated decision is based on the criterion of usefulness for an agreement, or on that of consent, the controller must implement measures for protecting the data subject’s rights and freedoms. In particular, the controller must ensure that, in addition to the right to notification, the data subjects also have:

1. 1)the right to obtain human intervention during the decision-making process,
2. 2)the right to express their own views,
3. 3)the right to obtain an explanation regarding the decision resulting from the automatic evaluation,
4. 4)the right to question such a decision.

#10.4.2.2 Complaints Procedure

In practice, the right to human intervention may be exercised through a complaints procedure.

A channel of communication with human participation should be provided. The person with whom the data subject communicates should respond in such a way that it is evident that he or she has become acquainted with and pre-analysed the data subject’s position and is able to explain his or her evaluation.

Control mechanisms should be in place for confirming that human verification is being conducted reliably and with real decision-making power. The less overruled automated decision, the harder to prove that the human intervention is real. One should anticipate that the supervisory authority will ask data relating to complaints.

#10.4.2.3 Automated Verification

A particular case of falling victim to an automated decision is when a data subject loses their digital identity, that is when such a person can no longer log in to their profile in a given service or application. Since the GDPR requires data controllers to provide a complaints procedure with human participation, this should also be the case within the scope of verification. Various complications arise, of course, but when designing verification methods in systems, applications and services, according to the rules of privacy by design one should remember to ensure there is a human channel for recovering identity, or at least a means of verifying system decisions. At the same time, verification cannot be based on verbal methods of communication (pursuant to Article 12 GDPR), so a compromise must be sought. Though data protection authorities usually do not like when data controllers copy identification documents of people (data subjects) this is a usual method of recovering digital identity in case of loss of credentials (e.g., login & password). In anti-money laundering procedures copying customers and beneficial owners IDs is a basic financial security measure, so it is not so rare.

#10.4.3 Profiling of Children

Children should not be covered by automated decision-making (Recital 71 in fine GDPR). As the commentaries emphasise, children are much more susceptible to adults to behavioural advertising and to having their profiles created on the internet. They are not able to understand all the consequences of their personal data being processed. And so they should be given special protection. According to the GDPR, ‘children’ means anyone under 16 years of age, although GDPR allows Member States to reduce that age limit down to 13 years.

In practice, though, it will not be easy to ban the automated processing of children’s data. To date, popular internet services have ‘managed pretty well’ with children by formally prohibiting persons below a certain age from using them. It remains to be seen whether this practice of forbidding and then turning a blind eye will continue to be approved.

#10.4.4 Automated Decisions Concerning Sensitive Data

Apart from the regulations discussed above, the GDPR severely restricts automated decision-making based on sensitive data. Pursuant to the GDPR, such data may be processed solely:

1. 1)on the basis of the express consent of the data subject, or
2. 2)if the processing is related to the performance of an important public interest on the basis of EU or domestic law.

In both cases, the data controller must guarantee that proper measures exist for protecting the rights, freedoms, and legitimate interests of the data subject.

Taken the above, unless a specific provision of law exists, a data controller cannot enter into a valid contract with a data subject requiring automated decisionmaking based on special categories of data of the counterparty. In order to deal with the requirements for consent to be valid, a data controller should allow for a contractual process not requiring automated decision-making so that such automated decisionmaking becomes optional. Otherwise, a consent will not be freely given.

#10.4.5 Conclusions

If you profile or use profiling – you should give notice of this.

If you base decisions on profiling – you should provide a human complaints procedure.

If you profile comprehensively – you should conduct a data protection impact assessment.

When creating a profile – you should consider whether you need to know the identity of the data subjects. Usually, this is not necessary.