The social and ethical challenges of AI and big data by Richard Benjamins

Autonomous cars use AI to learn how to drive. Once such cars are a practical reality, and on the market, they will transform the mobility ecosystem with many positive impacts. These conceivably include reduced accidents and casualties, emptier city streets and less pollution. But the same technology that enables autonomous cars also makes possible lethal autonomous weapons systems (LAWS), popularly called ‘killer robots.’ Through that lens, one has to ask if AI is a blessing or a curse. A specific use of deep learning, called Generative Adversarial Networks (Wikipedia, n.d.) can be used to bring back long-dead movie actors (Diamandis, 2019) or create art featuring imaginary models (Christie’s, 2018). However, it also enables the creation of ‘deepfakes,’1 invented videos that depict people saying things they never said, driving the proliferation of fake news. Blessing or curse?

Deep learning has also radically improved the task of perception, both in speech and image recognition. Google Duplex2 is able to hold a human-like dialogue to make an appointment with a hairdresser. But, that same deep learning technology applied on video also enables massive surveillance of populations in China using facial recognition, social credit scoring (Wikipedia, n.d.) and public shaming3 of people who cross a street while the traffic light is red. Blessing or curse?

Apart from the technical capability AI is providing, its popularity is mostly due to the many positive applications that are improving our lives, in areas like medical diagnosis, automatic translation, content recommendations, business optimization, chatbots, medicine discovery and predictive maintenance, to name just a few. Yet, that same technology, in the hands of the wrong people, can also create significant harm (Brundage, 2018), in particular related to digital, physical and political security.

So far, in this book, we’ve shared how large organizations can enjoy the enormous positive impact of AI and big data on businesses, to improve processes, reduce costs and increase revenues. However, there are also negative side effects of using this technology at large scale. We’ve all heard about black box algorithms, unfair discrimination and privacy breaches. Harvard data scientist Cathy O’Neil’s book, Weapons of Math Destruction (O’Neil, 2016), gives many examples of opaque AI decision systems with significant impact on people’s lives. Amazon had to withdraw an AI system (Dastin, 2018) that treated women unfairly com- pared to men during the company’s HR selection process. When it was introduced, the Apple Card was criticized for giving women less favourable loan conditions than men in comparable situations (Vigdor, 2019). And then there’s the Cambridge Analytica debacle, where the personal information of millions of Facebook users was co-opted for political advertising (Wikipedia, n.d.). Most of these ‘scandals’ are not the consequence of bad intentions, but of using new technology in large business applications without giving sufficient attention to all potential risks.

From these examples, and others, we can extract a number of challenges related to the use of AI and big data.

 

ETHICAL AND SOCIAL CHALLENGES OF THE USE OF AI AND BIG DATA

Much has already been written about the possible ethical and social implications associated with the use of AI and big data. Several of these implications are described and analysed in The Myth of the Algorithm: Tales and Truths of Artificial Intelligence (Benjamins and Salazar: 2020).

 

BIAS AND UNFAIR DISCRIMINATION (SENSITIVE ATTRIBUTES)

While machine learning is able to solve complex tasks with high performance, it might use information that’s undesirable from a societal or human rights perspective. For example, deciding whether to provide a loan based on race or religion is forbidden. While it’s possible to remove these unwanted attributes from data sets, there are other, less obvious attributes that might correlate with these attributes — so-called proxy variables. A well-known example is the attribute ‘postal code,’ which might have a significant correlation with race, and in the AI model could result in discrimination. Machine learning finds whatever pattern there is in the data, regardless of specific norms and values.

Another important aspect for avoiding discrimination is whether the data set is representative of the target group with respect to variables related to protected groups. If, for example, an AI system that helps with hiring people is trained with CVs from the IT sector, you should not use that system for hiring all kinds of candidates, because in the IT sector there are significantly more men than women. As a consequence, the system could discriminate against women in its recommendations.

Apart from bias in the training data leading to possible discrimination, this can also come from the algorithm.

A machine-learning algorithm tries to be as accurate as possible when fitting the model to the training data. At the same time, all machine-learning algorithms make mistakes, providing false positives and false negatives. If the proportion of these errors is not equal for variables considered sensitive in the application (race, gender, etc.) it could result in discrimination and have a negative impact on the affected individuals. Accuracy is often defined in terms of false positives and false negatives, often through a so-called confusion matrix. But the definition of this ‘accuracy’ measure, whether it tries to optimize only false positives or only false negatives, or both, can have an impact on the outcome of the algorithm, and therefore on the groups of people affected by the AI program. In safety-critical domains, such as health, justice and transportation, defining ‘accuracy’ is not a technical decision, but a domain or even a political decision.

 

BLACK BOX ALGORITHM OR ‘EXPLAINABILITY’

Deep learning algorithms can be highly successful, but people have a hard time understanding why they have come to a certain conclusion. As described earlier, they are proverbial ‘black boxes.’ For some applications, this explainability is an essential part of the decision itself, and lack of that makes the decision unacceptable. For example, a ‘robo-judge’ deciding on a dispute between a customer and a health insurer is unacceptable without the explanation of the decision. This is sometimes also referred to as the ‘interpretability’ problem. The book mentioned previously, Weapons of Math Destruction (O’Neil, 2016), gives many interesting examples of this.

 

DATA PRIVACY AND SECURITY

Big data and machine-learning systems exploit data, and many times this is personal data. As a side effect of using all this personal data, privacy might be compromised, even if unintentionally. The Cambridge Analytica scandal shows that this is a bigger issue than we might have thought (Wikipedia, n.d.).

 

AUTONOMOUS DECISIONS AND LIABILITY

When systems become autonomous and self-learning, accountability for their behaviour and actions becomes less obvious. In the pre-AI world, the user was accountable for incorrect usage of a device, while device failure was the manufacturer’s responsibility. When systems become autonomous, and learn over time without human intervention, some behaviours will not have been foreseen by the manufacturer. It therefore becomes less clear who would be liable when something goes wrong. A clear example of this is driverless cars. Who is responsible if something goes wrong: the manufacturer, the car itself, the auto dealer or the owner? There’s ongoing discussion about whether liability should be with the producer or the deployer (Committee on Legal Affairs, European Commission, 2020).

 

THE FUTURE OF WORK

AI can take over many boring, repetitive or dangerous tasks. But, if this happens on a massive scale, many jobs might disappear, and unemployment would skyrocket. Most experts and policymakers agree that, as with any techno- logical revolution, jobs will be lost, new jobs will be created, and the nature of many jobs will change. Nobody knows to what extent those changes will happen, and what percentage of workers simply won’t be able to make the shift to the required digital skills. In the event that fewer people are needed to maintain productivity, fewer and fewer people will work. Governments will then collect less income tax, while costs of social benefits will increase due to increased unemployment. How can this be made sustainable? Should there be a ‘robot tax’? How will governments be able to pay pensions when fewer people work? Is there a need for a universal basic income (UBI) for everybody? How will the unemployed survive if AI takes many of the current jobs, and what will be their purpose in life?

 

DATA AND WEALTH CONCENTRATION

AI and big data are currently dominated by a few large digital companies, including the ‘GAFA’ giants we talked about earlier and some Chinese mega-companies (Baidu, Alibaba, Tencent). This might lead to significant concentration of power and wealth in a few very large companies. This is mostly due to them having access to massive amounts of propriety data, which could lead to an oligopoly. Apart from the lack of competition, there is a danger that these companies keep AI as proprietary knowledge, not sharing anything with the larger society other than for the highest price possible. Another concern is that these companies could offer high-quality AI as a service, based on their data and propriety algorithms (the black box conundrum). When these AI services are used for public services, the opacity issue — no information on bias, undesirable attributes, performance, etc. — raises serious concerns. We saw this when the Los Angeles Police Department announced that it was using Amazon’s ‘Rekognition’ face-recognition solution for policing (Brandom, 2018).

 

THE PEOPLE-MACHINES RELATIONSHIP

How should people relate to robots and machines? If robots become more autonomous and learn during their ‘lifetime,’ what sort of relationship should be allowed between robots and people? Could one’s boss be a robot, or an AI system? In Asia, robots are already taking care of elderly people, offering companionship and stimulation. Could people get married to robots? One of the key aspects of such systems will be safety and security.

 

MALICIOUS USES

Everything mentioned above is of concern, because AI and data are applied with the intention to improve or optimize our lives. However, like any technology, AI and data can also be used with bad intentions. Think of AI-based cyber- attacks, terrorism, influencing important events with fake news, etc. (Brundage, 2018).

 

AI AND WARFARE

Another issue that requires attention is the application of AI for weapons and warfare, especially for LAWS armaments. Whether governments decide to use these types of applications is an explicit (political) decision, and certainly not something that will come as a surprise. Some will consider this a good use of AI, while others might call it an altogether nefarious misuse. Some organizations are already working on an international treaty to ban ‘killer robots’ (Delcker, 2018).

 

DEALING WITH THE SOCIETAL AND ETHICAL IMPACTS OF AI AND BIG DATA

It is for this reason that in the last two years many large organizations have publicly declared that they’ll adhere to AI principles or ethics guidelines. Harvard University ana- lysed the AI principles (Fjeld and Nagy, 2020) of the first 36 organizations in the world that published such guidelines. Harvard found nine categories of consideration, including human values, professional responsibility, human control, fairness & non-discrimination, transparency & explainability, safety & security, accountability, privacy and human rights. The not-for-profit organization Algorithm Watch maintains an open inventory of AI Guidelines4 with currently over 160 organizations. And the European Com- mission presented its Ethics Guidelines for Trustworthy AI (HLEG, 2019) in April 2019. My company, Telefónica, published its AI Principles5 in 2018, committing to the use of AI systems that are fair, transparent and explainable, human-centric, and with privacy & security.

These principles are an important first step toward the responsible use of AI, but principles alone are not enough. They need to be transformed into organizational processes such that they become BAU. Though initial experiences (Benjamins et al, 2019) are being shared and published (Benjamins, 2020), and there is a growing body of experience and learning, there’s still a long way to go (O’Brian et al., 2020), (Newmann Cussins, 2020).

While governments need to remain vigilant for malicious use of these powerful technologies, the positive opportunities for AI and big data are enormous and will continue to grow. We believe that in the future, from a technological perspective, it will to a large extent be possible to manage and avoid the unintended, negative consequences of AI, such as bias, discrimination and opaque algorithms.

CONCLUSION

This book is about how organizations can become more data- and AI-driven, helping them capture all opportunities. But in this chapter, we’ve seen that there are also potential negative consequences, even though most of them will be unintended. While it is important to remain vigilant for those ethical impacts, before deciding whether to use or not use AI and big data, you should also be aware that not using those technologies might be worse on many levels. Therefore, not using such technology might also have an ethical implication.

In the next chapter, we will see how companies and public entities can deal with these issues in practice, to pre- vent and mitigate the negative impacts as much as possible. AI and big data by themselves are not bad or good, but it’s the use organizations make of them that determines the impact, and that is a choice based on each organization’s norms and values. Generally speaking, we can think of an ethics continuum that ranges from good to bad (Figure 19.1), and it’s up to each organization to decide where it wants to be along this spectrum (Benjamins, 2020).

Figure 19.1 Ethics continuum of how AI can impact society