In his post-apocalyptic novel Towards the End of Time, John Updike imagined a war between the US and China in 2020. Looking back, the novel’s chief protagonist, one of Updike’s many white, angst-ridden, middle-class males, remarks that the war had been inspired by ‘highly-trained young men and women in sealed chambers of safety reading 3-D Computer graphics’. Sitting in sealed cubicles, typing codes and commands into computers whose screens display people as stylised icons, colour bars and pulsing network flows, it is precisely these highly-trained young men and women who are likely to be the principal characters in history’s next Great Power war (Updike 1997: 286).
But as war becomes increasingly dependent on technology, what of individual agency? Is war slipping out of our hands? How long will we continue to ‘own’ it? Agency is a tricky business–it is framed by the stories we tell ourselves, as well as others. And when it comes to modern technology, we have been telling ourselves some very distinctive stories for some time. Updike caught one trend–the extent to which humans are becoming even more distanced from their own actions, more machine readable, and compliant. But there is another story which I wish to address in this essay. Will be eventually replaced by robots?
There is nothing like an Economist headline. ‘Taming the Terminators’ appeared in an early January edition in 2019. It added that the ‘Campaign to stop killer robots’ which is headed by the International Committee of the Red Cross manages to combine two of the great and fearful tropes of science fiction–the peculiarly powerful weapon and the encounter with a non-human intelligence that might one day find it difficult to appreciate the value of human life and the significance of its loss. That is the great fear–that we will one day be replaced. In the 1930s Brecht penned a poem: “General, here we have a handy little gadget/called a ‘man’; he can fly a plane/he can kill. He has only one defect/he can think.” Well, some think that the robots of the future won’t suffer that defect which is why were frightened of them because they won’t be able to think through their actions; others fear that they will acquire the ability to think for themselves and unlike us they won’t be so easy to socialise into obedience. This essay sets out to debunk that fear. It discusses first the extent to which our agency has always been shaped by our relationship with tolls, and then technology; the extent to which Artificial Intelligence (AI) may or may not be the great game-changer; and the likelihood that we will continue to be in control thanks to our unique cognitive abilities. None of which detracts from the fact that our relationship with machines is already beginning to change–they are becoming not just tolls, but collaborators.
Agency in war as in life has always depended, at least hitherto, on a range of human qualities: willpower, courage, fear, tactical experience and creativity, what TE Lawrence described as the irrational tenth’ which could only be ensured, he added by “instinct, sharpened by thought.’ Less often acknowledged is the extent to which human agency has always been enhanced by our relationship with tools, both organic and non-organic, and latterly with technology.
In the thinking of Bruno Latour (1999), the ‘social’ represents more than a bonding with other people: it also constitutes an association between human beings and their tools, from the inanimate to the animate–horses and slaves (the Greeks called slaves ‘talking tools’). Ask yourself this question. If we did not have a hammer to hammer in a nail; if we tried to run a company without bookkeeping; if neither made a difference could we talk about agency at all? Agency, Latour adds, is what ‘allows’, ‘determines’, ‘permits’, ‘facilitates’, ‘renders possible or impossible’. It allows us to upgrade or improve weapons and thus increase the scope of our ambitions; it can determine the difference between success and failure in battle; it can even render war obsolete (as we once liked to think about the invention of nuclear weapons). From the first use of social tools that made possible social complexity such as the division of labour, we went on to invent intellectual tools such as writing and counting that made it possible in turn to organise human society into a mega-machine (civilisation). We then went on to assemble machines in the industrial era that were ruled by laws that had a sort of ‘social life’. Sociology, insists Latour, should be about the science of associations and not merely the science of the ‘social’; it should involve an understanding of relationships with non-human artefacts as well as other people.
Latour’s argument is especially interesting if you willing to acknowledge the way that at the same time technology has also transformed us as agents. This has actually been a recurring theme since the days of Plato. Read his last and most dyspeptic work The Laws, and you will find Socrates regretting the cognitive loss that was involved in evolving from an oral culture into a written one. This is Plato in his old age and at his most critical of his own culture. Today there are scores of books warning us of the cognitive loss involving the new technologies. Nicholas Carr’s The Shallows argues that technologies like the internet threaten to make our lives smaller and less emotionally rich by making us less sociable, an important factor, of course, if human agency requires working with others (Carr 2011). You will find a similar argument in Susan Greenfield’s book Tomorrow’s People (2003) with its subtitle: ‘How 21st Century Technology is changing the way we think and not always for the better’. Sherry Turkle is perhaps however the most famous proponent of the idea that the Internet for example is busy refashioning our minds. Her early fascination with the impact of computers and how we think about our minds, she tells us in one of her books, was prompted by a student who complained that a Freudian slip was no big deal–it was only an ‘information processing error’. It was at that moment that she realised she was witnessing an important new phenomenon; computers were beginning to alter the way we saw ourselves. We are already beginning to become more machine-like as well as more machine readable (Turkle 2011: 63).
What we can say with some confidence I think, is that digital technology is reshaping our experience of everyday life; it is re-mastering our experience of time and place; in a word it is changing our experience of the everyday. Code is increasingly shaping our behaviour. As individuals we are no longer an autonomous subject; instead, ‘our selfhood is smeared across a global mesh of nodes and links’. And what’s more, every time that we tweet or answer an email, we are rewiring our neuro-transmitter pathways, lighting up in the process the rewards circuits of our brain. Digital technology is so addictive for that reason.
What we are also beginning to experience is a third change in our relationship with tools and technology: not just the extension of our agency, and changes in the neural patterning of our brains; we may also be empowering ourselves by programming machines to manage us. Internet search engines, for example, now ‘manage’ us by reading our thoughts; directing us to what they think we might find more interesting (e.g. where to go on holiday next). They even read our moods and filter through what we read and select websites that they think may cheer us up. In the case of drone pilots, for example, computers now monitor their brain rhythms, heart rates and eye movements in order to scan their attentiveness, and thus ensure that they are focusing on the job. A pilot can be ‘shut off’ if she is deemed to be in danger of getting too stressed out, and control transferred to others.
The real game changer is going to be Artificial Intelligence (AI), and this is the main focus of this lecture. And the question we would have to raise, whether we like it or not is whether technology will eventually replace us: ‘Does the future still need us?’ was the provocative title of an article that appeared some years ago in the magazine Wired.
It is already transforming our lives, for good or ill. Companies are using it to manage stock indexes the better to optimise tax strategies and balance holdings between different portfolios. The professional money-manager does this once a year; AI does it every day. In 2016 an IT company in Hong Kong put an algorithm on its main board. As it is, algorithms are already doing the work of traders, journalists and lawyers. Narrative Science a Chicago based company expects that one day very soon computers will be researching, analysing, transcribing, drafting, editing, sub editing and despatching many of the articles we will read (Harari 2016: 232). Will robots and not humans soon be writing articles about our future? We already coexist with machines–employers ask whether we are good at working with them, whether our skills complement theirs, or whether the computers work better without us.
What of war? We are already debating the coming of autonomous machines–‘killer robots’ as the international Red Cross likes to call them in a bid to awaken us to our probable fate. But independent, self-directed killing machines are not just the weapons of the future; they’re already here. America’s ‘semi–autonomous’ Long Range Anti-ship Missile may need humans to identify targets but it deploys AI to decide how to destroy them. Israel already uses the Harap, a drone that seeks out and destroys radar systems on its own without human permission, loitering in the sky until a target appears. The Sea Hunter, an American anti–submarine vessel, can cruise the oceans for months at a time scouting for enemies with no one aboard, a veritable lethal Marie Celeste. If no fully autonomous weapon has yet been developed some 400 partly–autonomous weapons and robotic systems are currently under development around the world. The AlphaGo Zero machine developed to play the game board, Go clearly demonstrates that a machine is algorithmically capable of learning and developing creative strategies, as well solving problems through experience. The question I want to raise however is could a machine, however smart, however intelligently designed, also make ethical judgements on our behalf?
Ronald Arkin, who has been trying to design a conscience that can be programmed into the next generation of machines, thinks that “simply being human is the weakest point of the kill chain”–our biology works against us in our attempt to comply with the tenets of humanitarian law. Autonomous weapon systems, or so we are told, are likely to out-perform us in situations where bounded morality (i.e. situation-specific) apply. For it is the situation in which humans usually find themselves that usually encourages loss of human control, and many of the immoral actions that flow from it. Robots, by comparison, would not be situationists–largely because they would not have two wrestle with the fight or flight dynamic that is hard-wired into the rest of us. They are unlikely to suffer from the human prejudices that predispose us to see our enemies as ‘towel-heads’ or ‘gooks’, and they would not be prone to a familiar problem in social psychology, that of ‘scenario fulfilment’–the way in which our minds tend to reinforce pre-existing belief patterns. But then how intelligent would a robot have to be to assure us that there would always make the right judgement calls? And here’s another problem. The flipside of agency’s culpability, guilt and shame when we act in a way that we consider, often after the fact, wrong or simply immoral. Take agency away and evolve it to others, you may also remove guilt.
Unfortunately we are not always sure what we are talking about. In fact it is amazing that we should speak of ‘Artificial Intelligence’ when we do not really know what our own intelligence is. When we talk of intelligence do we mean rationality, or competence, or reasonableness? Some scientists then again draw a sharp distinction between consciousness and other essential processes such as cognition, sense and perception. It is also significant that ‘intelligence’, like ‘consciousness’ has different connotations in different languages. Take one of the most famous lines from Hamlet, the most famous of Shakespeare’s plays. “Thus conscience does make cowards of us all”. Conscience here means not only moral scruple, but also consciousness–the very act of thinking that may cause us–here I complete the quotation–“to lose the name of action” (i.e. to stop us from committing ourselves, to be reduced to agents who can’t actually act.)
Anyway, machine consciousness is probably a long way off, though Google’s Ray Kurtzweil has brought forward by three years the date of the so-called ‘Singularity’–the date when the machines become self-conscious for the first time. But insofar as machines are becoming, and will continue to become more autonomous–insofar as we really are probably locked into building ‘killer robots’ that that eventually may be able to re-program themselves in ways that we may not always be able to predict, they’re not going to be able to turn on us, or select totally different targets for a very simple reason: they will lack agency. At best will be actors (and perhaps collaborators) but not agents.
There are at least three reasons for this.
The first and most obvious is that machine intelligence, such as it is, is direction-less. We have motivated intelligence. Thanks to natural selection, we are governed by a mix of emotions, instincts and drives that are programmed into us so that we can reproduce. Natural selection gives us goals: to win, dominate and control others. We have to balance those goals with our need to survive–choosing actions that are life-sustaining. are not motivated by any innate desire to sustain their own existence.
And because we won’t ascribe intentions to machines we won’t need to engage in a social relationship with them. ‘Orders of Intentionality’ is a term the philosopher Daniel Dennett introduced to help us think about how social intelligence works. We have evolved social intelligence for a reason: to be able to cooperate with others. For example, if I believe you to know something, that I can cope with one order of intentionality. If I believe that you believe that I know something, then I can cope with two orders of intentionality. If I believe that you believe that my business partner believes that I know something, then I can cope with three orders of intentionality. We humans regularly encounter at least three orders in everyday life; Dennett believes that actually we can cope with up to 5. In other words, an entity that has no motivation is not one that can be networked into social life on other than the most basic terms. Agents are not programmed to act in predictable ways; natural selection allows us to adapt to our environments in ways that it’s impossible to always anticipate or predict (Dennett 2017: 130).
Secondly, machines are not only motivation-less, they are also non-teleological. Teleological questions include such basic questions as: what am I doing on this battlefield? Why am I taking risks? What is this entire conflict about? Am I willing to die, and if so, for what: a religion, country, family? All of these are aspirational and involve a teleological language, one that produces a sense of purpose, or end. I find this idea expressed most eloquently in a letter that the novelist Saul Bellow wrote to a friend about one of his most famous characters, Augie Marsh. Augie, he insisted, was the embodiment of a peculiar human trait: the willingness to serve others. Augie is a man who insists: “For God’s sake, make use of me, only do not use me to no purpose” (Bellow 2010). Surely, Bellow added, the greatest human desire is not to be used, so much as to be useful. Young Jihadists are often only too willing to surrender their lives in the hope of being useful to others. Being useful is what justifies martyrdom–an emotional response to life that we secularists have particular difficulty coming to terms with. Machines may be programmed to self-destruct but for them it is not a choice, or a gift–martyrdom is not in their emotional register.
Thirdly, we understand our own agency to be determined by rationality, not logic. Reason, not logic plays a central role in our lives though we must not fall into the trap of thinking, like our ancestors, that it is the human essence, let alone conclude that we are always rational, still less reasonable–we are not. And even when we try to apply reason unreservedly the result is often disastrous–it was Hegel who described his own subject philosophy as “common sense turned upside down’ and who criticised the most doctrinaire French revolutionaries for ‘trying to walk on their heads’.
Anyway we are not as rational as we like to think. We are still prone to all the problems that we’ve inherited from the mental toolkit of our distant ancestors: problems such as confirmation bias, cognitive dissonance and premature cognitive closure. And we do so for a reason. As Daniel Dennett hypothesises, they were probably programmed into us not to help us make better decisions, but to bond with the tribe against enemies, real or imagined which is why we continue to jump to conclusions and over-react, and frame our decisions in terms of our own communal/tribal experience (Dennett 2017: 94).
There are times when we put logic before reason–mathematicians have to do so but for most human activities we’ve discovered that logic can be dangerous, because it is often contrary to ordinary common sense. The physicist Nils Bohr put it very well when he told a student: “Stop being so bloody logical, and start thinking!” The point is that life is inconstant, and contradiction central to it. We don’t always recognise that fact because we built a world that takes note of it, and because we learnt over time to navigate the world we have built. Take Issaac Asimov’s Three Laws of Robotics. In his later tales as robots evolve their own intelligence, they find that exercising laws grounded in un-human logical structures of thought is almost impossible in a human-built world. By far the most intelligent exploration of this theme was by a computer scientist called Roger Clarke who published two articles in Computer Magazine in 1993, a year before Asimov’s death. Clarke set out a series of challenges that Asimov had had to grapple with as his robots evolved in his own imagination, as they became more complex and sophisticated, and more intelligent over time. As Clarke acknowledged, in the end Asimov’s Laws simply could not be applied logically–not because there was any inherent fault of design in the algorithms, but because moral life itself is not algorithmic. We have to make it up as we go along. This is our world, and war is our invention; it is what we do as a species and we do it very well (Clarke 1993, 1994).
It is because we can choose to break Asimov’s rules–i.e. our own moral rules of thumb that we are responsible for the choices we make. A human being has moral standing for that reason. Robots don’t choose whether to follow their program or not; consistency is their strong suit, not ours. We build machines to perform a sequence of operations that follow logically on from one another. They can do this faster than we can for a reason. Word-defined basic postulates or axioms on which pure logic draws is a strong suit in machines but not in humans.
For us, however, living ethically has never been about how best to optimise the good; it has involved instead the precept of right conduct, for example how we should treat Prisoners of War. It has involved cultivating virtues and refusing to perform actions that we cannot reconcile with our conscience or sense of self. Living ethically for that reason is rational, not logical: it involves balancing the claims of a variety of goods (such as winning ethically as against winning by suspending the rules), and calculating how different values should be applied in circumstances where there may often be no single idea of right or wrong.
AI is not going to change war for some time yet. What it will do is further amplify the way war will be driven by technological drivers (i.e. our own relationship with machines as we become increasingly absorbed into them, and they into us–the man/machine symbiosis, or what is often called the ‘post human condition’)
What has changed already is that as they have become more intelligent, we have begun to regard AI driven systems no longer as tools, but collaborators. For example, the extensive coding required to create the algorithms that guide autonomous systems requires multiple programmers working on the same programme, with none of them knowing the whole package. It’s for this reason that there’s been a push towards programming that would allow a system to communicate its decision processes directly to its user, and thus provide a greater level of interaction and communication between human and non-human entities. This is why what is probably the most comprehensive analysis of the factors influencing human–robot interaction–the multi-authored A Meta-analysis of factors affecting trust in human robot interaction (2011) begins with a discussion of human–robot team work. In other words as robotic capabilities increase we should see them not just as tools but also team mates.
It is not an entirely new idea. Actor Network theorists after all, have for some time rejected the idea that either the machine or a human being is a dominant actor in a relationship; they insist that both are inextricably interlinked in each other’s function and fate. Today we know that machines are simply better at monitoring risks; assembling information, analysing data (finding patterns, if they are there to be found); reacting faster, and even operating other machines. And we also know that we are becoming ever more reliant on them as they become more intelligent which is why, writes John Markoff, the chief science writer of The New York Times it is important that we should treat them as colleagues rather than slaves, otherwise we risk becoming over-reliant on them, and in the process losing the power to think or act independently if necessary. An example he gives is reliance on GPS for direction and the correction of navigational errors. It is all very well to deploy such a technology, but it also hinders our ability to remember and reason spatially–still a useful survival skill in the military.
But let me another way of thinking about cooperation with machines. Team players or not, we’ll be working with robots in the future; and even more important we will be relying on algorithms to a greater extent than ever. Our lives are open books that can be read by smart phones that take note of where we go, we talk to and what we say. Soldiers in the field too will soon have their heart rates measured to discover their stress levels and reliability. But we also have to be aware of what the software that increasingly governs our world can and cannot do. Most of the time technology does much better than we do in certain areas; at others it fails quite badly. Take the case of the driverless car that can navigate with great accuracy. Their visual systems however can be thrown into disarray by a single stray pixel. Volvo’s machines are baffled by kangaroos. Most of the software for face recognition used by the police is still deeply flawed. Apple’s Face ID system which ostensibly has a one–in–a–million error rate can easily be duped into unlocking iPhones by people with 3-D printed masks of the owners. The future probably lies not with robots but with us–as the reconstituted humans, or cyborgs, we already becoming–our lives are increasingly informed and guided by algorithms. But the ultimate responsibility for our actions will most likely continue to lie with us, not the algorithms.
In the end, it is probably best neither to exaggerate the extent of human control in war, nor the extent to which we can replace it with mechanical means, but recognise that we will probably continue to need each other. The dangers posed by machines in the end will correlate exactly with how much leeway with give them in fulfilling our own goals. And remember that each problem that we solve with AI will end up creating new ones–there’s always a trade-off. Often these bargains will be worth striking but as with any bargain it is best to make them in the full knowledge of what yourself getting yourself into.
And that’s important if you subscribe to Aristotle’s claim that the only purpose of war is peace. Drone attacks, for example, have transformed the way in which we try to control the human terrain but success so far has been mixed: they’ve destroyed many of our enemies but they have also created many more. We tend to forget the fact that although we may invent even smarter machines to conduct war on our behalf, only we can make peace with each other.
The author has no competing interests to declare.
Clarke, R. (1993). Asimov’s Laws of robotics: Implications for information technology – Part I. Computer, 26(12), 53–61. DOI: https://doi.org/10.1109/2.247652
Clarke, R. (1994). Asimov’s laws of robotics: implications for information technology – Part II. Computer, 27(1), 57–66. DOI: https://doi.org/10.1109/2.248881