mTags

Killer robotsAutonomous weapons

Addition date

Feb 25, 2022 11:52 AM

Column

Raphael Thys

Link

https://www.hrw.org/news/2021/12/01/crunch-time-killer-robots

LAN

At the United Nations in Geneva the Campaign to Stop Killer Robots called on governments to not allow the development of weapons systems that would select and attack targets without any human intervention.

The Sixth Review Conference of the Convention on Conventional Weapons (CCW), scheduled to be held at the United Nations in Geneva from December 13-17, 2021, is a major juncture for international talks on lethal autonomous weapons systems. After holding informal and formal discussions on the matter since 2013, states parties now face the pivotal decision of whether to approve a mandate to open negotiations of a protocol on the systems or to leave the CCW to initiate negotiations elsewhere.

Judging by an assessment of the two CCW meetings held so far this year, much debate at the Review Conference will center on views of the adequacy of existing international humanitarian law regarding autonomy in weapons systems. At the most recent CCW Group of Governmental Experts (GGE) meeting on lethal autonomous weapons systems in September-October, most CCW states parties called for a new legally binding instrument on the topic, while a minority countered that existing international humanitarian law is sufficient to address any problems raised by autonomous weapons systems. Other states have yet to express a clear position on the question.

This briefing paper explains why a new treaty on autonomous weapons systems is needed to clarify and strengthen existing international humanitarian law. Such an instrument would address the legal, ethical, accountability, and security concerns such systems pose by including the following elements:

• A broad scope that covers all weapons systems that select and engage targets on the basis of sensor inputs—that is, systems in which the object to be attacked is determined by sensor processing, not by humans;
• A general obligation to retain meaningful human control over the use of force;
• A prohibition on the development, production, and use of weapons systems that by their nature select and engage targets without meaningful human control;
• A prohibition on the development, production, and use of autonomous weapons systems that target people; and
• Positive obligations to ensure other autonomous weapons systems cannot be used without meaningful human control.

This briefing paper provides an overview of states’ positions on the adequacy of existing international humanitarian law, highlighting the widespread support for new law and noting that any divergence of views reinforces the need to clarify existing law. It examines justifications for a new instrument on grounds of international humanitarian law, ethics, international human rights law, accountability, and security. The paper then discusses the way forward, identifying potential forums for negotiating a new treaty outside of the CCW, including an independent stand-alone process and the United Nations General Assembly.

Recommendations

The emergence of autonomous weapons systems and the prospect of losing meaningful human control over the use of force are grave threats that demand urgent action. Human Rights Watch and Harvard Law School’s International Human Rights Clinic (IHRC) call on CCW states parties to move beyond diplomatic discussions and:

• Express their support for a new legally binding instrument on autonomous weapons systems at the next GGE meeting, scheduled for December 2-8, and at the Sixth Review Conference that follows;
• Call for a legally binding instrument that includes prohibitions and regulations to preserve meaningful human control over the use of force and bans autonomous weapons systems that target people; and
• Agree at the Sixth Review Conference to a mandate to negotiate a new CCW protocol, or, if that fails, commit to initiate as soon as possible negotiations of a legally binding instrument on autonomous weapons systems elsewhere.

Firm Support for New Law

The mandate of the 2021 GGE includes considering “possible recommendations” for the Review Conference with regard to a “normative and operational framework” on lethal autonomous weapons systems.[1] During the GGE’s September discussions of this framework, a dominant theme was the ongoing question of whether existing international humanitarian law is adequate to address the challenges posed by autonomous weapons systems. States agree that this body of law applies, but disagree about its adequacy. A combination of widespread state support for new law and the need to clarify current international legal rules bolsters the case for a legally binding instrument on autonomous weapons systems.

Most states that expressed their views in statements at or in submissions to the September GGE argued for supplementing existing international humanitarian law with new law to address the threats posed by autonomous weapons systems.[2] The list included at least 20 individual states, from Africa, Asia-Pacific, Europe, Latin America, and the Middle East, in addition to the Non-Aligned Movement.[3] Some delegations noted that law should continually evolve. The Philippines, for example, explained that existing norms are “ill-equipped” to govern the changing nature of warfare and must be made “future-proof.”[4] Algeria similarly stated that current law was not sufficient to deal with these cutting-edge systems and emphasized the importance of developing and codifying new regulations.[5] Other states saw the value in clarifying international humanitarian law (IHL). The representative for Ecuador stated, "I think that we can all agree … that there currently exists a lack of clarity about how IHL applies to these LAWS [lethal autonomous weapons systems]…. [I]t is absolutely essential to clarify these elements and the most appropriate here would be to have this in a legally binding instrument which would be specific to this type of weapon."[6]

Recognizing the divergence of views about the limits that international law places on autonomy in weapons systems, Austria said, “We need to avoid legal insecurity or ambiguities by setting out clear legally binding rules about what the international community would consider unacceptable and what would be accepted within agreed limits.”[7]

Proponents of a new legal instrument identified and advocated for specific prohibitions and regulations that would address the legal, ethical, accountability, and security concerns raised by autonomous weapons systems. The informal Group of 10, which consists of Argentina, Costa Rica, Ecuador, El Salvador, Palestine, Panama, Peru, the Philippines, Sierra Leone, and Uruguay, argued that a new CCW protocol should prohibit fully autonomous weapons systems, antipersonnel autonomous weapons systems, and “autonomous weapons systems lacking sufficient predictability, foreseeability, reliability, oversight, or explainability.”[8] Brazil, Mexico, and Chile, which generally spoke as a group, called for prohibitions on autonomous weapons systems that, for example, made life-and-death decisions, lacked human control, did not allow for the attribution of responsibility, and were too complex for a human to understand or explain.[9] The International Committee of the Red Cross (ICRC) and the Campaign to Stop Killer Robots proposed similar legal frameworks that encompassed prohibitions on inexplicable or unpredictable weapons systems (i.e., systems that lacked meaningful human control), prohibitions on weapons systems that targeted people, and regulations on other systems to ensure meaningful human control is preserved.[10] Supporters of such provisions emphasized that they should be international and legally binding to avoid national differences and promote enforceability.[11]

Despite widespread support for a new legally binding instrument, several states, most notably India, Russia, and the United States, countered that existing international humanitarian law is sufficient and opposed negotiation of a new legally binding instrument.[12]

Other states – China among them – did not offer clear positions on international humanitarian law’s adequacy.[13] Portugal proposed compiling a non-binding compendium of existing international humanitarian law relevant to autonomous weapons systems, akin to the Montreux Document on private military and security companies operations.[14] Portugal cautioned that its suggestion to restate existing law was “without prejudice … to an international legally binding instrument,” but several opponents of a new treaty latched onto the proposal as a way forward.[15] Others firmly dismissed it. Austria responded that this work had already been done by academics and civil society.[16] The Philippines called the exercise a “distraction” from the CCW’s work.[17] Dedicating time to such a compendium would also consume energy and resources that could be devoted to the negotiation and drafting of a new treaty.

The divergence of opinions on the adequacy of international humanitarian law does not undermine that fact that, as noted earlier, a majority of states believe there is a need for new law. Instead, the fact that states disagree about whether existing international humanitarian law is adequate indicates there is dangerous confusion about how that body of law will apply to this emerging technology. According to the ICRC:

Such divergences of views among states may result in autonomous weapons being developed and used by some, that others consider incompatible with international humanitarian law. Such a situation is of concern from the perspective of protecting the victims of war. It serves to highlight the need to clarify and strengthen legal protections in line with ethical considerations for humanity.[18]

To address that confusion, states should adopt a new legally binding instrument that clarifies existing law by providing specific, internationally accepted rules on autonomous weapons systems.

Arguments for a New Legally Binding Instrument

In making their case for a new legally binding instrument, states articulated arguments based on the legal, ethical, accountability, and security concerns raised by autonomous weapons systems. A close examination of these concerns reveals that existing international humanitarian law does not adequately address them. New law with tailored provisions and a broader scope would address gaps in the current legal regime by clarifying and strengthening existing law so that it could deal with the latest developments in the use of force.

International Humanitarian Law Concerns

The need to promote compliance with international humanitarian law and ensure clarity of its rules both argue for adoption of a new legally binding treaty on autonomous weapons systems. Brazil, Chile, and Mexico, for example, argued that new rules regulating the development and use of autonomous weapons systems are necessary “to ensure that international law, and in particular IHL, is fully observed.”[19] The Non-Aligned Movement, in a statement delivered by Venezuela, mentioned the international humanitarian law principles of distinction, proportionality, and precaution in its call for a new treaty.[20] Numerous proponents of a new instrument referred to the Martens Clause, which as explained below, links law and ethics, as important to their evaluation of the acceptability of autonomous weapons systems.[21]

It would be difficult for fully autonomous weapons systems, which would select and engage targets without meaningful human control, to distinguish between combatants and non-combatants as required under international humanitarian law.[22] Combatants often do not wear uniforms or insignia and deliberately seek to blend in with civilian populations. In addition, a soldier who is wearing a uniform may be wounded or trying to surrender and therefore be hors de combat and an unlawful target.[23] As a result, complying with the principle of distinction frequently demands the ability to assess an individual’s conduct and intentions, not just appearance. Such assessments may require interpreting subtle cues in a person’s tone of voice, facial expressions, or body language or being aware of local culture. According to the Group of 10, “[T]he identification of body language indicating surrender requires complex understanding of cultural nuances that would fail to be captured in biometric markers and could only be understood through human experience and emotion.”[24] Humans possess the unique capacity to identify with other human beings and are thus equipped to understand the nuances of unforeseen behavior in ways that machines, which must be programmed in advance, simply cannot.

The use of fully autonomous weapons systems also presents challenges for compliance with international humanitarian law’s proportionality principle, which prohibits attacks in which expected civilian harm is excessive in comparison to anticipated military advantage.[25] First, because a machine would have trouble distinguishing military from civilian targets, it will face obstacles to assessing the military advantage and civilian harm that would come from a possible attack. Second, the proportionality principle involves a qualitative balancing test that requires the application of human judgment and moral and ethical reasoning. According to the ICRC, judgments about whether a particular attack is proportionate “must above all be a question of common sense and good faith,” human characteristics that machines seem unlikely to possess through their programming.[26] Many states at the GGE, including Brazil, Chile, and Mexico and those from the Non-Aligned Movement, emphasized the importance of human judgment to ensuring compliance with the proportionality principle.[27] Third, proportionality requires contextual decisions at the moment of attack. The lawful response to a situation could change considerably by slightly altering the facts, and it would be impossible to pre-program a robot to be prepared for the infinite number of scenarios that it could face. A new legally binding instrument dedicated to autonomous weapons systems could codify clear prohibitions and regulations that ensure compliance with the principles of distinction and proportionality. In particular, a requirement that meaningful human control be maintained over the selection and engagement of targets would ensure that humans not machines make decisions about the legality of attacks.

Autonomous weapons systems raise additional concerns under the Martens Clause, customary international humanitarian law that numerous states invoked at the GGE.[28] As articulated by Additional Protocol I to the Geneva Conventions, the Martens Clause applies to situations not specifically covered by international agreements and mandates that in such circumstances, “civilians and combatants remain under the protection and authority of the principles of international law derived from established custom, from the principles of humanity and from the dictates of public conscience.” [29] The Martens Clause is relevant to autonomous weapons systems because there is no existing international law dedicated to this emerging technology.[30]

Established custom, the principles of humanity, and the dictates of public conscience all support creating a new legally binding instrument on autonomous weapons systems. Human beings have always made use-of-force decisions themselves; delegating that responsibility to machines would represent a dramatic break with international custom. The use of fully autonomous weapon systems and autonomous weapons that target people contravene the principles of humanity, which require actors to treat others humanely and to respect human life and dignity. Robots lack the compassion, empathy, mercy, and judgment necessary to treat humans humanely, and they cannot understand the inherent worth of human life.

The dictates of public conscience, the shared moral guidelines that shape the actions of states and individuals, are also aligned against such autonomous systems. There is widespread public opposition to these weapons from a diverse array of actors, expressed in public surveys, statements, and letters from the United Nations Secretary-General, Nobel Peace Laureates, faith leaders, artificial intelligence experts, private companies, and nongovernmental organizations (NGOs) working under the umbrella of the Campaign to Stop Killer Robots. Much of this opposition reflects moral repulsion at the idea of machines making life-and-death decisions.[31] The concerns raised under the Martens Clause illuminate the need for a dedicated treaty that would fill a gap in international treaty law and protect the principles of humanity and the dictates of public conscience in the face of emerging weapons technology.

Humanitarian disarmament precedent shows that when weapons are prone to violating distinction, proportionality, and the Martens Clause, states have found new law to be warranted. Due to their broad footprint and high failure rate, cluster munitions cannot distinguish between combatants and civilians, especially when they are used in populated areas.[32] Although some states claimed international law was adequate to deal with these indiscriminate weapons, others recognized the need for stronger and more specific norms and adopted the Convention on Cluster Munitions in 2008. The new treaty has ended use of these weapons by its 110 states parties and influenced the conduct of states not party, including the United States, which has launched only one cluster munition strike since the treaty was adopted.

In the 1990s, the potential development of blinding lasers implicated the Martens Clause. States expressed concern that the unnecessary suffering these weapons would have caused ran counter to the principles of humanity and the dictates of public conscience. States disagreed over whether blinding lasers could be used in compliance with existing international humanitarian law, but in 1995, they adopted CCW Protocol IV, which preemptively banned the weapons, because a dedicated instrument would bring clarity to the law.[33] With both cluster munitions and blinding lasers, new law created strong, unambiguous rules for weapons that arguably could not be used in compliance with existing international humanitarian law. A treaty on autonomous weapons systems could have a similar effect, even if not adopted by all states.

Moral and Ethical Considerations

The significant ethical concerns raised by autonomous weapon systems warrant the development of new rules because they are not adequately addressed by existing international humanitarian law. Austria, Brazil, Chile, Ireland, Luxembourg, Mexico, and New Zealand wrote in a joint submission to the GGE, for example, “Ethical considerations also underpin the necessity of establishing limits on autonomy in weapons systems and elaborating positive and negative obligations with respect to any potential development or use of such systems.”[34] The group argued that states should consider not only whether autonomous weapon systems are legally acceptable (“can the weapon be used in accordance with the law?”), but also whether they are ethically acceptable (“should we use this weapon?”).[35] The Non-Aligned Movement cited ethical and moral concerns in advocating for “regulations that will ensure meaningful human control over weapon systems.”[36] Panama similarly suggested that ethical considerations should be at “the heart” of any normative and operational framework.[37]

By removing humans from use-of-force decisions, and by reducing human targets to mere data points, fully autonomous weapons systems can trigger several moral and ethical threats. They would dehumanize war and remove the restraint of human compassion from the battlefield.[38] As noted by numerous states, including those in the Group of 10, fully autonomous weapons systems would also undermine human dignity, human agency, and moral responsibility.[39] According to scholar Peter Asaro, “When it comes to human dignity, what is crucial is both the manner in which the decision is made and the legitimacy of who is making the decision, not simply the final outcome of the decision.”[40] Asaro adds that to respect human dignity, a moral agent must “(1) recognize a human being as a human … ; (2) understand the value of life and the significance of its loss; and (3) reflect upon the reasons for taking life and reach a rational conclusion that killing is justified.”[41] Fully autonomous weapons systems inherently lack the qualitative capacities to respect, understand, and reflect that Asaro describes. In addition, these systems cannot be moral agents in the first place and do not possess the “consciousness, intentionality and free will” to be morally responsible for their actions.[42]

Autonomous weapons systems further raise the concern of algorithmic bias. Data sets are typically flawed and tend to disfavor traditionally disadvantaged groups. As explained by the Group of 10, “developers would base their programming on the information available to them and on their own personal experiences,” transferring the biases of their cultural upbringings into the algorithms.[43] New Zealand noted that the use of algorithms in civilian sectors has led to bias.[44] The use of weapons that rely on target profiles to attack humans would likely lead to comparable bias with lethal consequences.

Existing international humanitarian law does not fully address these ethical considerations. When the Geneva Conventions and their protocols were written, autonomous weapons systems were not yet a technological possibility, so the drafters would have presumed that only humans could apply force. The law, therefore, does not envision situations in which machines could make life-and-death determinations, and in so doing, threaten human dignity and raise the specter of algorithmic bias. As Brazil, Chile, and Mexico observed, meaningful human control does even not appear as a concept in the current legal regime.[45]

To adapt international humanitarian law to the latest revolution in warfare, ethical considerations should guide the development of a new legally binding instrument. According to the ICRC, “ethical concerns have always been an important driver for the evolution of international legal rules in warfare.”[46] In particular, states should adopt an instrument that ensures, through prohibitions and regulations, that meaningful human control is maintained over the use of force. The instrument should also prohibit autonomous weapons systems that target humans. These obligations would reduce the dehumanization of warfare, promote respect for human dignity, and avoid algorithmic bias. Some or all of these provisions have been proposed by Austria; Brazil, Chile, and Mexico; the Group of 10; and the Non-Aligned Movement, among others.[47]

International Human Rights Law Implications

The protection of fundamental rights and the need for a broad scope of application demand a new legally binding instrument that goes beyond existing international humanitarian law and incorporates international human rights law. At least 10 states at the GGE, in addition to the ICRC and civil society organizations, emphasized the applicability of international human rights law to autonomous weapons systems.[48] Brazil, Chile, and Mexico, for example, said in a joint statement that both international humanitarian law and international human rights law were at “the core” of the GGE’s work on these systems.[49]

The use of fully autonomous weapons systems would implicate several fundamental human rights. First, under the right to life, which prohibits any arbitrary deprivation of life, killing can only be lawful if it is necessary to protect human life, if it is a last resort, and if it is applied proportionately. Similar to international humanitarian law’s proportionality test discussed above, upholding the right to life requires making qualitative assessments through the application of human judgment, which would be difficult to replicate in fully autonomous weapons systems. Second, these systems would contravene the right to remedy, leaving a gap in accountability, which will be discussed further below. Palestine emphasized the importance of this right, and the obligation of states to “provide victims with adequate, effective, and prompt reparations for the harm suffered.”[50] Finally, as discussed above, fully autonomous weapons systems would undermine human dignity, which is a core international human rights law principle as well as an ethical one.[51] States, the ICRC, and civil society groups said taking this body of law into account is essential to addressing the concerns raised by autonomous weapons systems.[52]

International human rights law is also relevant to autonomous weapons systems because these systems will likely be used in a wide range of contexts, including armed conflict, law enforcement, and counterterrorism efforts. While international humanitarian law applies only in situations of armed conflict, international human rights law applies in all circumstances. As Sri Lanka said, “AWS [autonomous weapons systems] could be used both in situations of armed conflict and in other situations. Therefore, application of [international human rights law] to AWS is logical and pertinent.”[53]

A new legally binding instrument informed by international human rights law could close some of the gaps in existing international humanitarian law. By requiring meaningful human control over the use of force, it could protect fundamental human rights to life and a remedy, as well as the principle of human dignity. By applying its prohibitions and regulations “under any circumstances,” a treaty could ensure a comprehensive response to help prevent the use of these weapons systems for law enforcement, as well as military, purposes.

Accountability Gap

The need to ensure accountability for the use of force further justifies negotiation of a legally binding instrument tailored to autonomous weapons systems. Accountability was a major topic of discussion at the GGE, and there was consensus that “[h]uman responsibility for decisions on the use of weapons systems must be retained since accountability cannot be transferred to machines.”[54] Although some states said existing law adequately dealt with the issue, other states explicitly raised accountability in their calls for a new instrument.[55]

The use of weapons that operate without meaningful human control over the selection and engagement of targets would create an accountability gap under the current legal regime. It would be difficult to hold a commander or operator criminally responsible for the actions of a fully autonomous weapon system if that person could not predict or control the system’s actions. There are additional practical and legal obstacles to holding either the programmer or manufacturer of a fully autonomous weapon civilly liable.[56] The robot itself could not, of course, be held legally responsible for its action.

While existing international humanitarian law is ill-suited to address the liability challenges posed by autonomous weapons systems, a new legally binding instrument could be designed to ensure accountability. Many states recognized that maintaining meaningful human control is a prerequisite for ensuring individual responsibility. Brazil, Chile, and Mexico, for example, noted that preserving meaningful human control over the critical functions of autonomous systems prevents accountability gaps.[57] States should, therefore, adopt new international law to prevent the development, production, or use of weapons systems that operate without meaningful human control in order to ensure that a human will always be responsible for use-of-force decisions and to avoid accountability gaps.

Security Threats

Averting the security threats posed by autonomous weapons systems necessitates an instrument with comprehensive prohibitions and regulations. The Non-Aligned Movement called for a legally binding instrument to address “the humanitarian and international security challenges posed by emerging technologies in the area of LAWS.”[58] Pakistan warned that the absence of rules on autonomous weapons systems would produce security consequences contradicting “the very objectives and purposes” of the CCW.[59]

Autonomous weapons systems pose several security risks. First, the development and production of autonomous weapons systems could lead to an arms race as states rush to develop technologies to keep pace with or outpace their adversaries. The Group of 10 noted that such an arms race would increase “existing strategic inequalities and further erode international peace, stability, and security.”[60] Moreover, an arms race could create a second, spillover risk: proliferation to groups or states that may not care about civilian protection or may have criminal intentions. There was consensus at the GGE on the need to address “the risk of acquisitions by terrorist groups and the risk of proliferation.”[61] Autonomous weapons systems do not require rare materials and can be mass produced more easily than nuclear weapons, augmenting this proliferation risk.

Once developed and produced, autonomous weapons systems could increase the risk of armed conflict. Pakistan argued that states possessing autonomous weapons “will likely be more inclined to use them given the limited to no loss of soldiers and citizens on the battlefield. Such a scenario will certainly lower the thresholds for going to war.”[62] Although autonomous weapons systems’ potential to reduce military casualties would be a positive development, it could shift more of war’s burdens onto civilians. In addition, the speed at which these systems could operate could lead to rapid escalations of armed conflicts.

Many of the security risks detailed above—an arms race, proliferation, and lowering the threshold for war—stem not only from the potential use of autonomous weapons, but also from their development and production. International humanitarian law, however, focuses on the use of weapon systems. To mitigate security threats, therefore, a legally binding instrument should strengthen international humanitarian law by establishing clear prohibitions and regulations not only on the use of autonomous weapons systems, but also on their development and production.

Way Forward

When the CCW GGE convenes in December for its third session of 2021, states have an opportunity to recommend that the Review Conference adopt a mandate to negotiate a new protocol on autonomous weapons systems. Since the CCW follows a consensus decision-making process, however, staunch opposition by certain states means that this result is unlikely. Should states be unable to reach consensus, the CCW will have served its purpose and states should move to an independent process or to the UN General Assembly to negotiate a legally binding instrument. The Ottawa Process, which produced the 1997 Mine Ban Treaty, the Oslo Process, which adopted the 2008 Convention on Cluster Munitions, and the General Assembly negotiations that generated the 2017 Treaty on the Prohibition of Nuclear Weapons demonstrate the effectiveness of such alternatives.

CCW Process

States parties to the CCW have an opportunity to pursue a legally binding protocol under the auspices of that convention if they agree in December to a mandate to negotiate such an instrument. Some states parties suggested that it is beyond the GGE’s purview to recommend that the Review Conference initiate such negotiations.[63] A close reading of the group’s current mandate, adopted in 2019, does not support that position, however, and numerous states argued for a negotiating mandate during the September GGE.[64] CCW states parties tasked the GGE with producing consensus recommendations that relate to the “clarification, consideration and development of aspects of the normative and operational framework on emerging technologies in the area of lethal autonomous weapons systems” (emphasis added).[65] “Development” means “the act or process of creating something over a period of time,”[66] and thus would encompass any actions that would help bring about a normative and operational framework, including negotiation of a new legally binding instrument.

Procedurally, the most viable path to a protocol would involve three steps. First, the GGE, which is scheduled to meet from December 2-8, would agree to recommend that the Review Conference adopt a negotiating mandate. Second, the Review Conference would adopt the GGE’s recommendation, or if the GGE did not recommend negotiations, the Review Conference would adopt a negotiating mandate anyway. Third, states parties would negotiate and adopt a protocol. Action at each of these stages, however, would require consensus. Although a majority of states support these steps, a minority remain opposed.[67] Under the consensus rule that governs the CCW, a single state’s opposition is enough to derail progress altogether.

States parties should not settle for an extension to their current discussion mandate or a mandate to negotiate something less than a legally binding instrument, such as a political declaration or voluntary code of conduct, as some countries have suggested.[68] While the CCW has provided a forum for examining the legal, ethical, accountability, and security concerns raised by autonomous weapons systems, it is time to move from assessing the problems and identifying guiding principles to creating a binding solution. Most CCW states positions have coalesced around the need for a new legal instrument including prohibitions and regulations, and they should be prepared to act with the urgency that the situation demands by pursuing a treaty through an independent process or through the UN General Assembly. As the Campaign to Stop Killer Robots has noted, the forthcoming Sixth Review Conference is the final milestone by which states need to demonstrate that the CCW remains an appropriate forum to achieve regulation for autonomous weapon systems.[69] The CCW has served as an incubator for discussions of autonomous weapons systems, but barring unexpected progress at the Review Conference, it is time to take action elsewhere.

Independent Treaty Process

States seeking an alternative forum to the CCW have twice turned to an independent process to adopt strong international disarmament treaties. In 1996, at Canada’s initiative, states moved outside the CCW to negotiate the Mine Ban Treaty, in what became known as the Ottawa Process. At Norway’s invitation, states following the same fast-track approach negotiated the Convention on Cluster Munitions during the 2007-2008 Oslo Process.

Freed of CCW’s rules of procedure, these stand-alone processes had no consensus requirements, which made for stronger and more equitable outcomes. While a consensus forum tends to produce weak instruments because states must compromise to reach an agreement that usually ends up reflecting the lowest common denominator, processes that allow for voting enable states to follow through on their commitments to ban or regulate weapons systems without fear of a veto. The absence of a consensus requirement also makes the process more efficient and empowers small and middle-sized states, especially states in the Global South that are disproportionately harmed by armed conflict.[70] In forums like the CCW, large military powers have undue influence because they are generally the only ones willing to block consensus. Both the Ottawa and Oslo Processes exemplified the effectiveness of establishing voting rules—decisions of substance required support from a two-thirds majority. The potential for a vote allowed progressive states to aim high without deterring cooperation for a common humanitarian objective. In the end, no votes were held in either case and a strong convention was adopted by acclamation.[71]

An independent process can also be highly inclusive. The Ottawa Process demonstrated the importance of civil society to humanitarian disarmament, a people-centered approach to governing weapon systems.[72] The International Campaign to Ban Landmines, which along with its coordinator, Jody Williams, won the 1997 Nobel Peace Prize, pushed states to initiate the process. The global coalition of nongovernmental organizations fought to ensure that the final treaty included components critical to the treaty’s success, such as an unambiguous definition of landmines; a comprehensive ban on production, stockpile, and use of landmines; and obligations on stockpile destruction, clearance, victim assistance, and reporting. [73]

The Oslo Process featured a broad range of states, bringing together those affected by cluster munitions with more than half of the world’s stockpilers and two-thirds of the world’s users and producers. It also benefited from the involvement of international organizations and civil society groups led by the Cluster Munition Coalition.[74] Cluster munition survivors from Europe, the Middle East, and Southeast Asia lobbied intensively on key treaty provisions and provided technical input to diplomats that was central to the success of the treaty negotiations.[75] States and civil society worked in close partnership with the support of United Nations agencies and the ICRC.

An independent process also tends to move quickly, adopting new law within months rather than years. For example, less than six months after the CCW’s First Review Conference failed to prohibit antipersonnel landmines through Amended Protocol II, Canada invited interested states to Ottawa, where it announced a process to ban landmines. Austria, Germany, and Belgium hosted Ottawa Process conferences to discuss, revise, and debate draft texts of the convention. Norway hosted the negotiations in September 1997 where—less than a year after the launch of the Ottawa Process—states adopted the Mine Ban Treaty. [76] Today, 164 states are party to the treaty.[77]

The Oslo Process followed a similar trajectory. After states failed to adequately address cluster munitions under the CCW, Norway announced in 2006 that it would launch an independent process to prohibit cluster munitions that cause “unacceptable harm to civilians.”[78] Norway led a core group of seven states, including three stockpilers of cluster munitions, that took turns hosting conferences attended by as many as 138 countries to develop and negotiate a draft convention in 2007 and 2008.[79] At each successive meeting, the Core Group presented draft text for feedback, making steady progress towards a binding treaty. All 107 states that participated in the final negotiations in Dublin adopted the convention in May 2008, 15 months after the Oslo Process began.[80] The Convention on Cluster Munitions currently has 110 states parties and 13 signatories.

UN General Assembly Treaty Process

States have also turned to the UN General Assembly to provide a forum for negotiating a legally binding disarmament instrument after unsuccessful efforts to obtain consensus in other forums. By 2016, frustration with the unwillingness of nuclear-armed states to pursue good-faith disarmament under the auspices of the Nuclear Non-Proliferation Treaty (NPT) came to a head.[81] Concern about the catastrophic humanitarian consequences of nuclear weapons and the failure of nuclear armed states to disarm prompted non-nuclear armed states to organize the “Humanitarian Initiative” in partnership with civil society and the ICRC. International meetings convened in 2013-2014 by Norway, Austria, and Mexico refocused the global debate about nuclear weapons as a matter of humanitarian rather than national security concern.[82] Dissatisfied with what they saw as a failure by the NPT to adopt a consensus on nuclear disarmament at the 2015 Review Conference, supporters of the Humanitarian Initiative turned to UN General Assembly, which passed a resolution in 2016 mandating negotiations towards a nuclear weapons ban. Two conferences at the UN in New York, spread over the course of four weeks, culminated in adoption of the Treaty on the Prohibition of Nuclear Weapons in July 2017.

Similar to its independent counterparts, a General Assembly process does not require consensus. The 2016 General Assembly resolution mandating negotiations on a treaty passed 113 to 35 with 13 abstentions.[83] While every nuclear-armed state either voted no or abstained, the process still went forward, proving that disarmament is a matter of concern for more than just major military powers. The final vote on the Treaty on the Prohibition of Nuclear Weapons also did not achieve consensus. The treaty was adopted 122 to 1 with one abstention.[84] Instead of being defeated by the Netherlands’ sole no vote, as would have been the case in a CCW consensus process, the landmark treaty was adopted almost unanimously. The treaty entered into force on January 22, 2021, 90 days after Honduras provided the 50th ratification.

The UN General Assembly process, like an independent one, also allows for some degree of inclusivity. The General Assembly encompasses a wider range of states than the CCW’s list of states parties, and it is open to contributions from international organizations and civil society groups. The International Campaign to Abolish Nuclear Weapons (ICAN), the 2017 Nobel Peace Laureate, played an instrumental role in advancing the Humanitarian Initiative, securing the General Assembly resolution that mandated negotiations, and shaping the text of the final treaty. ICAN understood from the outset that consensus was impossible, but by leveraging lessons learned from civil society’s role in both the Oslo and Ottawa independent processes, it helped shepherd states through to the successful adoption of a binding treaty.[85]

When choosing the best forum for negotiating an autonomous weapons systems treaty, states should carefully consider the UN General Assembly model’s distinct characteristics. On the one hand, it carries the weight of UN authority. Since nearly all states in the world are members of the General Assembly, the results of its negotiations are harder for opponents to dismiss than those of an independent process. On the other hand, civil society is more marginalized in the General Assembly process compared to the independent process. Although civil society organizations were allowed to attend plenary sessions and continued to significantly influence the negotiations of the Treaty on the Prohibition of Nuclear Weapons, they did not have access to the working groups where the actual text of the treaty was negotiated. In the independent processes, by contrast, civil society’s role was almost on par with that of states. In addition, the General Assembly track is generally slower than that of an independent process. While the treaty negotiations themselves took only seven months, reaching that stage took several years. More than 100 states participating in the Humanitarian Initiative had pledged to prohibit nuclear weapons by mid-2015, but the General Assembly responded by creating an “open-ended working group” to consider the possibility. Only after the group’s work was completed did the General Assembly pass a resolution mandating negotiations.[86]

Conclusion

In the event CCW states parties are unable to reach consensus on a mandate to negotiate a new protocol on autonomous weapons systems when they convene in December, an independent process may prove to be the most inclusive and efficient alternative. Regardless which forum they pursue, however, it is crucial that states act quickly to capitalize on the momentum building in the CCW.

States should ensure a legally binding instrument that includes prohibitions and regulations to address the legal, ethical, accountability, and security concerns raised by autonomous weapons systems. The treaty should include (1) a general obligation to maintain meaningful human control over the use of force, (2) prohibitions on weapons systems that select and engage targets without meaningful human control or that target people, and (3) specific positive obligations ensuring meaningful human control is maintained in the use of all other systems. A treaty that incorporates these key elements will set strong norms that will influence states parties and those not party alike.

[1] “Report of the 2019 Session of the Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems,” CCW/GGE.1/2019/3, September 25, 2019, https://reachingcriticalwill.org/images/documents/Disarmament-fora/ccw/2019/gge/Documents/final-report.pdf (accessed November 13, 2021), para. 26(d).

[2] States’ positions are drawn from the UN audio files for the CCW Group of Governmental Experts meeting, which are available at https://conf.unog.ch/digitalrecordings, and in some cases, from Reaching Critical Will, CCW Report, vol. 9, no. 7, October 4, 2021, https://reachingcriticalwill.org/images/documents/Disarmament-fora/ccw/2021/gge/reports/CCWR9.7.pdf. Links to the audio files for specific statements are given when a statement is quoted or a specifically discussed. All these links were accessed on November 22, 2021.

[3] See, for example, Statement of Brazil on behalf of Brazil, Chile, and Mexico, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files),https://conf.unog.ch/dr/public/61.0500/9393801E-E15D-4CA3-B154-401AA343ACB1_10h19/chunks/snippet_lEs18-03t32-05.mp3; Statement of Algeria, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, October 1, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9ED00BAC-C1AA-4814-B9BF-B11CED83E406_15h16/chunks/snippet_lEs86-32t92-43.mp3; Statement of Austria, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, October 1, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9ED00BAC-C1AA-4814-B9BF-B11CED83E406_15h16/chunks/snippet_lEs92-54t101-47.mp3; Statement of Pakistan, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, October 1, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/E4CBCB25-F77B-4DD2-95DC-27EA860F1224_10h15/chunks/snippet_lEs137-59t140-48.mp3; Statement of Palestine on behalf of the Group of 10 (Argentina, Costa Rica, Ecuador, El Salvador, Palestine, Panama, Peru, the Philippines, Sierra Leone, and Uruguay), CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9393801E-E15D-4CA3-B154-401AA343ACB1_10h19/chunks/snippet_lEs62-44t70-24.mp3; Statement of Venezuela on behalf of the Non-Aligned Movement, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, October 1, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9ED00BAC-C1AA-4814-B9BF-B11CED83E406_15h16/chunks/snippet_lEs73-23t86-17.mp3. See also “Submission by Austria, Brazil, Chile, Ireland, Luxembourg, Mexico and New Zealand on Ethical Considerations to the Chair of the Group of Governmental Experts on Technologies in the Area of Lethal Autonomous Weapons Systems,” September 2021, https://reachingcriticalwill.org/images/documents/Disarmament-fora/ccw/2021/gge/documents/Austria-et-al_sept.pdf (accessed November 13, 2021), p. 3.

[4] Statement of the Philippines, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, October 1, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9ED00BAC-C1AA-4814-B9BF-B11CED83E406_15h16/chunks/snippet_lEs30-47t37-16.mp3.

[9] Statement of Mexico on behalf of Brazil, Chile, and Mexico, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9393801E-E15D-4CA3-B154-401AA343ACB1_10h19/chunks/snippet_lEs18-03t32-05.mp3.

[10] Statement of the International Committee of the Red Cross (ICRC), CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 29, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/2B332FF9-EAE8-4BF8-8FD2-9B85753FF17B_15h12/chunks/snippet_lEs134-07t145-11.mp3; Statement of the Campaign to Stop Killer Robots, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 30, 2021, https://reachingcriticalwill.org/images/documents/Disarmament-fora/ccw/2021/gge/statements/30Sept_CSKR.pdf.

[11] See, for example, Statements of Palestine of behalf of the Group of 10, and Brazil on behalf of Brazil, Chile, and Mexico, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files); Statement of Venezuela on behalf of the Non-Aligned Movement, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, October 1, 2021 (UN audio files).

[12] Statement of India, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9393801E-E15D-4CA3-B154-401AA343ACB1_10h19/chunks/snippet_lEs135-04t139-04.mp3; Statement of Russia, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9393801E-E15D-4CA3-B154-401AA343ACB1_10h19/chunks/snippet_lEs101-13t111-16.mp3; Statement of the United States, CCW GGE Meeting on Lethal Autonomous Weapons System, Geneva, September 27, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/F3453490-09B5-449B-96A6-ABC200CF6107_10h24/chunks/snippet_lEs47-40t58-09.mp3 (“[W]e believe the existing IHL and effective measures at the national level to implement IHL are sufficient to address the challenges posed by LAWS.”).

[13] Statement of China, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 29, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/4CE967CA-89B6-4FB0-BE11-AFD9769E4F5D_10h15/chunks/snippet_lEs20-38t22-11.mp3 (noting disagreement on whether existing international humanitarian law is adequate without itself taking a position); Statement of Finland, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 29, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/4CE967CA-89B6-4FB0-BE11-AFD9769E4F5D_10h15/chunks/snippet_lEs22-19t23-59.mp3 (supporting Portugal’s proposal of a non-binding compendium but offering little additional commentary); Statement of Sweden, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9393801E-E15D-4CA3-B154-401AA343ACB1_10h19/chunks/snippet_lEs81-15t85-40.mp3 (stating that weapons not allowed under international law are unlawful).

[14] The Montreux Document is a compilation of international humanitarian and human rights law pertaining to private military and security companies. It spells out best practices, legal obligations, and responsibilities and is explicitly non-binding upon states. ICRC, “The Montreux Document,” August 2009, https://www.icrc.org/en/doc/assets/files/other/icrc_002_0996.pdf (accessed November 12, 2021), p. 31.

[15] “Proposal by Portugal for a Consensus Recommendation by the GGE LAWS,” September 2021, https://reachingcriticalwill.org/images/documents/Disarmament-fora/ccw/2021/gge/documents/Portugal_sept.pdf (accessed November 19, 2021), p. 1. For examples of opponents of a legally binding instrument supporting the compendium proposal, see, for example, Statement of Russia, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 29, 2021 (UN audio files); Statements of Australia, India, and the United States, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, October 1, 2021 (UN audio files).

[16] Statement of Austria, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, October 1, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9ED00BAC-C1AA-4814-B9BF-B11CED83E406_15h16/chunks/snippet_lEs92-54t101-47.mp3.

[18] Statement of ICRC, CCW GEE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 29, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/2B332FF9-EAE8-4BF8-8FD2-9B85753FF17B_15h12/chunks/snippet_lEs134-07t145-11.mp3.

[19] Statement of Brazil on behalf of Brazil, Chile, and Mexico, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9393801E-E15D-4CA3-B154-401AA343ACB1_10h19/chunks/snippet_lEs18-03t32-05.mp3 (stating in full: “[T]here is a clear need for establishing a set of specific rules to regulate the development and the use of AWS and to shed light over the grey areas in which we have to ensure that international law, and in particular IHL, is fully observed for the sake of the principle of humanity and human dignity.”).

[20] Statement of Venezuela on behalf of the Non-Aligned Movement, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, October 1, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9ED00BAC-C1AA-4814-B9BF-B11CED83E406_15h16/chunks/snippet_lEs73-23t86-17.mp3.

[21] See, for example, Statements of Malta, New Zealand, and Palestine on behalf of the Group of 10, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files); Statements of Austria, the Philippines, and Chile on behalf of Brazil, Chile, and Mexico, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 29, 2021 (UN audio files).

[22] Protocol Additional to the Geneva Conventions of 12 August 1949, and Relating to the Protection of Victims of International Armed Conflicts (Protocol I), adopted June 8, 1977, 1125 U.N.T.S. 3, entered into force December 7, 1978, art. 51(3); Protocol Additional to the Geneva Conventions of 12 August 1949, and Relating to the Protection of Victims of Non-International Armed Conflicts (Protocol II), 1125 U.N.T.S. 609, entered into force December 7, 1978, art. 13(3).

[23] If a combatant is no longer participating in hostilities because they have surrendered or have suffered wounds, they are hors de combat and may no longer be attacked. Protocol I, arts. 41(1) and 85(3)(e).

[24] Statement of Palestine on behalf of the Group of 10, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9393801E-E15D-4CA3-B154-401AA343ACB1_10h19/chunks/snippet_lEs62-44t70-24.mp3.

[26] Claude Pilloud et al., Commentary of 1987 on Protocol I, eds. Yves Sandoz, Christophe Swinarski, and Bruno Zimmermann (ICRC, Geneva: Martinus Nijhoff Publishers, 1987), art. 57, para. 2208 (emphasis added) (“Even if this system is based to some extent on a subjective evaluation, the interpretation must above all be a question of common sense and good faith for military commanders. In every attack they must carefully weigh up the humanitarian and military interests at stake.”).

[27] Statement of Venezuela on behalf of the Non-Aligned Movement, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, October 1, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9ED00BAC-C1AA-4814-B9BF-B11CED83E406_15h16/chunks/snippet_lEs73-23t86-17.mp3; Statement of Mexico on behalf of Brazil, Chile, and Mexico, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 30, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/BBCD7228-3EEE-4C80-B2CA-7576A0EDFDE0_10h10/chunks/snippet_lEs107-12t116-37.mp3.

[28] See, for example, Statements of Malta, New Zealand, and Palestine on behalf of the Group of 10, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files); Statements of Austria, the Philippines, and Chile on behalf of Brazil, Chile, and Mexico, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 29, 2021 (UN audio files).

[29] Protocol I, art. 1(2). See also Statement of Chile on behalf of Brazil, Chile, and Mexico, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 27, 2021, (UN audio files),https://conf.unog.ch/dr/public/61.0500/8E516E80-E260-4CF2-ACBC-FC8567546263_15h10/chunks/snippet_lEs18-43t36-15.mp3, (noting that states cannot “contravene” the principles of humanity and the dictates of public conscience).

[30] Statement of Chile on behalf of Brazil, Chile, and Mexico, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 27, 2021, (UN audio files), https://conf.unog.ch/dr/public/61.0500/8E516E80-E260-4CF2-ACBC-FC8567546263_15h10/chunks/snippet_lEs18-43t36-15.mp3, (“[W]e agree that in all cases and circumstances relating to AWS not covered by existing international agreement or custom, the Martens Clause ... applies.”).

[31] For more on the Martens Clause and its application to autonomous weapons systems, see Human Rights Watch and the Harvard Law School International Human Rights Clinic (IHRC), Heed the Call: A Moral and Legal Imperative to Ban Killer Robots, August 2018, https://www.hrw.org/report/2018/08/21/heed-call/moral-and-legal-imperative-ban-killer-robots.

[32] Human Rights Watch, Meeting the Challenge: Protecting Civilians through the Convention on Cluster Munitions, November 2010, https://www.hrw.org/sites/default/files/reports/armsclusters1110webwcover.pdf, p. 100.

[33] For more on the precedent of blinding lasers for autonomous weapons, see Human Rights Watch and IHRC, Precedent for Preemption: The Ban on Blinding Lasers as a Model for a Killer Robots Prohibition, November 2015, https://www.hrw.org/news/2015/11/08/precedent-preemption-ban-blinding-lasers-model-killer-robots-prohibition.

[34] “Submission by Austria, Brazil, Chile, Ireland, Luxembourg, Mexico and New Zealand on Ethical Considerations to the Chair of the Group of Governmental Experts on Technologies in the Area of Lethal Autonomous Weapons Systems,” p. 3.

[35] Ibid., p. 1 (“Taking ethical considerations into account in the LAWS debate challenges us to ask not only if a weapon is legally acceptable (can the weapon be used in accordance with the law?) but, would its use be acceptable from an ethical perspective: (should we use this weapon?)”).

[36] Statement of Venezuela on behalf of the Non-Aligned Movement, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, October 1, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9ED00BAC-C1AA-4814-B9BF-B11CED83E406_15h16/chunks/snippet_lEs73-23t86-17.mp3.

[37] Statement of Panama, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 28, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/EC63DFB5-E767-4662-A9A0-306E25A6F2AB_15h12/chunks/snippet_lEs42-05t45-00.mp3.

[38] “Report of the Special Rapporteur on Extrajudicial, Summary or Arbitrary Executions, Christof Heyns: Lethal Autonomous Robotics and the Protection of Life,” UN Human Rights Council, A/HRC/23/47, April 9, 2013, para. 89 (“Taking humans out of the loop also risks taking humanity out of the loop.”). See also Statement of Malta, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9393801E-E15D-4CA3-B154-401AA343ACB1_10h19/chunks/snippet_lEs111-41t114-30.mp3 (“we also subscribe to the view that the application of LAWS, insofar as these systems are preprogrammed to attack, would remove the human judgment to exercise discretion, flexibility, or adaptability, or perhaps restraint or mercy.”).

[39] Group of 10, “Written Contribution for the Chair of the Group of Government Experts (GGE) on Lethal Autonomous Weapon Systems (LAWS),” September 2021, https://reachingcriticalwill.org/images/documents/Disarmament-fora/ccw/2021/gge/documents/G10_sept.pdf (accessed November 20, 2021), pp. 4-5. See also “Submission by Austria, Brazil, Chile, Ireland, Luxembourg, Mexico and New Zealand on Ethical Considerations to the Chair of the Group of Governmental Experts on Technologies in the Area of Lethal Autonomous Weapons Systems,” p. 3 (arguing that a loss of meaningful human control “would undermine international humanitarian law, international human rights law and, in particular violate the principle of human dignity”); Statements of Austria, Malta, and Brazil on behalf of Brazil, Chile, and Mexico, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files). Switzerland similarly argued that “autonomous weapon systems that would identify and engage targets without any human control would not only be unlawful, but they would also be questionable from an ethical point of view, particularly with regard to human dignity.” Statement of Switzerland, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, October 1, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/E4CBCB25-F77B-4DD2-95DC-27EA860F1224_10h15/chunks/snippet_lEs56-33t61-39.mp3.

[40] See, for example, Peter Asaro, “Autonomous Weapons and the Ethics of Artificial Intelligence,” in Ethics of Artificial Intelligence, ed. S. Matthew Liao, (Oxford: Oxford Scholarship Online, October 2020), p. 230. Asaro takes a more deontological moral view, where morality is based on certain principles such as human dignity, whereas international humanitarian law operates under a more utilitarian framework that focuses on outcomes.

[42] James H. Moor, “The Nature, Importance and Difficulty of Machine Ethics,” IEEE Intelligent Systems (2006), p.20; Anthony Beavers, “Moral Machines and the Threat of Ethical Nihilism,” in Robot Ethics: The Ethical and Social Implications of Robotics, eds. Patrick Lin, Keith Abney, and George A. Bekey (Cambridge, MA: Massachusetts Institute of Technology, 2012), p. 6. As opposed to machines, fully ethical agents “can be held accountable for their actions—in the moral sense, they can be at fault—precisely because their decisions are in some rich sense up to them.” Ibid.

[43] Statement of Palestine on behalf of the Group of 10, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September, 24, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9393801E-E15D-4CA3-B154-401AA343ACB1_10h19/chunks/snippet_lEs62-44t70-24.mp3.

[45] Statement of Chile on behalf of Brazil, Chile, and Mexico, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/04D76E9F-E3A8-44A0-8E8D-646905C2901C_15h14/chunks/snippet_lEs57-59t62-51.mp3.

[47] Statement of Austria, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9393801E-E15D-4CA3-B154-401AA343ACB1_10h19/chunks/snippet_lEs91-21t100-52.mp3; ; Statement of Chile on behalf of Brazil, Chile, and Mexico, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 27, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/8E516E80-E260-4CF2-ACBC-FC8567546263_15h10/chunks/snippet_lEs18-43t36-15.mp3; Statement of Palestine on behalf of the Group of 10, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9393801E-E15D-4CA3-B154-401AA343ACB1_10h19/chunks/snippet_lEs62-44t70-24.mp3; Statement of Venezuela on behalf of the Non-Aligned Movement, CCW GGE Meeting on Lethal Autonomous Weapons Systems, October 1, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9ED00BAC-C1AA-4814-B9BF-B11CED83E406_15h16/chunks/snippet_lEs73-23t86-17.mp3.

[48] See, for example, Statements of Argentina, Austria, Brazil, Chile, Costa Rica, Mexico, New Zealand, Palestine, Panama, and the Philippines (reported in Reaching Critical Will, CCW Report, vol. 9, no. 7, October 4, 2021, pp. 5, 8, 20, 33, 34, 41). But see Statements of Israel and India (reported in Reaching Critical Will, CCW Report, vol. 9, no. 7, October 4, 2021, pp. 21, 33) (opposing reference to international human rights law).

[49] Statement of Brazil on behalf of Brazil, Chile, and Mexico, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 28, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/EC63DFB5-E767-4662-A9A0-306E25A6F2AB_15h12/chunks/snippet_lEs154-07t159-43.mp3.

[51] For more on the human rights concerns raised by autonomous weapons systems, see Human Rights Watch and IHRC, Shaking the Foundations: The Human Rights Implications of Killer Robots, May 2014, https://www.hrw.org/report/2014/05/12/shaking-foundations/human-rights-implications-killer-robots, p. 23.

[52] See, for example, Statement of Costa Rica, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021; Statements of Panama, Sri Lanka, and Amnesty International, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 29, 2021; Statement of the ICRC, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, October 1, 2021 (UN audio files).

[53] Statement of Sri Lanka, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 29, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/4CE967CA-89B6-4FB0-BE11-AFD9769E4F5D_10h15/chunks/snippet_lEs119-49t122-19.mp3. Multiple parties shared the view that international human rights law addresses circumstances outside armed conflict. See also Statements of the Netherlands, the Philippines, and Amnesty International, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 29, 2021 (UN audio files).

[54] “Report of the Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems,” CCW/GGE.1/2019/3, September 25, 2019, p. 13, Guiding Principle (b).

[55] For example of the latter view, see Statement of Austria, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, October 1, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9ED00BAC-C1AA-4814-B9BF-B11CED83E406_15h16/chunks/snippet_lEs92-54t101-47.mp3. For a summary of the discussion of accountability, see Reaching Critical Will, CCW Report, vol. 9, no. 7, October 4, 2021, pp. 38-43.

[56] For more analysis of the issue of accountability, see Human Rights Watch and IHRC, Mind the Gap: The Lack of Accountability for Killer Robots, April 2015, https://www.hrw.org/report/2015/04/09/mind-gap/lack-accountability-killer-robots.

[57] These states emphasized several elements of human control, including a human’s ability to understand the machine and supervise, approve, and intervene in its actions. See Statement of Mexico on behalf of Brazil, Chile, and Mexico, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 30, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/BBCD7228-3EEE-4C80-B2CA-7576A0EDFDE0_10h10/chunks/snippet_lEs107-12t116-37.mp3. Many other states supported a call to require meaningful human control. See, for example,Statements of Group of 10, Austria, and Switzerland (reported in Reaching Critical Will, CCW Report, vol. 9, no. 7, October 4, 2021, pp. 67, 47, 64.

[58] Statement of Venezuela on behalf of the Non-Aligned Movement, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, October 1, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9ED00BAC-C1AA-4814-B9BF-B11CED83E406_15h16/chunks/snippet_lEs73-23t86-17.mp3.

[59] Statement of Pakistan, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 27, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/F3453490-09B5-449B-96A6-ABC200CF6107_10h24/chunks/snippet_lEs119-58t124-58.mp3. Pakistan also warned that the absence of restraints would lead to “costly arms races in the area of LAWS, use of these weapons by non-state actors, unintended or uncontrolled levels of escalation, spirals of reprisals or expanding conflicts.”

[60] Statement of Palestine on behalf of the Group of 10, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9393801E-E15D-4CA3-B154-401AA343ACB1_10h19/chunks/snippet_lEs62-44t70-24.mp3.

[61] “Report of the Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems,” CCW/GGE.1/2019/3, September 25, 2019, p. 13, Guiding Principle (f).

[62] Statement of Pakistan, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 27, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/F3453490-09B5-449B-96A6-ABC200CF6107_10h24/chunks/snippet_lEs119-58t124-58.mp3. In a similar vein, the Group of 10 noted, “Another challenge posed by autonomous weapon systems is that they further increase the physical and psychological distance of humans from the specific time and place of force application thus increasing existing asymmetries and reducing the threshold for the use of force.” Statement of Palestine on behalf of the Group of 10, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 24, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9393801E-E15D-4CA3-B154-401AA343ACB1_10h19/chunks/snippet_lEs62-44t70-24.mp3.

[63] See, for example, Statement of Russia, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 29, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/2B332FF9-EAE8-4BF8-8FD2-9B85753FF17B_15h12/chunks/snippet_lEs147-05t150-48.mp3; Statement of Israel CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 29, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/2B332FF9-EAE8-4BF8-8FD2-9B85753FF17B_15h12/chunks/snippet_lEs165-21t166-04.mp3.

[64] See, for example, Statements by Argentina, Costa Rica, Ecuador, El Salvador, Palestine, Panama, Peru, the Philippines, Sierra Leone, and Uruguay, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 22, 2021 (reported in Reaching Critical Will, CCW Report, vol. 9, no. 7, October 4, 2021, p. 14).

[65] “Report of the Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems,” CCW/GGE.1/2019/3, September 25, 2019, para. 26(d).

[66] Merriam-Webster.com Dictionary, s.v. “development, https://www.merriam-webster.com/dictionary/development (accessed November 10, 2021).

[67] For support for a CCW protocol prohibiting and regulating autonomous weapons systems, see, for example, Statements of Argentina, Ecuador, Mexico, Palestine, and the Philippines, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 27, 2021 (UN audio files); Statement of Chile, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 29, 2021 (UN audio files); Statements of Algeria, Panama, and Peru, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 30, 2021 (UN audio files); Statement of Costa Rica, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, October 1, 2021 (UN audio files). For arguments that existing international humanitarian law is sufficient to govern autonomous weapons systems and that a new normative and operational framework is not necessary, see, for example, Statements of India and Russia, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 28, 2021 (UN audio files); Statement of Israel, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 29, 2021 (UN audio files).

[68] Russia proposed that the GGE’s discussion mandate be extended until the Seventh CCW Review Conference in 2026. Statement of Russia, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, October 1, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/9ED00BAC-C1AA-4814-B9BF-B11CED83E406_15h16/chunks/snippet_lEs44-25t48-19.mp3. See also Statement of the Republic of Korea, CCW GGE Meeting on Lethal Autonomous Weapons Systems, Geneva, September 27, 2021 (UN audio files), https://conf.unog.ch/dr/public/61.0500/F3453490-09B5-449B-96A6-ABC200CF6107_10h24/chunks/snippet_lEs125-14t127-40.mp3 (expressing its openness "to the option of adopting or formulating some sort of political declaration seeking to reaffirm our previously-agreed consensus as one of the outcomes of the Sixth Review Conference").

[69] Campaign to Stop Killer Robots, “Recommendations on the Normative and Operational Framework for Autonomous Weapon Systems,” June 2021, https://www.stopkillerrobots.org/wp-content/uploads/2021/09/CSKR_Normative-and-Operational-Framework-Written-Commentary.pdf (accessed November 15, 2021).

[72] For a discussion on humanitarian disarmament, see Bonnie Docherty, “A ‘Light for All Humanity’: The Treaty on the Prohibition of Nuclear Weapons and the Progress of Humanitarian Disarmament,” Global Change, Peace & Security, vol. 30, issue 2 (May 2018), pp. 167-169.

[73] Jody Williams and Stephen Goose, “The International Campaign to Ban Landmines,” in To Walk Without Fear: The Global Movement to Ban Landmines, eds. Maxwell A. Cameron, Robert J. Lawson, and Brian W. Tomlin (Oxford University Press, 1998), pp. 20-47. See also Stuart Maslen and Peter Herby, “An International Ban on Anti-Personnel Mines: History and Negotiation of the ‘Ottawa Treaty’,” The International Committee of the Red Cross, No. 325, December 31, 1998, https://www.icrc.org/en/doc/resources/documents/article/other/57jpjn.htm (accessed November 1, 2021).

[74] Human Rights Watch, Meeting the Challenge, pp. 121-22. There were campaigners from 61 countries represented in the Cluster Munition Coalition at the final negotiations. Ibid., p. 134.

[77] International Campaign to Ban Landmines, Landmine Monitor 2021, November 2021 http://www.the-monitor.org/en-gb/reports/2021/landmine-monitor-2021.aspx (accessed November 14, 2021).

[81] Elizabeth Minor, “Changing the Discourse on Nuclear Weapons: The Humanitarian Initiative,” International Review of the Red Cross, no. 97, 2015, https://international-review.icrc.org/sites/default/files/irc97_12.pdf (accessed November 3, 2021), pp. 715-717.

[85] Motoko Mekata, “How Transnational Civil Society Realized the Ban Treaty: An Interview with Beatrice Fihn,” Journal for Peace and Nuclear Disarmament, vol. 1, issue 1 (March 13, 2018), pp. 79, 82.