Assertion Supported AI assessment confidence: 95% certainty 4/5 debate potential 2/5

Brundage: Offense generally defeats defense in adversarial machine learning competitions

Miles Brundage · A.I. Policy and Public Perception - Miles Brundage and Tim Hwang · Y Combinator · Apr 25, 2018 · at 12:10

Miles Brundage, AI Policy Research Fellow at the Future of Humanity Institute, discusses machine learning robustness and technical security challenges.

0:00 / 0:12exact quote · 12.9s

▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark

“If you look at the, like, offense and defense and competitions on adversarial examples, like, the offense generally wins. Like, we don't really know how to make neural nets robust against deliberate or even unintentional things that could mess them up.”

quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →

The claim, plainly

In competitions testing adversarial examples in machine learning, offensive attacks generally defeat defensive techniques, and researchers do not know how to make neural networks robust against deliberate tampering or distribution shifts.

a quote alone can be ambiguous, so this restates it with the references and context of the conversation around it resolved. AI-written; the tape below the player is the source.

Assessment

As of April 2018, empirical benchmark competitions and peer-reviewed machine learning research substantiated both aspects of the claim. In the primary benchmark of the period, the NIPS 2017 Adversarial Attacks and Defenses Competition (reported in March 2018), defenses demonstrated severe fragility against worst-case attacks (the highest worst-case accuracy achieved was only 53.6%), with top-performing defenses subsequently circumvented under adaptive and white-box conditions. Furthermore, contemporary academic literature (including Athalye et al., February 2018, and Carlini & Wagner, 2017) demonstrated that proposed defenses and detection mechanisms routinely failed under rigorous evaluation, while consensus reports (such as Brundage et al., February 2018) affirmed that creating neural networks robust against deliberate perturbations and out-of-distribution shifts remained an unsolved challenge.

The NIPS 2017 Adversarial Attacks and Defenses Competition report (arXiv:1804.00097, March 31, 2018) showed that defensive submissions were severely vulnerable to worst-case adversarial attacks, achieving at most 53.6% worst-case accuracy, and leading defenses were bypassed under adaptive attacks.
Athalye, Carlini, and Wagner (arXiv:1802.00420, February 2018) demonstrated that 7 of 9 defenses accepted to ICLR 2018 relied on obfuscated gradients and were broken via adaptive attack methods.
The multi-institutional report 'The Malicious Use of Artificial Intelligence' (February 2018) and related literature confirmed that general robustness against deliberate tampering and distribution shifts remained an open, unsolved problem in deep learning.

Checked against

[1] arxiv.org [2] kaggle.com [3] arxiv.org [4] arxiv.org [5] engineering.org.cn [6] arxiv.org [7] mlr.press [8] github.com

"The Malicious Use of Artificial Intelligence" Brundage February 2018 "adversarial" "NIPS 2017" "Adversarial Attacks and Defenses" competition results "attacks defeated" OR "accuracy" OR "defenses" NIPS 2017 adversarial attacks defenses competition results Kurakin paper Obfuscated Gradients Give a False Sense of Security Athalye 2018 ICLR "Towards Evaluating the Robustness of Neural Networks" Carlini Wagner 2017

assessed as of 2026-09-05

This assessment is an AI reading of a public statement, made against the cited references with the confidence shown. It can be wrong. The quote, the tape and the references are all on this page; if something is off, use report an error below. full disclaimer →

Mentioned

adversarial examples neural nets

⚑ Spot an error? Report it

More from Miles Brundage

Prediction Not checkable as stated

Brundage: Scalable AI vulnerability discovery could cause a catastrophic cyber attack

“It could be that you know, something truly catastrophic could happen if you sort of combine the, Scalability of AI and digital technology in general with, like, the adaptability of human intelligence for, like, finding vulnerabilities. If you put those togethe…”

Miles Brundage Apr 25, 2018 ▶ 8:54 A.I. Policy and Public Perception - Miles Brundage and Tim Hwang · Y Combinator

Insight

Brundage: Reducing AI existential risk by 0.1% preserves massive expected value

“Reducing the probability of existential risk is super important, even if AI is you know, decades or centuries away, and even if we can only, you know, decrease the probability, you know, of that happening by, like.1% or whatever in expectation, that's, like, a…”

Miles Brundage Apr 25, 2018 ▶ 33:39 A.I. Policy and Public Perception - Miles Brundage and Tim Hwang · Y Combinator

Opinion

Brundage: Technical transparency in AGI could enable international non-aggression pacts

“And I think if you actually had the full development of the FAT methods, and you had accountability and transparency for even general AI systems or superintelligent systems, I think that would open up the door for a lot more collaboration. If you could sort of…”

Miles Brundage Apr 25, 2018 ▶ 38:47 A.I. Policy and Public Perception - Miles Brundage and Tim Hwang · Y Combinator

Insight

Brundage: AGI accountability may be easier than thought if corrigibility stabilizes feedback

“Corrigibility, what he calls corrigibility, and what others have called corrigibility, might actually be, like, a stable basin of attraction, in the sense that if a system, you know is designed in such a way that it's able to, like, take critical feedback, and…”

Miles Brundage Apr 25, 2018 ▶ 39:55 A.I. Policy and Public Perception - Miles Brundage and Tim Hwang · Y Combinator

Opinion

Brundage: China will not slow AI deployment over interpretability concerns

“In China, there's, like, much let, or I haven't seen as much concern about interpretability, though there are some, like, good papers coming out of China, but in terms of, like, governance, I haven't gotten the sense that they're gonna, like, hold back the dep…”

Miles Brundage Apr 25, 2018 ▶ 17:48 A.I. Policy and Public Perception - Miles Brundage and Tim Hwang · Y Combinator

Opinion

Brundage: AI Governance Lacks the Methodological Rigor of Climate Science

“People have been talking about AI AI ethics and AI governance for a long time, but there hasn't been much dialogue between, you know, this world and then the other worlds of, like, you know, science policy and public policy, and, you know, one way to think abo…”

Miles Brundage Apr 25, 2018 ▶ 21:46 A.I. Policy and Public Perception - Miles Brundage and Tim Hwang · Y Combinator

Brundage: Offense generally defeats defense in adversarial machine learning competitions

More from Miles Brundage

Turn any episode into a week of clips.