Research Scientist, Google Brain
nicholas [at] carlini [dot] com
GitHub | 中国5G商用，在开放合作中“提速”_新视听 - jnnc.com:2021-11-19 · 据路透社报道，全球移动通信系统协会GSMA智库发布研究指出，至2021年，中国预计将会有6亿5G用户，在绝对数量上领先全球。
Nicholas Carlini
Research Scientist, Google Brain
nicholas [at] carlini [dot] com
GitHub | Google Scholar
I am a research scientist at Google Brain working at the intersection of machine learning and computer security. My most recent line of work studies properties of neural networks from an adversarial perspective. I received my Ph.D. from UC Berkeley in 2018, and my B.A. in computer science and mathematics (also from UC Berkeley) in 2013.
Generally, I am interested in developing attacks on machine learning systems; most of my work develops attacks demonstrating security and privacy risks of these systems. I have received best paper awards at ICML and IEEE S&P, and my work has been featured in the New York Times, the BBC, Nature Magazine, Science Magazine, Wired, and Popular Science.
Previously I interned at Google Brain, evaluating the privacy of machine learning; Intel, evaluating Control-Flow Enforcement Technology (CET); and Matasano Security, doing security testing and designing an embedded security CTF.
A complete list of my publications are online, along with some of my code, and some extra writings.
Recent Work
Last year I made a doom clone in JavaScript. Until recently all content on this website was research, and while writing papers can be fun Who are we kidding? Writing is never fun. But it's the cost of admission when doing research, which definitely is. , sometimes you just need to blow off a little steam. The entire game fits in 13k---the 3d renderer, shadow mapper, game engine, levels, enemies, and music. The post talks about the process of designing the game and how to make it all happen under the constraints.
At CAMLIS 2024 I gave a talk covering what it means to evaluate adversarial robustness. This is a much higher-level talk for an audience that isn't deeply familiar with the area of adversarial machine learning research. (For a more technical version of this talk, see my recent USENIX Security invited talk that discusses these same topics in more depth.) The talk covers what adversarial examples are, how to generate them, how to (try to) defend against them, and finally what the future may hold.
At ICML 2018, I presented a paper I wrote with Anish Athalye and my advisor David Wagner: 一套大屏、一本白皮书、四场主题活动，创头条闪亮全国双创 ...:2021-6-20 · 其中，全国双创数据大屏、2021科技创新创业论坛、2021孵化载体特色发展大会、“风向标—中国创新创业先锋论坛”、2021阿里巴巴全球诸神之战创客 .... In this paper, we demonstrate that most of the ICLR'18 adversarial example defenses were, in fact, ineffective at defending against attack and in fact just broke existing attack algorithms. We introduce stronger attacks that work in the presence of what we call “obfuscated gradients”. Because we won best paper, we were able to give two talks, the talk linked here is plenary talk where I argue that the evaluation methodology used widely in the community today is insufficient, and can be improved.
At the 2nd IEEE Deep Learning and Security Workshop, I received the best paper award for a paper with my advisor David Wagner Audio Adversarial Examples: Targeted Attacks on Speech-to-Text. In this paper, we demonstrate that it is possible to construct two audio samples that sound nearly indistinguishable but where a machine learning algorithm would recognize them completely differently. This paper in part builds on our prior work, where we constructed audio that sounds like noise to humans but speech to machine learning algorithms. This demonstration picked up a few rounds of press and was covered by the New York Times, Tech Crunch, and CNET (among others).
In 2017 at IEEE S&P I received the best student paper award for a paper with my advisor David Wagner Towards Evaluating the Robustness of Neural Networks. In this paper, we introduce a class of attacks for generating adversarial examples based on optimization methods using gradient descent. We argue that iterative optimization-based attacks are significantly more effective than prior attacks, and demonstrate that fact on multiple datasets.