Farhad Manjoo: Cameras With Brains!

Technology
The Sublime and Scary Future of Cameras With A.I. Brains

Farhad Manjoo

STATE OF THE ART FEB. 27, 2018

Something strange, scary and sublime is happening to cameras, and it’s going to complicate everything you knew about pictures. Cameras are getting brains.

Until the past few years, just about all cameras — whether smartphones or point-and-shoots or CCTV surveillance — were like eyes disconnected from any intelligence.

They captured anything you put in front of them, but they didn’t understand a whit about what they were seeing. Even basic facts about the world eluded them. It’s crazy, for instance, that in 2018, your smartphone doesn’t automatically detect when you’ve taken naked pictures of yourself and offer to house them under an extra-special layer of security.

But all this is changing. There’s a new generation of cameras that understand what they see. They’re eyes connected to brains, machines that no longer just see what you put in front of them, but can act on it — creating intriguing and sometimes eerie possibilities.

At first, these cameras will promise to let us take better pictures, to capture moments that might not have been possible with every dumb camera that came before. That’s the pitch Google is making with Clips, a new camera that went on sale on Tuesday. It uses so-called machine learning to automatically take snapshots of people, pets and other things it finds interesting.

Others are using artificial intelligence to make cameras more useful. You’ve heard how Apple’s newest iPhone uses face recognition to unlock your phone. A start-up called Lighthouse AI wants to do something similar for your home, using a security camera that adds a layer of visual intelligence to the images it sees. When you mount its camera in your entryway, it can constantly analyze the scene, alerting you if your dog walker doesn’t show up, or if your kids aren’t home by a certain time after school.

It doesn’t take long to imagine the useful and very creepy possibilities of cameras that can decipher the world. Digital cameras brought about a revolution in photography, but until now, it was only a revolution of scale: Thanks to microchips, cameras got smaller and cheaper, and we began carrying them everywhere.

Now, A.I. will create a revolution in how cameras work, too. Smart cameras will let you analyze pictures with prosecutorial precision, raising the specter of a new kind of surveillance — not just by the government but by everyone around you, even your loved ones at home.

The companies making these devices are aware of the privacy dangers. Many are moving into the field gingerly, slathering their products with safeguards that they say reduce the creepiness.

Take Google’s Clips, which I’ve used for the past week and half. It’s one of the most unusual devices I’ve ever encountered. The camera is about the size of a tin of mints, and it has no screen. On its front, there’s a lens and a button. The button takes a picture, but it’s only there if you really need it.

Instead, most of the time, you just rely on the camera’s intuition, which has been trained to recognize facial expressions, lighting, framing and other hallmarks of nice photos. It also recognizes familiar faces — the people you’re with more often are those it deems most interesting to photograph.

Clips, which sells for $249, makes taking pictures unconscious and all but invisible. Carry it around wherever you go; the camera has a handy case with a big bendy clip, so it can be affixed to your jacket, set on a tabletop, carried in your palm or placed anywhere else with a view.

From there, it’s all A.I.

Clips watches the scene, and when it sees something that looks like a compelling shot, it captures a 15-second burst picture (something like a short animated GIF or Live Photo on your iPhone).

I took a trip with my family to Disneyland last week, and over two highly photographable days, I barely took a photo. Instead, this tiny device automatically did the work, capturing a couple hundred short clips of our vacation.

Some of them were quite good, getting the high points of our trip in the same way I might have with my phone. Here’s a clip of my son driving a car.
Photo

But what was really interesting was the stuff I wouldn’t have consciously captured. I got dozens of pictures like these:
Photo
Photo
Photo

Aesthetically, these pictures aren’t masterworks. Emotionally, they’re on a higher plane. Clips caught moments of my kids goofing off and fighting in Disney’s endless lines, playing catch at home, dancing like adorable maniacs — moments too spontaneous or seemingly slight for me to have caught with my camera, but which will probably paint a more accurate and edifying picture of our lives in 30 years.

Regular readers of this column know that capturing moments of my kids’ childhood is an acute anxiety of mine; I’ve even wired my home with cameras to preserve a kind of reality show of life in my house.

But you don’t have to be as crazy as I am to share slivers of this fear — that your kids or your pets are constantly doing things you’ll later want to remember, but which smartphones often miss. An intelligent camera gets these times because it doesn’t ask you to break the moment to capture it.

But, obviously, setting up a camera that doesn’t need to be specifically triggered to take a picture is problematic. It raises the worry of spying — both that Google can spy on you, or that you can use it to spy on others.

Google addresses that creepiness in two ways. The device is mostly unconnected from the internet. It can take pictures without a connection, and it requires your phone for viewing or saving the clips. But even then, all its A.I. happens on the device, and it doesn’t even need you to have a Google account, the company said.

“We spent a lot of time thinking about privacy, and making sure this was a device people would actually want,” said Eva Snee, who heads Google’s research on how people interact with Clips. “What we learned was that cameras don’t creep people out when they’re used deliberately and the person is part of the process.”

Clips also raises the memory of other products in this vein, including Snap’s Spectacles and Google Glass, the search company’s failed attempt to get consumers to use glasses that can take photos.

To ensure that, Clips is designed to look like a camera. When it’s on, it flashes a white LED to signal that it could be recording. It also does not record audio, because that might have felt too much like spying.

Lighthouse, which I’ve also used for a few weeks, is meant to be an upgrade over the internet-connected home security cameras that have become popular. Those devices can be annoying because they freak out every time they spot any movement.

Lighthouse’s special trick is a camera system that can sense 3D space and learn and recognize faces — intelligence meant to avoid false alarms. It also has a nifty natural-language interface, so you can ask it straightforward questions: “What did the kids do when I was gone?” will show you clips of your kids when you were gone.

Lighthouse, which sells for $299 and requires a $10 monthly subscription, feels like a work in progress. It was mostly accurate in differentiating people in my house, but it was also tripped up into thinking I had an intruder by a mylar balloon floating around my living room.

The company is young and I expect its software to improve. I can see it being of genuine use to people who wonder what’s happening at home when they’re away. Want to know if your dog is jumping on your couch? Ask Lighthouse; it can recognize dogs jumping around couches, and will instantly show you the clip. (Well, it can get close. I don’t have a dog, so when I asked Lighthouse that question, it pulled up a clip of my kid kicking his stuffed tiger off the couch.)

But what if it’s your spouse, not your pooch, you’re worried about? I trust my wife, but for the sake of this column, I asked the device to show me any clips of her in the house with an unfamiliar person. There she was one evening, with the babysitter, whom Lighthouse hadn’t seen before.

It was a case of straight-up spying on my family. But it’s an obvious possibility with a camera that understands the world so well.

Alex Teichman, Lighthouse’s chief executive, said it could add safeguards against inter-family spying, for instance by restricting face identification only to unrecognized faces. He also pointed out that the system has numerous fine-grained privacy controls that allow you to turn off any recording when certain family members are present.

I found his response credible. Both Lighthouse and Clips are well-crafted against abuse. It should be noted that neither one allows for much more spying than we can already accomplish with smartphones; constant social surveillance is the norm in 2018.

But they are guides to the future. Tomorrow, all cameras will have their capabilities. And they won’t just watch you — they’ll understand, too.

Email: farhad.manjoo@nytimes.com; Twitter: @fmanjoo.