In some stores, sophisticated systems are tracking customers in almost every imaginable way, from recognizing their faces to gauging their age, their mood, and virtually gussying them up with makeup. The systems rarely ask for people’s permission, and for the most part they don’t have to. In our season 1 finale, we look at the explosion of AI and face recognition technologies in retail spaces, and what it means for the future of shopping.
- RetailNext CTO Arun Nair
- L’Oreal’s Technology Incubator Global VP Guive Balooch
- Modiface CEO Parham Aarabi
- Biometrics pioneer and Chairman of ID4Africa Joseph Atick
This episode was reported and produced by Jennifer Strong, Anthony Green, Tate Ryan-Mosley, Emma Cillekens and Karen Hao. We’re edited by Michael Reilly and Gideon Lichfield.
Strong: Retailers have been using face recognition and AI tracking technologies for years.
[Audio from Face First: What if you could stop retail crime before it happens by knowing the moment a shoplifter enters your store? And what if you could know about the presence of violent criminals before they act? With Face First you can stop crime before it starts.]
Strong: That’s one of the largest providers of this tech to retail stores. It detects faces, voices, objects and claims it can analyze behavior. But face recognition systems have a well-documented history of misidentifying women and people of color.
[Sound from 2019 Congressional hearing on facial recognition (Ocasio-Cortez): We have a technology that was created and designed by one demographic that is only mostly effective on that one demographic. And they’re trying to sell it and impose it on the entirety of the country?]
Strong: This is Representative Alexandria Ocasio-Cortez at a 2019 congressional hearing on facial recognition. Photo technologies work better on lighter skin. And datasets used by companies to train facial analysis systems are largely based on faces collected from the internet where content tends to skew white, male and western.
[Sound from 2019 Congressional hearing on facial recognition (Ocasio-Cortez): And do you think that this could exacerbate the already egregious, uh, inequalities in our, in our criminal justice system]
[Sound from 2019 Congressional hearing on facial recognition (Buolamwini): And It already is.]
Strong: Joy Buolamwini is an activist and computer scientist.
[Sound from 2019 Congressional hearing on facial recognition (Buolamwini): So, there’s a case with Mr. Bah, an 18-year-old African American man who was misidentified in Apple stores as a thief. And in fact, he was falsely arrested multiple times because of this kind of misidentification.
Strong: As awareness of these issues grows, more places are looking to put restrictions around its use such as in Portland, Oregon, which recently passed the most sweeping ban on face ID in the US.
[Sound from store in Portland, Oregon: please look into the camera for entry]
Strong: The ban takes effect in January and when it does that voice and camera will go away from places like this food store where the tech unlocks the door to late night shoppers. But use elsewhere is moving well beyond fighting crime (and is starting to play other retail roles) like remembering your past orders and payment details.
Miller: These face-based technologies, uhh artificial intelligence, machine vision allow us to see our customer in the offline world like amazon sees its customer in the online world. That allows us to create tailored experiences for the customer and also allows us to directly target that customer in new ways when they come back to the restaurant.
Strong: That’s the chairman of Cali Group, John Miller, its fast-food restaurant Caliburger tries out technologies it later markets to the entire industry. Other retailers use face recognition to know when VIP shoppers or celebrities are in their stores, not unlike this scene from the film Minority Report where as Tom Cruise strolls through a mall, his eyes are scanned and the ads address his character by name.
[Sound from Minority Report where voices address John Anderson in person]
Strong: The face measurements powering these applications can also be used for many other things besides just identifying someone. For example, some shopping malls use it to help set their store rents by counting how many people walk by, and using face data to gauge gender, age, and other demographics. Sometimes face recognition cameras are even hidden inside in mall directories. And inside stores, retailers use it to better understand what shoppers are interested in. It’s also embedded within shopping apps and store mirrors that let people try on anything from eyeglasses to makeup virtually.
I’m Jennifer Strong and this episode, we wrap up our first season (and our latest miniseries on face recognition) with a look at how it’s used to watch, understand and influence your shopping habits.
Strong: So I’m out front of what used to be the largest store in the world. This is Macy’s on 34th Street in Manhattan. The building fills an entire city block and in some ways it’s kind of the center of gravity for the holiday shopping season here as, among other things, the inspiration for one of New York’s most famous Christmas films, Miracle on 34th Street.
But the company may also have a history of using face recognition and a lawsuit was filed about that in Illinois which has a biometric privacy law requiring companies get permission before using it on customers. That suit alleges Macy’s is a client of ClearviewAI. We’ve had the founder on this show Hoan Ton-That and his product works by matching images, in this case of shoppers or shoplifters, against a database of perhaps billions of photos taken from social media posted by people who haven’t changed their settings to make the photos private just to their friends.
Now, New York City’s councilmembers just passed a biometrics measure here that if signed by the mayor will make retailers here also tell shoppers that face recognition is being used and perhaps what’s happening with that data. But you know it’s too soon to say what that might look like. I mean does walking as part of a big crowd of shoppers past a wall plaque that says face recognition is present, does that equal being informed, let alone giving consent? But I’m going to go inside with my producer, Anthony Green, and see if we can find totally different applications of face mapping to show you.
Several of these beauty counters have iPads that double as mirrors with augmented reality. We tried out three of them just one though asked for consent to analyze our faces. Two of the systems saw us just fine through our masks. The other didn’t recognize our faces at all.
I walked up to a mirror and it says my lighting is okay. Come closer until your face fills a circle. Apparently I have dark circles, uneven texture. irritation and redness and eyelines. At least we’re on the less side? I don’t know. Woah. Hey Anthony, you should see this. I wasn’t sure it was doing anything and now look in the mirror.
Strong: I don’t really have words for describing this, but it is so funny seeing myself this made up.
Green: Just kind of like glammed up.
Strong: Yeah. I’m like super glammed up. And literally all I was doing was looking in this mirror and then I looked down on an iPad and Holy, wow.
Green: This is working with your mask on.
Strong: This is with my mask on. And if I pull my mask down, I am made up everywhere.
Green: Oh yea.
Strong: Like glossed and all. Oh, look at you.
Strong: Okay, so Anthony just took a step over towards me and now he’s made up to the nines. Okay. These experiences are among the many many ways that face mapping can be applied.
But because they’re so controversial most brands simply don’t want to talk about it. And mostly, they don’t have to. There’s no national requirement that companies disclose the way they gather or use our biometric data even though we can imagine a not-so-distant future when that data becomes more important than any document we have. This personal data is likely to replace all of them proving who we are and what we own.
Most of what we know about the use of face recognition by retailers started in 2013 when it became public that identity company NEC had about a dozen brands and hotels as clients and they were using its face-reading technology to identify celebrities and other VIPs as they walked through their doors.
The following year Facebook announced it applied neural networks to face ID for the first time, making it work significantly better. And retailers, including Walmart, began testing it as a way to identify people caught shoplifting.
By 2016 fast food companies were experimenting with other use cases. One partnership, between KFC and the Chinese tech giant Baidu, recommended menu items to customers based on their age and mood as deemed by face scanning. These days it’s also possible to pay with your face, though so far, these applications haven’t really caught on. And so, wherever you shop, it’s reasonable to assume you might encounter some aspect of this technology and it could be combined with any number of other trackers. But it’s equally true that much of the tracking that’s done in retail stores using computer vision involves no facial recognition at all.
Nair: If you build a website today, there are a lot of tools available that you can use to give you data, like how many people visited your website, who they were, how they navigated your website and so on and for e-commerce sites the eventual purchase activity as well. And you can use all of this data to understand visitor behavior and optimize your site. We do the exact same thing, but for physical spaces. My name is Arun Nair. I’m the CTO and co-founder of RetailNext.
Strong: Their tracking software is deployed in offices, museums, even bowling alleys, but their primary market is retail. Ceiling cameras equipped with computer vision track customers as they journey through the store. It can guess basic demographic information like gender, who’s an employee—based on whether they go behind the register, even interactions between employees and customers.
Nair: We even have a prediction algorithm that will tell you based on historical information when your store is going to be busy later in the day, later in the week. And it is extremely helpful for staffing. So making sure that when you do expect a peak, that there are people there to assist shoppers and they’re not standing in queue and so on as well as you’re not always staffed when no one needs to be there.
Strong: He says the company is capable of determining what you’re looking at, but it doesn’t track eye gaze, expressions, or faces. And they don’t individually identify anyone.
Nair: We do not know who they are as individuals, and we specifically try not to as well. And in actually a lot of cases, once we get that information, we throw away the video or we blur the video.
Strong: When it comes to privacy, he believes systems using face recognition for identity should be opt-in
Nair: Consent is not just about like, Oh, I put my data out there so you can do what you want. I think consent is also about you know, we want you to do this so that we can do this in return for you. Are you okay with that?
Strong: But he admits that’s easier said than done.
Nair: It’s not easy to opt out of those things. And even if you opt out, the challenge is that let’s say, you say that, Hey, I want to opt out of my face. As a technology company, I still have to store a digitized version of your face to make sure I don’t track you again in the future cause next time I see your face, I need something to map against to say that, Oh, I should be dropping this person’s face. But then again, you know, in a weird way, I’m now storing a digitized version of your face, which. Again, it’s not really your face, but it is a representation of it.
Strong: And these challenges aren’t going away. Most tracking technologies aren’t regulated, and we simply don’t know how often things like face data gets captured. What is clear the retail industry is shifting to a world that’s centered around real-time analysis of customer experiences.
Nair: I think they’re going to see more and more of that moving forward, where there’s fewer purchases actually happening in these locations, but that’s kind of how you’re learning about the brand. [00:12:15] Almost like advertising, as well as kind of building a brand loyalty.
Strong: Tracking customers and their interaction with the store doesn’t just help retailers know what’s selling It also gives them insight on what customers want.
Nair: You introduce a new product. And you want to make sure that people are seeing that product. Our algorithms will tell you if people actually go into an area of the store and interact with a product and actually make a purchase afterwards.
Balooch: I think that it’s a combination of AI with physical objects that creates really an exciting moment in time. You know, you could never really try a trend and then actually dispense it. That wasn’t possible ever. But now because of AI, we are able to really go through trends really quickly. We’re able to curate trends, we’re able to give people what they desire. My name is Guive Balooch and I run the global technology incubator at L’Oreal. I’ve been at the company for 15 years and my job is to find the intersection between beauty and technology.
Strong: L’Oreal is the world’s largest cosmetics company with Estee Lauder, Maybelline, Garnier and countless other consumer brands under its corporate umbrella.
Balooch: We started about eight years ago with an augmented reality app called makeup genius. That was the world’s first virtual try-on. And since then we’ve launched projects around personalized beauty like skincare personalization, foundation personalization. We’ve launched a UV sensor at the Apple store that’s a wearable that has no battery and can measure your UV exposure. And now we’re, we’re moving more and more towards mass personalization and finding ways to combine technologies like AR and AI to create new physical objects that can be magical for beauty consumers and hopefully delight our users.
Strong: And this is harder than it might sound. Designing experiences that let customers try on makeup in augmented reality presents huge technical challenges for face detection.
Balooch: You need to detect where the eye is and where the eyebrow is. And it has to be at a level of accuracy that when the product’s on there, it doesn’t look like it’s not exactly on your lip. And it’s, it’s funny because I come from an academic background with a PhD. So I didn’t realize how complicated that specific part of this technology is. I thought, “Oh, it’s okay. We’ll just get the software. It will be easy. We’ll just make it work.” But it turns out no, it’s really complicated because people’s lips can vary in shape, the color between your skin tone and your lip can also be very different. And so you need to have an algorithm that can detect it and make sure it works on people from very light to very dark skin.
Strong: And he says one of the largest impacts of AI in the beauty market could be more inclusivity—something the industry has long struggled with.
Balooch: I’m under this, you know, strong belief that inclusivity is the future of beauty and inclusivity means that every human being has the right to have a product that is what they need for themselves and to showcase to the world how they want to be showcased. And I think that only through things like AI and tech, will we be able to reach that level of personal relationship with people’s desires for their beauty habits.
Strong: Those habits are shaped around our skin. And skin tone has historically been one of the hardest technical and cultural challenges.
Balooch: We launched this project called which is this foundation blender. And when I first started this project, I thought it was going to be very simple because when I went to Home Depot umm I’m not really a handyman, but I went with my, my dad a lot to Home Depot and he would buy paint. He would match the paint and they would just make the paint right there. And I said, okay, it’s that easy? So when we first started the project, we realized, okay, you know, you just take a skin tone from a piece of, you know, a paper and you can just match the foundation. And I realized later that our skin is not like a wall, it’s biological tissue that changes depending on what kind of skin tone you have.
Strong: In short, the algorithm didn’t work.
Balooch: And so we had to stop and spend another six months to improve it. First we did that with a little device that kind of measures your skin tone, using a physical object, because your skin tone is hard to measure if you don’t actually touch the skin cause the light can change the color of your skin. And so depending on if you’re outside or if you’re inside, you could have a big difference in the measurement. But not anymore. Thanks to AI, I think more and more with AI, we’re going to be able to get accurate measurements. We have to test them and make sure that they work as well as objects. But once we get to a point, when we think we’re getting close to that, then you can solve some really, really big challenges. And in foundation, 50% of women can’t find the right shade of foundation. And there’s no way that the number of products on the shelf will ever solve that because you will always have more skin tones in the world than products you can put on the shelf.
Strong: And the future could open up a whole new class of personalized beauty tools.
Balooch: We can make objects that are, you know, not huge–handheld–and can do incredible things. Like in the future, you could imagine that you can dispense eyeshadow on your eyelid automatically just through detecting the face and being able to have an object that could dispense it.
Strong: To build that future, L’Oreal acquired a company called Modiface which makes augmented reality tools for more than 70 of the world’s top beauty brands.
Aarabi: One big step that happened a few years ago was going from photos to live video simulation. Really hard feat technologically, but really impactful on the consumer experience. Instead of having to take a photo and upload it, they could see a live video.
Strong: Parham Aarabi is the Founder and CEO of Modiface.
Aarabi: The next big step that I see that I’m really excited about is a combination of AI understanding of the face, along with our simulation. So not only telling you, okay, so you choose a lipstick and this is what it looks like, but saying, because you chose this lipstick and because your, you know, you have blue eyes, we believe this eye shadow might match it the best.
Strong: His background is in face and lip tracking.
Aarabi: And so we had created this sample demo where you could track someone’s lips and swap the lips with a celebrity, for example. My co-founder had the idea that before we do this, we should actually apply some changes on the, on the skin. And so it was really the combination of these two ideas that became the foundation of Modiface.
Strong: The beauty industry thrives on the in-person shopping experience. And even though e-commerce sales have long been on the rise this sector has been a lot slower than others. For context, the top ecommerce seller in beauty of 2018 was shampoo. But the pandemic is speeding things up. Online sales at beauty giant Sephora jumped 30 percent in the U.S. this year. And it’s also partnered with Modiface to develop an app that acts as a virtual store, complete with product tutorials and an augmented reality beauty counter.
Aarabi: You see a try-on button, you press that, and a window opens up. You see your own video in that window, but with different virtual products being shown.
Strong: And building consumer trust in these simulated products means engineering an experience as seamless as looking in a mirror.
Aarabi: If someone actually tries on a lipstick and a hair color and then videotapes themselves versus using our technology and then having a virtual simulation of those products, the two should be indistinguishable. The lag, within the simulation being applied versus when you’re looking at your face and you’re seeing movements needs to be not apparent to the user. And so these are huge challenges. One is of realism. You don’t want the eyeliner to be flickering on someone’s eyes and the second is to do it so fast that on a website in live video, you don’t notice any lag. So these are major, major challenges.
Strong: And it’s more than just cosmetics. Elements of face detection are increasingly used in medicine to diagnose disease. And he believes in future their products will detect all kinds of skin disorders.
Aarabi: So we’ve been pushing on this skin assessment, um, direction by looking at someone’s image. And based on that, knowing what skin care products are best for them, and more, the more we do this and the more that better we train our AI systems, we find that they’re increasing in the level of accuracy matching that of dermatologists. And I think if you follow that line, that this AI, that can actually not replace dermatologists, but really helped them as.. an objective tool that can look at someone’s face and make recommendations.
Strong: It feels like there’s more awareness of face recognition of its risks, immaturies and biases but also its increased presence in our lives and just raw potential. To me, it seems like we’ve just scratched the surface – in this messy digital race to something different and big. And it got me wondering how might one of its inventors feel about all this?
Atick: I started working on the human brain about a year after I graduated and made together with my, collaborators made some fundamental breakthroughs, which led to the creation of a field called the biometric industry and the first commercially viable face recognition. That’s why people refer to me as a founding father of face recognition and the biometric industry.
Strong: That’s Dr. Joseph Atick. He developed one of the first face recognition algorithms back in 1994.
Atick: The algorithm for how a human brain would recognize familiar faces became clear while we’re doing mathematical research at the Institute for advanced study in Princeton.
Strong: But the technology needed to capture those faces wasn’t yet in everyone’s pockets.
Atick: At the time, computers did not have cameras. Phones that had cameras did not exist. We had to build the eyes for the brain. We had a brain, we thought we knew how the brain would analyze signals, but we did not have the eyes that would get the information and the visual signal to the brain.
Strong: Webcams came along in the 90s and computers with video capabilities arrived on the market a few years after.
Atick: And that was an exciting time because all of a sudden the brain that we had built had finally the pair of eyes that would be necessary to, to see.
Strong: This was the breakthrough he and his team needed to bring their concept to life. So they started coding.
Atick: it was a long period of months of programming and failure and programming and failure
Strong: But eventually…
Atick: And one night, early morning, actually, we had just finalized, um, a version of the algorithm. We submitted the, source code for compilation in order to get a run code. And we stepped out, I stepped out to go to the washroom. And then when I stepped back into the room it spotted my face, extracted it from the background and it pronounced “I see Joseph”. And that was the moment where the hair on the back–I felt like something had happened. We were a witness. And I started, um, to call on the other people who were still in the lab and each one of them, they would come into the room. And I would say, it would say, I see Norman. I would see Paul, I would see Joseph. And we would sort of take turns running around the room just to see how many it can spot in the room.
Strong: They had built something that had never been built before. Months of math and coding and long nights seemed to be paying off. But within a few years that excitement turned to concern.
Atick: My, my concern about the technology that I helped create and invent started very quickly after I had invented it. I saw a future where our privacy would be at jeopardy if we did not put in place protection measures to prevent the abuse of this powerful technology.
Strong: And he wanted to do something about it.
Atick: So in 1998, I lobbied the industry and I said, we need to put together principles for responsible use. And this is where an organization called IBIA was born in 1998 as an industry association to promote responsible use. Um, and so I was the founder of that, that organization. And I felt good for a while because I felt we have gotten it right. I felt we’ve invented the technology, but then we put in place a responsible use code to be followed by whatever is the implementation. However, that code did not live the test of time. And the reason behind it is we did not anticipate the emergence of social media.
Strong: Face recognition relies on a database of images. The size, quality, and privacy conditions of this database is largely what determines how safe or intrusive the technology is. In 1998, Atick built his databases by manually scanning thousands of pictures and tagging them with names. It was tedious and limiting in size.
Atick: We have allowed the beast out of the bag by feeding it billions of faces and helping it by tagging ourselves. We are now in a world where machine learning is now allowing for the emergence of over 400 different algorithms of face recognition in the world. Therefore, any hope of controlling and requiring everybody to be, to be responsible in their use of face recognition is difficult.
Strong: And this is made worse by scraping, where a database is created by scanning the entire internet for public photos.
Atick: And so I began to panic in 2011, and I wrote an op-ed article saying it is time to press the panic button because the world is heading in a direction where face recognition is going to be omnipresent and faces are going to be everywhere available in, in, in databases. Computing power is becoming very, very massive to the point that we could potentially recognize billions of people. And at the time people said I was an alarmist, but they’re realizing that it’s exactly what’s happening today.
Strong: So in a way, he’s kind of lobbying against his own invention even though he still uses biometrics to help build things he believes might benefit the greater good like digital ID for people in developing nations.
Atick: The chilling effect is something that is unforgivable. If I cannot go outside in the street, because I believe somebody’s using an iPhone, could take a picture of me and connect me to my online profile and, this online and offline connection is, is a dangerous thing. And it’s happening right now.
Strong: And he thinks we urgently need some legal ground rules.
Atick: And so it’s no longer a technological issue. We cannot contain this powerful technology through technology. There has to be some sort of legal frameworks.
Strong: The way he sees it, the technological edge will keep pushing forward—with AI at the forefront. But the people building and using it? They’re at the center.
Atick: I believe there has to be some harmony between what technology can do for us and helps us live with dignity and have easier lives and connect with the people we love, but at the same time, it has to be within what our morals and our expectations as human beings allow it to be.
Strong: In other words, once again… it seems up to us. This episode was reported and produced by me, Anthony Green, Emma Cillekens, Tate Ryan-Mosley and Karen Hao. We’re edited by Michael Reilly and Gideon Lichfield. Thanks too to Kate Kaye with the Banned in PDX podcast. That’s it for season one. Thanks so much for choosing to spend your time with us. We’ll meet you back here in the new year until then happy holidays and… Thanks for listening, I’m Jennifer Strong.
How a tiny media company is helping people get vaccinated
More than 132 million people in the US have received at least one dose of a covid-19 vaccine, and as of this week, all Americans over 16 are eligible.
But while the US has vaccinated more people than any other country in the world, vulnerable people are still falling through the cracks. Those most affected include people who don’t speak English, people who aren’t internet-savvy, and shift workers who don’t have the time or computer access to book their own slots. In many places, community leaders, volunteers, and even news outlets have stepped in to help.
One of those groups is Epicenter-NYC, a media company that was founded during the pandemic to help neighbors navigate covid-19. Based in the Queens neighborhood of Jackson Heights, which was particularly hard hit by the virus, the organization publishes a newsletter on education, business, and other local news.
But Epicenter-NYC has gone further and actually booked more than 4,600 vaccine appointments for people in New York and beyond. People who want to get vaccinated can contact the organization—either through an intake form, a hotline, a text, or an email—for help setting up an appointment.
Throughout the vaccine rollout, the group has also been documenting and sharing what it has learned about the process with a large audience of newsletter readers.
We spoke with S. Mitra Kalita, the publisher of Epicenter-NYC, who was previously a senior vice president at CNN Digital and is also the cofounder and CEO of URL Media, a network for news outlets covering communities of color.
This interview has been condensed and edited for clarity.
Q: How did you start setting people up with vaccine appointments?
A: It began with two areas of outreach. First, when I had to register my own parents for a vaccine and found the process to be pretty confusing, I immediately wondered how well elderly residents, their friends and neighbors, manage this process. I just started messaging them.
The second was when a restaurant [from our small business spotlight program] reached out and said, “Do you guys know how to get vaccines for our restaurant workers?” Because I had been navigating some of this for the elderly, I started to help the restaurant workers. There started to be a similar network effect. One of the workers at this restaurant has a boyfriend who is a taxi driver; when I helped her, she asked if I could help her boyfriend; then the boyfriend texted me with some of his friends; and it kept spreading in that way.
Q: How is Epicenter-NYC filling gaps in vaccine distribution right now? What is your process like, and who are you helping?
A: We’ve had between 200 and 250 people reach out to volunteer. The outreach efforts range from putting up fliers, doing translations, and calling people to literally booking the appointments.
I don’t care if you’re a Bangladeshi taxi driver in Queens and your cousin is in New Jersey. We’re going to help both of you. A woman on the Upper East Side who’s 102 years old who is homebound and needs a visit is absolutely going to get Epicenter’s help.
What we’re doing now is continuing the route of connecting people to each other and opportunities. There’s a lot of matchmaking going on. We can sort through a list of about 7,500 to 8,000 people who said they need help, and then find places in proximity. We’ve become this wonderful marriage—a centralized operation that also embraces decentralized solutions.
Q: We know that vaccination rates lag in many communities that were hit the hardest. Why is that? What issues and barriers are people experiencing?
A: Just before the latest Johnson & Johnson pause announcement, I said, “We’re at a point where everybody remaining is a special case.”
I think we’ve leapfrogged to vaccine hesitancy without solving for vaccine access. We don’t see a lot of hesitancy, but we do see a lot of concerns over some issues. Number one would be scheduling. We’re dealing with populations that are working two, maybe three jobs, and when they say “I have this window on Sunday at 3 p.m. until maybe 6 p.m., when my next shift starts,” they really mean that’s the only window.
Q: People have been asked to prove who they are, where they work, and where they live in order to qualify for a vaccine. This was especially true when eligibility was more limited. How did you help people face barriers around getting the documents they needed?
A: New York State has been explicit in saying you can still get a vaccine even if you are undocumented. But that messaging doesn’t really match the on-the-ground reality.
Police in Ogden, Utah and small cities around the US are using these surveillance technologies
One afternoon, I accompanied Heather West, the detective who’d been perusing gray pickups in the license-plate database, and Josh Terry, the analyst who’d spotted the kidnapper with the Cowboys jacket, to fly a drone over a park abutting a city-owned golf course on the edge of town. West was at the controls; Terry followed the drone’s path in the sky and maintained “situational awareness” for the crew; another detective focused on the iPad showing what the drone was seeing, as opposed to where and how it was flying.
Of all the gadgets under the hood at the real time crime center, drones may well be the most tightly regulated, subject to safety (but not privacy) regulations and review by the Federal Aviation Administration. In Ogden, neighbor to a large Air Force base, these rules are compounded by flight restrictions covering most of the city. The police department had to obtain waivers to get its drones off the ground; it took two years to develop policies and get the necessary approvals to start making flights.
The police department purchased its drones with a mind to managing large public events or complex incidents like hostage situations. But, as Dave Weloth soon found, “the more we use our drones, the more use cases we find.” At the real time crime center, Terry, who has a master’s in geographic information technology, had given me a tour of the city with images gathered on recent drone flights, clicking through to cloud-shaped splotches, assembled from the drone’s composite photographs, that dotted the map of Ogden.
Above 21st Street and Washington, he zoomed in on the site of a fatal crash caused by a motorcycle running a red light. A bloody sheet covered the driver’s body, legs splayed on the pavement, surrounded by a ring of fire trucks. Within minutes, the drone’s cameras had scanned the scene and created a 3D model accurate to a centimeter, replacing the complex choreography of place markers and fixed cameras on the ground that sometimes leave major intersections closed for hours after a deadly collision.
When the region was hit by a powerful windstorm last September, Terry flew a drone over massive piles of downed trees and brush collected by the city. When county officials saw the resulting volumetric analysis—12,938 cubic yards—that would be submitted as part of a claim to the Federal Emergency Management Agency, they asked the police department to perform the same service for two neighboring towns. Ogden drones have also been used to pinpoint hot spots after wildland fires, locate missing persons, and fly “overwatch” for SWAT team raids.
This flight was more routine. When I pulled into the parking lot, two officers from Ogden’s community policing unit looked on as West steered the craft over a dense stand of Gambel oak and then hovered over a triangular log fort on a hillside a couple of hundred yards away. Though they’d never encountered people on drone sweeps through the area, trash and makeshift structures were commonplace. Once the RTCC pinpointed the location of any encampments, the community service officers would go in on foot to get a closer look. “We get a lot of positive feedback from runners, hikers,” one officer explained. After one recent visit to a camp near a pond on 21st Street, he and the county social service workers who accompanied him found housing for two people they’d met there. When clearing camps, police also “try and connect [people] with services they need,” Weloth said. The department recently hired a full-time homeless outreach coordinator to help. “We can’t police ourselves out of this problem,” he said, comparing the department’s efforts to keep new camps from springing up to “pushing water uphill.”
NASA has flown its Ingenuity drone helicopter on Mars for the first time
The news: NASA has flown an aircraft on another planet for the first time. On Monday, April 19, Ingenuity, a 1.8-kilogram drone helicopter, took off from the surface of Mars, flew up about three meters, then swiveled and hovered for 40 seconds. The historic moment was livestreamed on YouTube, and Ingenuity captured the photo above with one of its two cameras. “We can now say that human beings have flown a rotorcraft on another planet,” said MiMi Aung, the Ingenuity Mars Helicopter project manager at NASA’s Jet Propulsion Laboratory, at a press conference. “We, together, flew at Mars, and we, together, now have our Wright brothers moment,” she added, referring to the first powered airplane flight on Earth in 1903.
In fact, Ingenuity carries a tribute to that famous flight: a postage-stamp-size piece of material from the Wright brothers’ plane tucked beneath its solar panel. (The Apollo crew also took a splinter of wood from the Wright Flyer, as it was named, to the moon in 1969.)
The details: The flight was a significant technical challenge, thanks to Mars’s bone-chilling temperatures (nights can drop down to -130 °F/-90 °C) and its incredibly thin atmosphere—just 1% the density of Earth’s. That meant Ingenuity had to be light, with rotor blades that were bigger and faster than would be needed to achieve liftoff on Earth (although the gravity on Mars, which is only about one-third of Earth’s, worked in its favor). The flight had originally been scheduled to take place on April 11 but was delayed by software issues.
Why it’s significant: Beyond being a significant milestone for Mars exploration, the flight will also pave the way for engineers to think about new ways to explore other planets. Future drone helicopters could help rovers or even astronauts by scoping out locations, exploring inaccessible areas, and capturing images. Ingenuity will also help inform the design of Dragonfly, a car-size drone that NASA is planning to send to Saturn’s moon Titan in 2027.
What’s next: In the next few weeks, Ingenuity will conduct four more flights, each lasting up to 90 seconds. Each one is designed to further push the limits of Ingenuity’s capabilities. Ingenuity is only designed to last for 30 Martian days, and is expected to stop functioning around May 4. Its final resting place will be in the Jezero Crater as NASA moves on to the main focus of its mission: getting the Perseverance rover to study Mars for evidence of life.