Ten years. Two countries. Multiple redesigns. Some US $80 million invested. And, finally, Zero Zero Robotics has a product it says is ready for consumers, not just robotics hobbyists—the HoverAir X1. The company has sold several hundred thousand flying cameras since the HoverAir X1 started shipping last year. It hasn’t gotten the millions of units into consumer hands—or flying above them—that its founders would like to see, but it’s a start.
“It’s been like a 10-year-long Ph.D. project,” says Zero Zero founder and CEO Meng Qiu Wang. “The thesis topic hasn’t changed. In 2014 I looked at my cell phone and thought that if I could throw away the parts I don’t need—like the screen—and add some sensors, I could build a tiny robot.”
I first spoke to Wang in early 2016, when Zero Zero came out of stealth with its version of a flying camera—at $600. Wang had been working on the project for two years. He started the project in Silicon Valley, where he and cofounder Tony Zhang were finishing up Ph.D.s in computer science at Stanford University. Then the two decamped for China, where development costs are far less.
Flying cameras were a hot topic at the time; startup Lily Robotics demonstrated a $500 flying camera in mid-2015 (and was later charged with fraud for faking its demo video), and in March of 2016 drone-maker DJI introduced a drone with autonomous flying and tracking capabilities that turned it into much the same type of flying camera that Wang envisioned, albeit at the high price of $1400.
Wang aimed to make his flying camera cheaper and easier to use than these competitors by relying on image processing for navigation—no altimeter, no GPS. In this approach, which has changed little since the first design, one camera looks at the ground and algorithms follow the camera’s motion to navigate. Another camera looks out ahead, using facial and body recognition to track a single subject.
The current version, at $349, does what Wang had envisioned, which is, he told me, “to turn the camera into a cameraman.” But, he points out, the hardware and software, and particularly the user interface, changed a lot. The size and weight have been cut in half; it’s just 125 grams. This version uses a different and more powerful chipset, and the controls are on board; while you can select modes from a smart phone app, you don’t have to.
I can verify that it is cute (about the size of a paperback book), lightweight, and extremely easy to use. I’ve never flown a standard drone without help or crashing but had no problem sending the HoverAir up to follow me down the street and then land on my hand.
It isn’t perfect. It can’t fly over water—the movement of the water confuses the algorithms that judge speed through video images of the ground. And it only tracks people; though many would like it to track their pets, Wang says animals behave erratically, diving into bushes or other places the camera can’t follow. Since the autonomous navigation algorithms rely on the person being filmed to avoid objects and simply follows that path, such dives tend to cause the drone to crash.
Since we last spoke eight years ago, Wang has been through the highs and lows of the startup rollercoaster, turning to contract engineering for a while to keep his company alive. He’s become philosophical about much of the experience.
Here’s what he had to say.
We last spoke in 2016. Tell me how you’ve changed.
Meng Qiu Wang: When I got out of Stanford in 2014 and started the company with Tony [Zhang], I was eager and hungry and hasty and I thought I was ready. But retrospectively, I wasn’t ready to start a company. I was chasing fame and money, and excitement.
Now I’m 42, I have a daughter—everything seems more meaningful now. I’m not a Buddhist, but I have a lot of Zen in my philosophy now.
I was trying so hard to flip the page to see the next chapter of my life, but now I realize, there is no next chapter, flipping the page itself is life.
You were moving really fast in 2016 and 2017. What happened during that time?
Wang: After coming out of stealth, we ramped up from 60 to 140 people planning to take this product into mass production. We got a crazy amount of media attention—covered by 2,200 media outlets. We went to CES, and it seemed like we collected every trophy there was there.
And then Apple came to us, inviting us to retail at all the Apple stores. This was a big deal; I think we were the first third party robotic product to do live demos in Apple stores. We produced about 50,000 units, bringing in about $15 million in revenue in six months.
Then a giant company made us a generous offer and we took it. But it didn’t work out. It was a certainly lesson learned for us. I can’t say more about that, but at this point if I walk down the street and I see a box of pizza, I would not try to open it; there really is no free lunch.
This early version of the Hover flying camera generated a lot of initial excitement, but never fully took off.Zero Zero Robotics
How did you survive after that deal fell apart?
Wang: We went from 150 to about 50 people and turned to contract engineering. We worked with toy drone companies, with some industrial product companies. We built computer vision systems for larger drones. We did almost four years of contract work.
But you kept working on flying cameras and launched a Kickstarter campaign in 2018. What happened to that product?
Wang: It didn’t go well. The technology wasn’t really there. We filled some orders and refunded ones that we couldn’t fill because we couldn’t get the remote controller to work.
We really didn’t have enough resources to create a new product for a new product category, a flying camera, to educate the market.
So we decided to build a more conventional drone—our V-Coptr, a V-shaped bi-copter with only two propellers—to compete against DJI. We didn’t know how hard it would be. We worked on it for four years. Key engineers left out of total dismay, they lost faith, they lost hope.
We came so close to going bankrupt so many times—at least six times in 10 years I thought I wasn’t going to be able to make payroll for the next month, but each time I got super lucky with something random happening. I never missed paying one dime—not because of my abilities, just because of luck.
We still have a relatively healthy chunk of the team, though. And this summer my first ever software engineer is coming back. The people are the biggest wealth that we’ve collected over the years. The people who are still with us are not here for money or for success. We just realized along the way that we enjoy working with each other on impossible problems.
When we talked in 2016, you envisioned the flying camera as the first in a long line of personal robotics products. Is that still your goal?
Wang: In terms of short-term strategy, we are focusing 100 percent on the flying camera. I think about other things, but I’m not going to say I have an AI hardware company, though we do use AI. After 10 years I’ve given up on talking about that.
Do you still think there’s a big market for a flying camera?
Wang: I think flying cameras have the potential to become the second home robot [the first being the robotic vacuum] that can enter tens of millions of homes.
As the world’s 5G rollout continues with its predictable fits and starts, the cellular technology is also starting to move into a space already dominated by another wireless tech: Wi-Fi. Private 5G networks—in which a person or company sets up their own facility-wide cellular network—are today finding applications where Wi-Fi was once the only viable game in town. This month, the Newbury, England–based telecom company Vodafone is releasing a Raspberry Pi–based private 5G base station that it is now being aimed at developers, who might then jump-start a wave of private 5G innovation.
“The Raspberry Pi is the most affordable CPU[-based] computer that you can get,” says Santiago Tenorio, network architecture director at Vodafone. “Which means that what we build, in essence, has a similar bill of materials as a good quality Wi-Fi router.”
“In a Raspberry Pi—in this case, a Raspberry Pi 4 is what we used—then you can be sure you can run that anywhere, because it’s the tiniest processor that you can have,” Tenorio says.
Santiago Tenorio holds one of Lime Microsystems’ private 5G base-station kits.Vodafone
Private 5G for Drones and Bakeries
There are a range of reasons, Tenorio says, why someone might want their own private 5G network. At the moment, the scenarios mostly concern companies and organizations—although individual expert users could still be drawn to, for instance, 5G’s relatively low latency and network flexibility.
Tenorio highlighted security and mobility as two big selling points for private 5G.
A commercial storefront business, for instance, might be attracted to the extra security protections that a SIM card can provide compared to password-based wireless network security. Because each SIM card contains its own unique identifier and encryption keys, thereby also enabling a network to be able to recognize and authorize each individual connection, Tenorio says private 5G network security is a considerable selling point.
Plus, Tenorio says, it’s simpler for customers to access. Envisioning a use case of a bakery with its own privately deployed 5G network, he says, “You don’t need a password. You don’t need a conversation [with a clerk behind a counter] or a QR code. You simply walk into the bakery, and you are into the bakery’s network.”
As to mobility, Tenorio suggests one emergency relief and rescue application that might rely on the presence of a nearby 5G station that causes devices in its range to ping.
Setting up a private 5G base station on a drone, Tenorio says, would enable that drone to fly over a disaster area and, via its airborne network, send a challenge signal to all devices in its coverage area to report in. Any device receiving that signal with a compatible SIM card then responds with its unique identification information.
“Then any phone would try to register,” Tenorio says. “And then you would know if there is someone [there].”
Not only that, but because the ping would be from a device with a SIM card, the private 5G rescue drone in the above scenario could potentially provide crucial information about each individual, just based on the device’s identifier alone. And that user-identifying feature of private 5G isn’t exactly irrelevant to a bakery owner—or to any other commercial customer—either, Tenorio says.
“If you are a bakery,” he says, “You could actually know who your customers are, because anyone walking into the bakery would register on your network and would leave their [International Mobile Subscriber Identity].”
Winning the Lag Race
According to Christian Wietfeld, professor of electrical engineering at the Technical University of Dortmund in Germany, private 5G networks also bring the possibility of less lag. His team has tested private 5G deployments—although, Wietfeld says that they haven’t yet tested the present Vodafone/Lime Microsystem base station—and have found private 5G to provide reliably better connectivity.
“The additional cost and effort to operate a private 5G network pays off in lower downtimes of production and less delays in delivery of goods,” Wietfeld says. “Also, for safety-critical use cases such as on-campus teleoperated driving, private 5G networks provide the necessary reliability and predictability of performance.”
For Lime Networks, according to the company’s CEO and founder Ebrahim Bushehri, the challenge comes in developing a private 5G base station that maximized versatility and openness to whatever kinds of applications developers could envision—while still being reasonably inexpensive and retaining a low-power envelope.
“The solution had to be ultraportable and with an optional battery pack which could be mounted on drones and autonomous robots, for remote and tactical deployments, such as emergency-response scenarios and temporary events,” Bushehri says.
Meanwhile, the crowdfunding behind the device’s rollout, via the website Crowd Supply, allows both companies to keep tabs on the kinds of applications the developer community is envisioning for this technology, he says.
“Crowdfunding,” Bushehri says, “Is one of the key indicators of community interest and engagement. Hence the reason for launching the campaign on Crowd Supply to get feedback from early adopters.”
Insects have long been an inspiration for robots. The insect world is full of things that are tiny, fully autonomous, highly mobile, energy efficient, multimodal, self-repairing, and I could go on and on but you get the idea—insects are both an inspiration and a source of frustration to roboticists because it’s so hard to get robots to have anywhere close to insect capability.
We’re definitely making progress, though. In a paper published last month in IEEE Robotics and Automation Letters, roboticists from Shanghai Jong Tong University demonstrated the most buglike robotic bug I think I’ve ever seen.
Okay so it may not look the most buglike, but it can do many very buggy bug things, including crawling, taking off horizontally, flying around (with six degrees of freedom control), hovering, landing, and self-righting if necessary. JT-fly weighs about 35 grams and has a wingspan of 33 centimeters, using four wings at once to fly at up to 5 meters per second and six legs to scurry at 0.3 m/s. Its 380 milliampere-hour battery powers it for an actually somewhat useful 8-ish minutes of flying and about 60 minutes of crawling.
While that amount of endurance may not sound like a lot, robots like these aren’t necessarily intended to be moving continuously. Rather, they move a little bit, find a nice safe perch, and then do some sensing or whatever until you ask them to move to a new spot. Ideally, most of that movement would be crawling, but having the option to fly makes JT-fly exponentially more useful.
Or, potentially more useful, because obviously this is still very much a research project. It does seem like there’s a bunch more optimization that could be done here. For example, JT-fly uses completely separate systems for flying and crawling, with two motors powering the legs and two additional motors powering the wings—plus two wing servos for control. There’s currently a limited amount of onboard autonomy, with an inertial measurement unit, barometer, and wireless communication, but otherwise not much in the way of useful payload.
Insects are both an inspiration and a source of frustration to roboticists because it’s so hard to get robots to have anywhere close to insect capability.
It won’t surprise you to learn that the researchers have disaster-relief applications in mind for this robot, suggesting that “after natural disasters such as earthquakes and mudslides, roads and buildings will be severely damaged, and in these scenarios, JT-fly can rely on its flight ability to quickly deploy into the mission area.” One day, robots like these will actually be deployed for disaster relief, and although that day is not today, we’re just a little bit closer than we were before.
Shooting fireworks out of a helicopter sounds fun. Shooting fireworks out of a helicopter at a Lamborghini sports car sounds really fun, especially if everyone on the helicopter and everyone in the Lamborghini consents. Alex Choi, a YouTube and Instagram vlogger in California, produced a video of him and his crew doing just that. But he forgot to ask one important group for permission: the federal government.
Earlier this week, the feds indicted Choi for "causing the placement of explosive or incendiary device on an aircraft," a crime with a maximum penalty of 10 years in prison. The indictment also revealed that the Federal Aviation Administration (FAA) had revoked the license from Choi's helicopter pilot in January 2024 for flying less than 500 feet from people, failing to display the helicopter's registration number, and creating "a hazard to persons or property" without the necessary FAA waivers.
By all accounts, the only danger was to people directly involved in the video, which has since been removed from Choi's YouTube and Instagram accounts. (Clips of the stunt are still available elsewhere.) Choi and his crew filmed the stunt at El Mirage dry lake bed, an off-roading recreation area miles away from any town. The indictment quotes Choi talking about his "crazy stupid ideas" and one of his crew members saying that the fireworks are "so loud; it's actually terrifying," which only makes the video sound cooler.
The FAA moved very quickly when it caught wind of the stunt. Choi posted the video on the Fourth of July last year. On July 18, an FAA inspector interviewed the person who transported cars for Choi. A few days later, the FAA tracked down the helicopter pilot and a Bureau of Land Management agent went out to the dry lake to photograph Choi's tire tracks. Since the lake bed is federal land, the indictment notes, Choi should have gotten federal permission.
Soon after the FAA interrogations began, Choi texted an associate that the FAA inspector "has a personal issue with my helicopter pilot friend and every time i do a shoot with him, tries to get more information about him so he can go after him," according to the indictment.
The Department of Transportation's Office of Inspector General then decided to charge Choi with a crime. The law against taking an explosive on board an aircraft clearly seems to be aimed at would-be bombers, but the feds argue that it applies to firing explosives out of an aircraft as well.
The case against Choi parallels the case of Austin Haughwout almost a decade ago. In 2015, when consumer drone technology was still in its infancy, the teenage Haughwout filmed himself flying a drone with a pistol attached and firing into the woods. The 14-second video, titled "Flying Gun," caused a national media panic about the danger of armed drones. Haughwout also posted a video of himself roasting meat with a drone-mounted flamethrower.
The FAA subpoenaed Haughwout and his father because the videos showed potentially unsafe piloting of an aircraft. The Haughwout family fought the subpoena in court, arguing that drones are not "aircraft" within the FAA's jurisdiction. (Their lawyer compared the situation to the FAA regulating baseballs, paper airplanes, or birthday balloons.) A district court ruled in favor of the subpoena, and although Haughwout was not charged with an aviation crime, the case became a key precedent for the FAA's ability to regulate drones.
Since then, the FAA has scoured social media for potential drone violations. Earlier this year, a federal court banned Philadelphia YouTuber Michael DiCiurcio from flying drones and fined him $182,000 for violating FAA rules. DiCiurcio had gotten famous for making slapstick videos of himself fighting birds, buzzing fishermen, and crashing into himself with his drone, all while narrating in a thick South Philly accent.
Last year, aviation vlogger Joe Costanza had a friend follow his small Piper Cub airplane down a private runway with a drone. When Costanza posted the video to a Facebook group—and joked that "the pilot knew that the drone was there because I was flying both at the same time"—he was contacted by an FAA inspector. In the end, the FAA did not press any charges, but Constanza took to YouTube to complain about the investigation.
"You know, no matter how stupid the complaint is or how out of the ordinary it is, we have to investigate every single complaint that comes out way," the inspector said, according to Constanza.
Last week, Reasonreported on the rising trend of Colorado police departments increasingly using aerial drones as first responders to certain 911 calls.
A new investigation out this week reveals how such a system could work in practice, with startling implications for privacy and civil liberties.
In WIRED, Dhruv Mehrotra and Jesse Marx write about Chula Vista, a town in southern California roughly equidistant between San Diego and Tijuana. In 2018, the Chula Vista Police Department (CVPD) launched the Drone as First Responder (DFR) program, allowing 911 operators to deploy drones either in lieu of or in addition to uniformed officers—the first U.S. city to do so.
DFR "is not a replacement for officers, it's an enhancement," Police Chief Roxana Kennedy told KPBS at the time. In the program's first week, drones responded to 30 calls and led to three arrests, including a domestic violence case in which a man suspected of stabbing a woman fled back to a homeless encampment and a pursuing drone led police to his location. The program was initially limited to within one mile of the police station, but it expanded over time before receiving federal authorization to operate citywide in March 2021.
In the nearly six years since, as Mehrotra and Marx detail, CVPD drones have taken nearly 20,000 flights, "often dispatched for serious incidents like reports of armed individuals [but] also routinely deployed for minor issues such as shoplifting, vandalism, and loud music. Early in the Covid-19 pandemic, the city even used drones to broadcast public service announcements to homeless encampments."
WIRED examined "nearly 10,000 drone flight records from July 2021 to September 2023," encompassing "more than 22.3 million coordinates from flight paths," to assess CVPD's claim that drones are only dispatched in response to specific 911 calls or lawful searches and do not merely go roaming in search of suspicious activity.
"Drones were used in about 7 percent of the city's service requests," the authors found, including "nearly half of the incidents involving reports of armed individuals and about a quarter of those related to violent crime," plus mental health and domestic violence calls.
"The vast majority" of the 10,000 flight records analyzed "could be linked to corresponding 911 calls. But not all of them." In fact, about 10 percent "lacked a stated purpose and could not be connected to any relevant 911 call; for 498 flights, the department lists the reason as an 'unknown problem.'" Further, "nearly 400 [flights] didn't come within half a mile of where any call in the preceding half hour originated."
Even specifically sanctioned flights may be cause for concern: "Operators are trained to start recording with the drone's camera immediately, capturing video throughout the entire flight, from takeoff to landing," Mehrotra and Marx note. The cameras, "powerful enough to capture faces clearly and constantly recording while in flight, have amassed hundreds of hours of video footage of the city's residents," the vast majority of which the city has refused to release.
"On average, each drone flight passes above 13 census blocks and potentially exposes approximately 4,700 of the residents below to a drone's camera," the WIRED analysis found. And potential exposure did not fall equally: "Residents on a typical block in the working-class and largely immigrant west side of Chula Vista had drones in the skies above 10 times longer than a resident of a typical east-side block," Mehrotra wrote in WIRED's Politics Lab newsletter yesterday. West-side residents "alleged that police drones were following them personally, lingering unnecessarily in their backyards, or watching them during their most intimate moments," and others complained about the noise of drone rotors. (The CVPD claimed the disparity is due to the unequal number of 911 calls that each area receives; the WIRED analysis "confirm[ed] that this is largely the case.")
Interestingly, support for the drone program is also strongest among the lower-income Chula Vista residents most likely to be subjected to it. One Latino man—who lives in an apartment complex that CVPD drones have flown over more than 300 times since July 2021—told WIRED that the drones make him feel safer, especially after a stranger tried to steal his child and police deployed a drone to look for the suspect. This isn't unheard of: Polls show black Americans are significantly more afraid of the police than their white neighbors, yet they still want a strong, effective police presence in their neighborhoods.
Regardless, Chula Vista's drone program could be a concerning sign of where American policing is headed. Even apart from DFR, city residents have been subject to a shocking amount of surveillance in recent years: automated license plate readers, facial recognition software, and a partnership with Amazon for access to its Ring doorbell cameras. In December 2017, the CVPD partnered with a company to share its data with other law enforcement agencies, including federal agencies like Immigration and Customs Enforcement (ICE) and Customs and Border Protection (CBP).
As Colorado's example makes clear, police departments increasingly see DFR programs as a plausible alternative to traditional policing, in which an officer would be dispatched to the scene of an emergency. While that's certainly true, it would also expose everyday citizens to a shocking new world of state surveillance.
Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.
RoboCup 2024: 17–22 July 2024, EINDHOVEN, NETHERLANDS
Do you have trouble multitasking? Cyborgize yourself through muscle stimulation to automate repetitive physical tasks while you focus on something else.
By combining a 5,000 frame-per-second (FPS) event camera with a 20-FPS RGB camera, roboticists from the University of Zurich have developed a much more effective vision system that keeps autonomous cars from crashing into stuff, as described in the current issue of Nature.
Mitsubishi Electric has been awarded the GUINNESS WORLD RECORDS title for the fastest robot to solve a puzzle cube. The robot’s time of 0.305 second beat the previous record of 0.38 second, for which it received a GUINNESS WORLD RECORDS certificate on 21 May 2024.
Sony’s AIBO is celebrating its 25th anniversary, which seems like a long time, and it is. But back then, the original AIBO could check your email for you. Email! In 1999!
MIT CSAIL researchers enhance robotic precision with sophisticated tactile sensors in the palm and agile fingers, setting the stage for improvements in human-robot interaction and prosthetic technology.
We present a novel adversarial attack method designed to identify failure cases in any type of locomotion controller, including state-of-the-art reinforcement-learning-based controllers. Our approach reveals the vulnerabilities of black-box neural network controllers, providing valuable insights that can be leveraged to enhance robustness through retraining.
In this work, we investigate a novel integrated flexible OLED display technology used as a robotic skin-interface to improve robot-to-human communication in a real industrial setting at Volkswagen or a collaborative human-robot interaction task in motor assembly. The interface was implemented in a workcell and validated qualitatively with a small group of operators (n=9) and quantitatively with a large group (n=42). The validation results showed that using flexible OLED technology could improve the operators’ attitude toward the robot; increase their intention to use the robot; enhance their perceived enjoyment, social influence, and trust; and reduce their anxiety.
We introduce InflatableBots, shape-changing inflatable robots for large-scale encountered-type haptics in VR. Unlike traditional inflatable shape displays, which are immobile and limited in interaction areas, our approach combines mobile robots with fan-based inflatable structures. This enables safe, scalable, and deployable haptic interactions on a large scale.
We present a bioinspired passive dynamic foot in which the claws are actuated solely by the impact energy. Our gripper simultaneously resolves the issue of smooth absorption of the impact energy and fast closure of the claws by linking the motion of an ankle linkage and the claws through soft tendons.
In this video, a 3-UPU exoskeleton robot for a wrist joint is designed and controlled to perform wrist extension, flexion, radial-deviation, and ulnar-deviation motions in stroke-affected patients. This is the first time a 3-UPU robot has been used effectively for any kind of task.
“UPU” stands for “universal-prismatic-universal” and refers to the actuators—the prismatic joints between two universal joints.
In this paper, we propose a system for the artist-directed authoring of stylized bipedal walking gaits, tailored for execution on robotic characters. To demonstrate the utility of our approach, we animate gaits for a custom, free-walking robotic character, and show, with two additional in-simulation examples, how our procedural animation technique generalizes to bipeds with different degrees of freedom, proportions, and mass distributions.
The European drone project Labyrinth aims to keep new and conventional air traffic separate, especially in busy airspaces such as those expected in urban areas. The project provides a new drone-traffic service and illustrates its potential to improve the safety and efficiency of civil land, air, and sea transport, as well as emergency and rescue operations.
This Carnegie Mellon University Robotics Institute seminar, by Kim Baraka at Vrije Universiteit Amsterdam, is on the topic “Why We Should Build Robot Apprentices and Why We Shouldn’t Do It Alone.”
For robots to be able to truly integrate human-populated, dynamic, and unpredictable environments, they will have to have strong adaptive capabilities. In this talk, I argue that these adaptive capabilities should leverage interaction with end users, who know how (they want) a robot to act in that environment. I will present an overview of my past and ongoing work on the topic of human-interactive robot learning, a growing interdisciplinary subfield that embraces rich, bidirectional interaction to shape robot learning. I will discuss contributions on the algorithmic, interface, and interaction design fronts, showcasing several collaborations with animal behaviorists/trainers, dancers, puppeteers, and medical practitioners.
Stephen Cass: Hello. I’m Stephen Cass, Special Projects Director at IEEE Spectrum. Before starting today’s episode hosted by Eliza Strickland, I wanted to give you all listening out there some news about this show.
This is our last episode of Fixing the Future. We’ve really enjoyed bringing you some concrete solutions to some of the world’s toughest problems, but we’ve decided we’d like to be able to go deeper into topics than we can in the course of a single episode. So we’ll be returning later in the year with a program of limited series that will enable us to do those deep dives into fascinating and challenging stories in the world of technology. I want to thank you all for listening and I hope you’ll join us again. And now, on to today’s episode.
Eliza Strickland: Hi, I’m Eliza Strickland for IEEE Spectrum‘s Fixing the Future podcast. Before we start, I want to tell you that you can get the latest coverage from some of Spectrum’s most important beats, including AI, climate change, and robotics, by signing up for one of our free newsletters. Just go to spectrum.IEEE.org/newsletters to subscribe.
Around the world, about 60 countries are contaminated with land mines and unexploded ordnance, and Ukraine is the worst off. Today, about a third of its land, an area the size of Florida, is estimated to be contaminated with dangerous explosives. My guest today is Gabriel Steinberg, who co-founded both the nonprofit Demining Research Community and the startup Safe Pro AI with his friend, Jasper Baur. Their technology uses drones and artificial intelligence to radically speed up the process of finding land mines and other explosives. Okay, Gabriel, thank you so much for joining me on Fixing the Future today.
Gabriel Steinberg: Yeah, thank you for having me.
Strickland: So I want to start by hearing about the typical process for demining, and so the standard operating procedure. What tools do people use? How long does it take? What are the risks involved? All that kind of stuff.
Steinberg: Sure. So humanitarian demining hasn’t changed significantly. There’s been evolutions, of course, since its inception and about the end of World War I. But mostly, the processes have been the same. People stand from a safe location and walk around an area in areas that they know are safe, and try to get as much intelligence about the contamination as they can. They ask villagers or farmers, people who work around the area and live around the area, about accidents and potential sightings of minefields and former battle positions and stuff. The result of this is a very general idea, a polygon, of where the contamination is. After that polygon and some prioritization based on danger to civilians and economic utility, the field goes into clearance. The first part is the non-technical survey, and then this is clearance. Clearance happens one of three ways, usually, but it always ends up with a person on the ground basically doing extreme gardening. They dig out a certain standard amount of the soil, usually 13 centimeters. And with a metal detector, they walk around the field and a mine probe. They find the land mines and nonexploded ordnance. So that always is how it ends.
To get to that point, you can also use mechanical assets, which are large tillers, and sometimes dogs and other animals are used to walk in lanes across the contaminated polygon to sniff out the land mines and tell the clearance operators where the land mines are.
Strickland: How do you hope that your technology will change this process?
Steinberg: Well, my technology is a drone-based mapping solution, basically. So we provide a software to the humanitarian deminers. They are already flying drones over these areas. Really, it started ramping up in Ukraine. The humanitarian demining organizations have started really adopting drones just because it’s such a massive problem. The extent is so extreme that they need to innovate. So we provide AI and mapping software for the deminers to analyze their drone imagery much more effectively. We hope that this process, or our software, will decrease the amount of time that deminers use to analyze the imagery of the land, thereby more quickly and more effectively constraining the areas with the most contamination. So if you can constrain an area, a polygon with a certainty of contamination and a high density of contamination, then you can deploy the most expensive parts of the clearance process, which are the humans and the machines and the dogs. You can deploy them to a very specific area. You can much more cost-effectively and efficiently demine large areas.
Strickland: Got it. So it doesn’t replace the humans walking around with metal detectors and dogs, but it gets them to the right spots faster.
Steinberg: Exactly. Exactly. At the moment, there is no conception of replacing a human in demining operations, and people that try to push that eventuality are usually disregarded pretty quickly.
Strickland: How did you and your co-founder, Jasper, first start experimenting with the use of drones and AI for detecting explosives?
Steinberg: So it started in 2016 with my partner, Jasper Baur, doing a research project at Binghamton University in the remote sensing and geophysics lab. And the project was to detect a specific anti-personnel land mine, thePFM-1. Then found— it’s a Russian-made land mine. It was previously found in Afghanistan. It still is found in Afghanistan, but it’s found in much higher quantities right now in Ukraine. And so his project was to detect the PFM-1 anti-personnel land mine using thermal imagery from drones. It sort of snowballed into quite an intensive research project. It had multiple papers from it, multiple researchers, some awards, and most notably, it beat NASA at a particular Tech Briefs competition. So that was quite a morale boost.
And at some point, Jasper had the idea to integrate AI into the project. Rightfully, he saw the real bottleneck as not the detecting of land mines in drone imagery, but the analysis of land mines in drone imagery. And that really has become— I mean, he knew, somehow, that that would really become the issue that everybody is facing. And everybody we talked to in Ukraine is facing that issue. So machine learning really was the key for solving that problem. And I joined the project in 2018 to integrate machine learning into the research project. We had some more papers, some more presentations, and we were nearing the end of our college tenure, of our undergraduate degree, in 2020. So at that time– but at that time, we realized how much the field needed this. We started getting more and more into the mine action field, and realizing how neglected the field was in terms of technology and innovation. And we felt an obligation to bring our technology, really, to the real world instead of just a research project. There were plenty of research projects about this, but we knew that it could be more and that it should. It really should be more. And we felt we had the– for some reason, we felt like we had the capability to make that happen.
So we formed a nonprofit, the Demining Research Community, in 2020 to try to raise some funding for this project. Our for-profit end of that, of our endeavors, was acquired by a company called Safe Pro Group in 2023. Yeah, 2023, about one year ago exactly. And the drone and AI technology became Safe Pro AI and our flagship product spotlight. And that’s where we’re bringing the technology to the real world. The Demining Research Community is providing resources for other organizations who want to do a similar thing, and is doing more research into more nascent technologies. But yeah, the real drone and AI stuff that’s happening in the real world right now is through Safe Pro.
Strickland: So in that early undergraduate work, you were using thermal sensors. I know now the Spotlight AI system is using more visual. Can you talk about the different modalities of sensing explosives and the sort of trade-offs you get with them?
Steinberg: Sure. So I feel like I should preface this by saying the more high tech and nascent the technology is, the more people want to see it apply to land mine detection. But really, we have found from the problems that people are facing, by far the most effective modality right now is just visual imagery. People have really good visual sensors built into their face, and you don’t need a trained geophysicist to observe the data and very, very quickly get actionable intelligence. There’s also plenty of other benefits. It’s cheaper, much more readily accessible in Ukraine and around the world to get built-in visual sensors on drones. And yeah, just processing the data, and getting the intelligence from the data, is way easier than anything else.
I’ll talk about three different modalities. Well, I guess I could talk about four. There’s thermal, ground penetrating radar, magnetometry, and lidar. So thermal is what we started with. Thermal is really good at detecting living things, as I’m sure most people can surmise. But it’s also pretty good at detecting land mines, mostly large anti-tank land mines buried under a couple millimeters, or up to a couple centimeters, of soil. It’s not super good at this. The research is still not super conclusive, and you have to do it at a very specific time of day, in the morning and at night when, basically the soil around the land mine heats up faster than the land mine and you cause a thermal anomaly, or the sun causes a thermal anomaly. So it can detect things, land mines, in some amount of depth in certain soils, in certain weather conditions, and can only detect certain types of land mines that are big and hefty enough. So yeah, that’s thermal.
Ground penetrating radar is really good for some things. It’s not really great for land mine detection. You have to have really expensive equipment. It takes a really long time to do the surveys. However, it can get plastic land mines under the surface. And it’s kind of the only modality that can do that with reliability. However, you need to train geophysicists to analyze the data. And a lot of the time, the signatures are really non-unique and there’s going to be a lot of false positives. Magnetometry is the other-- by the way, all of this is airborne that I’m referring to. Ground-based GPR and magnetometry are used in demining of various types, but airborne is really what I’m talking about.
For magnetometry, it’s more developed and more capable than ground penetrating radar. It’s used, actually, in the field in Ukraine in some scenarios, but it’s still very expensive. It needs a trained geophysicist to analyze the data, and the signatures are non-unique. So whether it’s a bottle can or a small anti-personnel land mine, you really don’t know until you dig it up. However, I think if I were to bet on one of the other modalities becoming increasingly useful in the next couple of years, it would be airborne magnetometry.
Lidar is another modality that people use. It’s pretty quick, also very expensive, but it can reliably map and find surface anomalies. So if you want to find former fighting positions, sometimes an indicator of that is a trench line or foxholes. Lidar is really good at doing that in conflicts from long ago. So there’s a paper that theHALO Trust published of flyinga lidar mission over former fighting positions, I believe, in Angola. And they reliably found a former trench line. And from that information, they confirmed that as a hazardous area. Because if there is a former front line on this position, you can pretty reliably say that there is going to be some explosives there.
Strickland: And so you’ve done some experiments with some of these modalities, but in the end, you found that the visual sensor was really the best bet for you guys?
Steinberg: Yeah. It’s different. The requirements are different for different scenarios and different locations, really. Ukraine has a lot of surface ordnance. Yeah. And that’s really the main factor that allows visual imagery to be so powerful.
Strickland: So tell me about what role machine learning plays in your Spotlight AI software system. Did you create a model trained on a lot of— did you create a model based on a lot of data showing land mines on the surface?
Steinberg: Yeah. Exactly. We used real-world data from inert, non-explosive items, and flew drone missions over them, and did some physical augmentation and some programmatic augmentation. But all of the items that we are training on are real-life Russian or American ordnance, mostly. We’re also using the real-world data in real minefields that we’re getting from Ukraine right now. That is, obviously, the most valuable data and the most effective in building a machine learning model. But yeah, a lot of our data is from inert explosives, as well.
Strickland: So you’ve talked a little bit about the current situation in Ukraine, but can you tell me more about what people are dealing with there? Are there a lot of areas where the battle has moved on and civilians are trying to reclaim roads or fields?
Steinberg: Yeah. So the fighting is constantly ongoing, obviously, in eastern Ukraine, but I think sometimes there’s a perspective of a stalemate. I think that’s a little misleading. There’s lots of action and violence happening on the front line, which constantly contaminates, cumulatively, the areas that are the front line and the gray zone, as well as areas up to 50 kilometers back from both sides. So there’s constantly artillery shells going into villages and cities along the front line. There’s constantly land mines, new mines, being laid to reinforce the positions. And there’s constantly mortars. And everything is constant. In some fights—I just watched the video yesterday—one of the soldiers said you could not count to five without an explosion going off. And this is just one location in one city along the front. So you can imagine the amount of explosive ordnance that are being fired, and inevitably 10, 20, 30 percent of them are sometimes not exploding upon impact, on top of all the land mines that are being purposely laid and not detonating from a vehicle or a person. These all just remain after the war. They don’t go anywhere. So yeah, Ukraine is really being littered with explosive ordnance and land mines every day.
This past year, there hasn’t been terribly much movement on the front line. But in the Ukrainian counteroffensive in 2020— I guess the last major Ukrainian counteroffensive where areas of Mykolaiv, which is in the southeast, were reclaimed, the civilians started repopulating the city almost immediately. There are definitely some villages that are heavily contaminated, that people just deserted and never came back to, and still haven’t come back to after them being liberated. But a lot of the areas that have been liberated, they’re people’s homes. And even if they’re destroyed, people would rather be in their homes than be refugees. And I mean, I totally understand that. And it just puts the responsibility on the deminers and the Ukrainian government to try to clear the land as fast as possible. Because after large liberations are made, people want to come back almost all the time. So it is a very urgent problem as the lines change and as land is liberated.
Strickland: And I think it was about a year ago that you and Jasper went to the Ukraine for a technology demonstration set up by the United Nations. Can you tell about that, and what the task was, and how your technology fared?
Steinberg: Sure. So yeah, the United Nations Development Program invited us to do a demonstration in northern Ukraine to see how our technology, and other technologies similar to it, performed in a military training facility in Ukraine. So everybody who’s doing this kind of thing, which is not many people, but there are some other organizations, they have their own metrics and their own test fields— not always, but it would be good if they did. But the UNDP said, “No, we want to standardize this and try to give recommendations to the organizations on the ground who are trying to adopt these technologies.” So we had five hours to survey the field and collect as much data as we could. And then we had 72 hours to return the results. We—
Strickland: Sorry. How big was the field?
Steinberg: The field was 25 hectares. So yeah, the audience at home can type 25 hectares to amount of football fields. I think it’s about 60. But it’s a large area. So we’d never done anything like that. That was really, really a shock that it was that large of an area. I think we’d only done half a hectare at a time up to that point. So yeah, it was pretty daunting. But we basically slept very, very little in those 72 hours, and as a result, produced what I think is one of the best results that the UNDP got from that test. We didn’t detect everything, but we detected most of the ordnance and land mines that they had laid. We also detected some that they didn’t know were there because it was a military training facility. So there were some mortars being fired that they didn’t know about.
Strickland: And I think Jasper told me that you had to sort of rewrite your software on the fly. You realized that the existing approach wasn’t going to work and you had to do some all-nighter to recode?
Steinberg: Yeah. Yeah, I remember us sitting in a Georgian restaurant— Georgia, the country, not the state, and racking our brain, trying to figure out how we were going to map this amount of land. We just found out how big the area was going to be and we were a little bit stunned. So we devised a plan to do it in two stages. The first stage was where we figured out in the drone images where the contaminated regions were. And then the second stage was to map those areas, just those areas. Now, our software can actually map the whole thing, and pretty casually too. So not to brag. But at the time, we had lots less development under our belt. And yeah, therefore we just had to brute force it through Georgian food and brainpower.
Strickland: You and Jasper just got back from another trip to the Ukraine a couple of weeks ago, I think. Can you talk about what you were doing on this trip, and who you met with?
Steinberg: Sure. This trip was much less stressful, although stressful in different ways than the UNDP demo. Our main objectives were to see operations in action. We had never actually been to real minefields before. We’d been in some perhaps contaminated areas, but never in a real minefield where you can say, “Here was the Russian position. There are the land mines. Do not go there.” So that was one of the main objectives. That was very powerful for us to see the villages that were destroyed and are denied to the citizens because of land mines and unexploded ordnance. It’s impossible to describe how that feels being there. It’s really impactful, and it makes the work that I’m doing feel not like I have a choice anymore. I feel very much obligated to do my absolute best to help these people.
Strickland: Well, I hope your work continues. I hope there’s less and less need for it over time. But yeah, thank you for doing this. It’s important work. And thanks for joining me on Fixing the Future.
Steinberg: My pleasure. Thank you for having me.
Strickland: That was Gabriel Steinberg speaking to me about the technology that he and Jasper Baur developed to help rid the world of land mines. I’m Eliza Strickland, and I hope you’ll join us next time on Fixing the Future.
Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please
send us your events for inclusion.
RoboCup 2024: 17–22 July 2024, EINDHOVEN, NETHERLANDS
NAVER 1784 is the world’s largest robotics testbed. The Starbucks on the second floor of 1784 is the world’s most unique Starbucks, with more than 100 service robots called “Rookie” delivering Starbucks drinks to meeting rooms and private seats, and various experiments with a dual-arm robot.
This video is presented at the ICRA 2024 conference and summarizes recent results of our Learning AI for Dextrous Manipulation Lab. It demonstrates how our learning AI methods allowed for breakthroughs in dextrous manipulation with the mobile humanoid robot DLR Agile Justin. Although the core of the mechatronic hardware is almost 20 years old, only the advent of learning AI methods enabled a level of dexterity, flexibility and autonomy coming close to human capabilities.
With all the humanoid stuff going on, there really should be more emphasis on intentional contact—humans lean and balance on things all the time, and robots should too!
LimX Dynamics W1 is now more than a wheeled quadruped. By evolving into a biped robot, W1 maneuvers slickly on two legs in different ways: non-stop 360° rotation, upright free gliding, slick maneuvering, random collision and self-recovery, and step walking.
Animal brains use less data and energy compared to current deep neural networks running on Graphics Processing Units (GPUs). This makes it hard to develop tiny autonomous drones, which are too small and light for heavy hardware and big batteries. Recently, the emergence of neuromorphic processors that mimic how brains function has made it possible for researchers from Delft University of Technology to develop a drone that uses neuromorphic vision and control for autonomous flight.
In the beginning of the universe, all was darkness — until the first organisms developed sight, which ushered in an explosion of life, learning and progress. AI pioneer Fei-Fei Li says a similar moment is about to happen for computers and robots. She shows how machines are gaining “spatial intelligence” — the ability to process visual data, make predictions and act upon those predictions — and shares how this could enable AI to interact with humans in the real world.
Arthur Erickson discovered drones during his first year at college studying aerospace engineering. He immediately thought the sky was the limit for how the machines could be used, but it took years of hard work and some nimble decisions to turn that enthusiasm into a successful startup.
Today, Erickson is the CEO of Houston-based Hylio, a company that builds crop-spraying drones for farmers. Launched in 2015, the company has its own factory and employs more than 40 people.
Arthur Erickson
Occupation:
Aerospace engineer and founder, Hylio
Location:
Houston
Education:
Bachelor’s degree in aerospace, specializing in aeronautics, from the University of Texas at Austin
Erickson founded Hylio with classmates while they were attending the University of Texas at Austin. They were eager to quit college and launch their business, which he admits was a little presumptuous.
“We were like, ‘Screw all the school stuff—drones are the future,’” Erickson says. “I already thought I had all the requisite technical skills and had learned enough after six months of school, which obviously was arrogant.”
His parents convinced him to finish college, but Erickson and the other cofounders spent all their spare time building a multipurpose drone from off-the-shelf components and parts they made using their university’s 3D printers and laser cutters.
By the time he graduated in 2017 with a bachelor’s degree in aerospace, specializing in aeronautics, the group’s prototype was complete, and they began hunting for customers. The next three years were a wild ride of testing their drones in Costa Rica and other countries across Central America.
A grocery delivery service
A promotional video about the company that Erickson posted on Instagram led to the first customer, the now-defunct Costa Rican food and grocery delivery startup GoPato. The company wanted to use the drones to make deliveries in the capital, San José, but rather than purchase the machines, GoPato offered to pay for the founders’ meals and lodging and give them a percentage of delivery fees collected.
For the next nine months, Hylio’s team spent their days sending their drones on deliveries and their nights troubleshooting problems in a makeshift workshop in their shared living room.
“We had a lot of sleepless nights,” Erickson says. “It was a trial by fire, and we learned a lot.”
One lesson was the need to build in redundant pieces of key hardware, particularly the GPS unit. “When you have a drone crash in the middle of a Costa Rican suburb, the importance of redundancy really hits home,” Erickson says.
“Drones are great for just learning, iterating, crashing things, and then rebuilding them.”
The small cut of delivery fees Hylio received wasn’t covering costs, Erickson says, so eventually the founders parted ways with GoPato. Meanwhile, they had been looking for new business opportunities in Costa Rica. They learned from local farmers that the terrain was too rugged for tractors, so most sprayed crops by hand. This was both grueling and hazardous because it brought the farmers into close proximity to the pesticides.
The Hylio team realized its drones could do this type of work faster and more safely. They designed a spray system and made some software tweaks, and by 2018 the company began offering crop-spraying services, Erickson says. The company expanded its business to El Salvador, Guatemala, and Honduras, starting with just a pair of drones but eventually operating three spraying teams of four drones each.
The work was tough, Erickson says, but the experience helped the team refine their technology, working out which sensors operated best in the alternately dusty and moist conditions found on farms. Even more important, by the end of 2019 they were finally turning a profit.
Drones are cheaper than tractors
In hindsight, agriculture was an obvious market, Erickson says, even in the United States, where spraying with herbicides, pesticides, and fertilizers is typically done using large tractors. These tractors can cost up to half a million dollars to purchase and about US $7 a hectare to operate.
A pair of Hylio’s drones cost a fifth of that, Erickson says, and operating them costs about a quarter of the price. The company’s drones also fly autonomously; an operator simply marks GPS waypoints on a map to program the drone where to spray and then sits back and lets it do the job. In this way, one person can oversee multiple drones working at once, covering more fields than a single tractor could.
Arthur Erickson inspects the company’s largest spray drone, the AG-272. It can cover thousands of hectares per day.Hylio
Convincing farmers to use drones instead of tractors was tough, Erickson says. Farmers tend to be conservative and are wary of technology companies that promise too much.
“Farmers are used to people coming around every few years with some newfangled idea, like a laser that’s going to kill all their weeds or some miracle chemical,” he says.
In 2020, Hylio opened a factory in Houston and started selling drones to American farmers. The first time Hylio exhibited its machines at an agricultural trade show, Erickson says, a customer purchased one on the spot.
“It was pretty exciting,” he says. “It was a really good feeling to find out that our product was polished enough, and the pitch was attractive enough, to immediately get customers.”
Today, selling farmers on the benefits of drones is a big part of Erickson’s job. But he’s still involved in product development, and his daily meetings with the sales team have become an invaluable source of customer feedback. “They inform a lot of the features that we add to the products,” he says.
He’s currently leading development of a new type of drone—a scout—designed to quickly inspect fields for pest infestations or poor growth or to assess crop yields. But these days his job is more about managing his team of engineers than about doing hands-on engineering himself. “I’m more of a translator between the engineers and the market needs,” he says.
Focus on users’ needs
Erickson advises other founders of startups not to get too caught up in the excitement of building cutting-edge technology, because you can lose sight of what the user actually needs.
“I’ve become a big proponent of not trying to outsmart the customers,” he says. “They tell us what their pain points are and what they want to see in the product. Don’t overengineer it. Always check with the end users that what you’re building is going to be useful.”
Working with drones forces you to become a generalist, Erickson says. You need a basic understanding of structural mechanics and aerodynamics to build something airworthy. But you also need to be comfortable working with sensors, communications systems, and power electronics, not to mention the software used to control and navigate the vehicles.
Erickson advises students who want to get into the field to take courses in mechatronics, which provide a good blend of mechanical and electrical engineering. Deep knowledge of the individual parts is generally not as important as understanding how to fit all the pieces together to create a system that works well as a whole.
And if you’re a tinkerer like he is, Erickson says, there are few better ways to hone your engineering skills than building a drone. “It’s a cheap, fast way to get something up in the air,” he says. “They’re great for just learning, iterating, crashing things, and then rebuilding them.”
This article appears in the June 2024 print issue as “Careers: Arthur Erickson.”
Instead of dispatching an officer each time, several Colorado police departments may soon dispatch a drone to respond to certain 911 calls. While the proposal has promise, it also raises uncomfortable questions about privacy.
As Shelly Bradbury reported this week in The Denver Post, "A handful of local law enforcement agencies are considering using drones as first responders—that is, sending them in response to 911 calls—as police departments across Colorado continue to widely embrace the use of the remote-controlled flying machines."
Bradbury quotes Arapahoe County Sheriff Jeremiah Gates saying, "This really is the future of law enforcement at some point, whether we like it or not." She notes that while there are currently no official plans in place, "Gates envisions a world where a drone is dispatched to a call about a broken traffic light or a suspicious vehicle instead of a sheriff's deputy, allowing actual deputies to prioritize more pressing calls for help."
The Denver Police Department—whose then-chief in 2013 called the use of drones by police "controversial" and said that "constitutionally there are a lot of unanswered questions about how they can be used"—is also starting a program, buying several drones over the next year that can eventually function as first responders.
In addition to Denver and Arapahoe County, Bradbury lists numerous Colorado law enforcement agencies that also have drone programs, including the Colorado State Patrol, which has 24 drones, and the Commerce City Police Department, which has eight drones and 12 pilots for a city of around 62,000 people and plans to begin using them for 911 response within a year.
In addition to helping stem the number of calls an officer must respond to in person, some law enforcement agencies see this as a means of saving money. One Commerce City police official told The Denver Post that "what we see out of it is, it's a lot cheaper than an officer, basically." And Denver intends for its program to make up for an $8.4 million cut to the police budget this year.
On one hand, there is certainly merit to such a proposal: Unless they're of the Predator variety, drones are much less likely than officers to kill or maim innocent civilians—or their dogs. And as Gates noted, drones could take some of the busywork out of policing by taking some of the more mundane tasks off an officer's plate.
But it also raises privacy concerns to farm out too much police work to unmanned surveillance aircraft.
"Sending out a drone for any time there is a 911 call, it could be dangerous and lead to more over-policing of communities of color," Laura Moraff, a staff attorney for the American Civil Liberties Union of Colorado, told The Denver Post. "There is also just the risk that the more that we normalize having drones in the skies, the more it can really affect behavior on a massive scale, if we are just looking up and seeing drones all over the place, knowing that police are watching us."
Indeed, while this sort of dystopic panopticon would certainly make life easier for officers day to day, it would signal the further erosion of the average Coloradan's Fourth Amendment rights.
In Michigan, for example, police hired a drone pilot to take pictures of a person's property rather than go to the trouble of getting a warrant. Earlier this month, the state supreme court upheld the search, ruling that since the purpose was for civil code enforcement and not a criminal violation, it didn't matter whether the search violated the Fourth Amendment.
Thankfully, there are some positive developments on that front: In March, the Alaska Supreme Court ruled against state troopers who flew a plane over a suspect's house and took pictures with a high-powered zoom lens to see if he was growing marijuana.
"The fact that a random person might catch a glimpse of your yard while flying from one place to another does not make it reasonable for law enforcement officials to take to the skies and train high-powered optics on the private space right outside your home without a warrant," the court found. "Unregulated aerial surveillance of the home with high-powered optics is the kind of police practice that is 'inconsistent with the aims of a free and open society.'"
Early on a June morning in 2023, my colleagues and I drove down a bumpy dirt road north of Kyiv in Ukraine. The Ukrainian Armed Forces were conducting training exercises nearby, and mortar shells arced through the sky. We arrived at a vast field for a technology demonstration set up by the United Nations. Across the 25-hectare field—that’s about the size of 62 American football fields—the U.N. workers had scattered 50 to 100 inert mines and other ordnance. Our task was to fly our drone over the area and use our machine learning software to detect as many as possible. And we had to turn in our results within 72 hours.
The scale was daunting: The area was 10 times as large as anything we’d attempted before with our drone demining startup,
Safe Pro AI. My cofounder Gabriel Steinberg and I used flight-planning software to program a drone to cover the whole area with some overlap, taking photographs the whole time. It ended up taking the drone 5 hours to complete its task, and it came away with more than 15,000 images. Then we raced back to the hotel with the data it had collected and began an all-night coding session.
We were happy to see that our custom machine learning model took only about 2 hours to crunch through all the visual data and identify potential mines and ordnance. But constructing a map for the full area that included the specific coordinates of all the detected mines in under 72 hours was simply not possible with any reasonable computational resources. The following day (which happened to coincide with the short-lived
Wagner Group rebellion), we rewrote our algorithms so that our system mapped only the locations where suspected land mines were identified—a more scalable solution for our future work.
In the end we detected 74 mines and ordnance scattered across the surface of that enormous field, and the U.N. deemed our results impressive enough to invite us back for a second round of demonstrations. While we were in Ukraine, we also demonstrated our technology for the
State Special Transportation Service, a branch of the Ukrainian military responsible for keeping roads and bridges open.
All our hard work paid off. Today, our technology is being used by several humanitarian nonprofits detecting land mines in Ukraine, including the
Norwegian People’s Aid and the HALO Trust, which is the world’s largest nonprofit dedicated to clearing explosives left behind after wars. Those groups are working to make Ukraine’s roads, towns, and agricultural fields safe for the Ukrainian people. Our goal is to make our technology accessible to every humanitarian demining operation, making their jobs safer and more efficient. To that end, we’re deploying and scaling up—first across Ukraine, and soon around the world.
The Scale of the Land-Mine Problem
The remnants of war linger long after conflicts have died down. Today, an estimated 60 countries are still contaminated by mines and unexploded ordnance, according to the
2023 Landmine Monitor report. These dangers include land mines, improvised explosive devices, and shells and artillery that didn’t explode on landing—all together, they’re known as explosive ordnance (EO). More than 4,700 people were killed or wounded by EO in 2022, according to the Landmine Monitor report, and the vast majority of those casualties were civilians. Today, Ukraine is the most contaminated place in the world. About a third of its land—an area the size of Florida—is estimated to contain EO.
In humanitarian mine-clearing work, the typical process for releasing EO-contaminated land back to the community hasn’t changed much over the past 50 years. First a nontechnical survey is conducted where personnel go out to talk with local people about which areas are suspected of being contaminated. Next comes the technical survey, in which personnel use metal detectors, trained dogs, mechanical demining machines, and geophysical methods to identify all the hazards within a mined area. This process is slow, risky, and prone to false positives triggered by cans, screws, or other metal detritus. Once the crew has identified all the potential hazards within an area, a team of explosive-ordnance-disposal specialists either disarm or destroy the explosives.
Unexploded ordnance lies by the road in a Ukrainian town near the war’s front lines. John Moore/Getty Images
Most deminers would agree that it’s not ideal to identify the EO as they walk through the contaminated area; it would be much better to know the lay of the land before they take their first steps. That’s where drones can be literal lifesavers: They take that first look safely from up above, and they can quickly and cheaply cover a large area.
What’s more, the scale of the problem makes artificial intelligence a compelling part of the solution. Imagine if drone imagery was collected for all of Ukraine’s suspected contaminated land: an area of more than 170,000 square kilometers. It takes about 60,000 drone images to cover 1 km2 at a useful resolution, and we estimate that it takes at minimum 3 minutes for a human expert to analyze a drone image and check for EO. At that rate, it would take more than 500 million person-hours to manually search imagery covering all of Ukraine’s suspected contaminated land for EO. With AI, the task of analyzing this imagery and locating all visible EO in Ukraine will still be a massive endeavor, but it’s within reason.
“Today, our technology is being used by several humanitarian nonprofits detecting land mines in Ukraine.”
Humanitarian demining groups are slow to adopt new technologies because any mistake, including ones caused by unfamiliarity with new tech, can be fatal. But in the last couple of years, drones seem to have reached an inflection point. Many government agencies and nonprofit groups that work on land-mine detection and removal are beginning to integrate drones into their standard procedures. Besides collecting aerial imagery of large areas with suspected hazards, which helps with route planning, the drones are prioritizing areas of clearance, and in some cases, detecting land mines themselves.
After several years of research on this topic during my undergraduate education, in 2020 I cofounded the company now known as Safe Pro AI to push the technology forward and make deployment a reality. My cofounder and I didn’t know at the time that Russia’s full-scale invasion of Ukraine in February 2022 would soon make this work even more vital.
How We Got Started With Drones for Demining
I became interested in land-mine detection while studying geological science as an undergraduate at Binghamton University, New York. Through my work in the Geophysics and Remote Sensing Laboratory run by Timothy de Smet and Alex Nikulin, I got involved in a project to detect the PFM-1, a Russian-made antipersonnel land mine also known as the butterfly mine due to its unique shape and because it’s typically scattered by aircraft or artillery shells. Afghanistan is still contaminated with many of these mines, left behind more than 40 years ago after the Soviet-Afghan War. They’re particularly problematic because they’re mostly made of plastic, with only a few small metal components; to find them with a metal detector requires turning up the equipment’s sensitivity, which leads to more false positives.
In 2019, we trained a machine learning model by scattering inert PFM-1 land mines and collecting visual imagery via drone flights in various environments, including roads, urban areas, grassy fields, and places with taller vegetation. Our resulting model correctly detected 92 percent of PFM-1s in these environments, on average. While we were pleased with its performance, the model could identify only that one type of land mine, and only if they were above ground. Still, this work provided the proof of concept that paved the way for what we’re doing today. In 2020, Steinberg and I founded the Demining Research Community, a nonprofit whose goal is to advance the field of humanitarian mine removal through research in remote sensing, geophysics, and robotics.
Over the next few years, we continued to develop our software and make contacts in the field. At the 2021 Mine Action Innovation Conference in Geneva, we heard about a researcher named John Frucci at Oklahoma State University who directs the OSU Global Consortium for Explosive Hazard Mitigation. In the summer of 2022, we spent two weeks with Frucci at OSU’s explosives range, which has more than 50 types of unexploded ordnance. We used our drones to collect visual training data for many different types of explosives: small antipersonnel mines, larger antitank mines, improvised explosive devices, grenades, and many other dangerous explosive things you never want to encounter.
Our Software Solution for Demining by Drone
To develop our technology for real-world use, Steinberg and I cofounded Safe Pro AI and joined Safe Pro Group, a company that provides drone services and sells protective gear for demining crews. Going into this work, we were aware of many academic proposals for new methods of EO detection that haven’t gotten out of the lab. We wanted to break that paradigm, so we spent a lot of time talking with demining personnel about their needs. Safe Pro Group’s director of operations in Ukraine, Fred Polk, spent more than 200 days last year talking to deminers in Ukraine about the problems they face and the solutions they’d like to see. In light of those conversations, we developed a user-friendly Web application called SpotlightAI. Any authorized person can log on to the website and upload their imagery from a commercial off-the-shelf drone; our system will then run the visual data through our AI model and return a map with all the coordinates of the detected explosive ordnance.
We don’t anticipate that the technology will replace human labor—personnel will still have to go through fields with metal detectors to be sure the drones haven’t missed anything. But the drones can speed up the process of the initial nontechnical survey and can also help demining operators figure out which areas to prioritize. The drone-based maps can also give personnel more situational awareness going into an inherently dangerous situation.
“Drones can be literal lifesavers: They take the first look at a minefield safely from up above.”
The first big test of our technology was in 2022 in Budapest at a Hungarian Explosive Ordnance Disposal test range. At that time, I was at Mount Okmok, a volcano in Alaska’s Aleutian Islands, doing field work on volcanology for my Ph.D., so Steinberg represented Safe Pro AI at that event. He told me via satellite phone that our model detected 20 of the 23 pieces of ordnance, returning the results in under an hour.
After Budapest we made two trips to Ukraine, first to field-test our technology in a real-world minefield environment and then for the 2023 U.N. demonstration previously described. In another trip this past March, we visited minefields in eastern Ukraine that are currently being demined by nonprofit organizations using our SpotlightAI system. We were accompanied by Artem Motorniuk, a Ukrainian software developer who joined Safe Pro Group in 2023. It was incredibly saddening to see the destruction of communities firsthand: Even after the front line has moved, explosive remnants of war still hinder reconstruction. Many people flee, but the ones who stay are faced with difficult decisions. They must balance essential activities such as farming and rebuilding with the risks posed by pursing those activities in areas that might have land mines and explosive ordnance. Seeing the demining operations firsthand reinforced the impact of the work, and listening to the demining operators’ feedback in the field helped us further refine the technology.
4 Ways to Sense Danger
We’ve continued to improve the performance of our model, and it has finally reached a point where it’s almost as good as an expert human in detecting EO on the surface from visual imagery, while performing this task many times faster than any human could. Sometimes it even finds items that are heavily obscured by vegetation. To give it superhuman capabilities to peer under the dirt, we need to bring in other detection modalities. For example, while we originally rejected thermal imaging as a stand-alone detection method, we’re now experimenting with using it in conjunction with visual imaging. The visual--imagery-based machine learning model returns the detection results, but we then add a thermal overlay that can reveal other information—for example, it might show a ground disturbance that suggests a buried object.
The biggest challenge we’re grappling with now is how to detect EO through thick and high vegetation. One strategy I developed is to use the drone imagery to create a 3D map, which is used to estimate the vegetation height and coverage. An algorithm then converts those estimates into a heat map showing how likely it is that the machine learning model can detect EO in each area: For example, it might show a 95 percent detection rate in a flat area with low grass, and only a 5 percent detection rate in a region with trees and bushes. While this approach doesn’t solve the problem posed by vegetation, it gives deminers more context for our results. We’re also incorporating more vegetation imagery into our training data itself to improve the model’s detection rate in such situations.
To offer these services in a scalable way, Safe Pro AI has partnered with Amazon Web Services, which is providing computational resources to deal with large amounts of visual imagery uploaded to SpotlightAI. Drone-based land-mine detection in Ukraine is a problem of scale. An average drone pilot can collect more than 30 hectares (75 acres) of imagery per day, roughly equal to 20,000 images. Each one of these images covers an area of 10 by 20 meters, within which the system must detect a land mine the size of your hand and the color of grass. AWS allows us to utilize extremely powerful computers on demand to process thousands of images a day through our machine learning model to meet the needs of deminers in Ukraine.
What’s Next for Our Humanitarian Demining Work
One obvious way we could improve our technology is by enabling it to detect buried EO, either by visually detecting disturbed earth or using geophysical sensors. In the summer of 2023, our nonprofit experimented with putting ground-penetrating radar, aerial magnetometry, lidar, and thermal sensors on our drones in an attempt to locate buried items.
We found that lidar is useful for detecting trenches that are indicative of ground disturbance, but it can’t detect the buried objects themselves. Thermal imagery can be useful if a buried metal item has a very different thermal signature than the surrounding soil, but we typically see a strong differential only in certain environments and at certain times of day. Magnetometers are the best tools for detecting buried metal targets—they’re the most similar to handheld metal detectors that deminers use. But the magnetic signal gets weaker as the drone gets farther from the ground, decreasing at an exponential rate. So if a drone flies too high, it won’t see the magnetic signatures and won’t detect the objects; but if it flies too low, it may have to navigate through bushes or other terrain obstacles. We’re continuing to experiment with these modalities to develop an intelligent sensor-fusion method to detect as many targets as possible.
Right now, SpotlightAI can detect and identify more than 150 types of EO, and it’s also pretty good at generalization—if it encounters a type of land mine it never saw in its training data, it’s likely to identify it as something worthy of attention. It’s familiar with almost all American and Russian munitions, as well as some Israeli and Italian types, and we can make the model more robust by training it on ordnance from elsewhere. As our company grows, we may want to fine-tune our algorithms to offer more customized solutions for different parts of the world. Our current model is optimized for Ukraine and the types of EO found there, but many other countries are still dealing with contamination. Maybe we’ll eventually have separate models for places such as Angola, Iraq, and Laos.
Our hope is that in the next few years, our technology will become part of the standard procedure for demining teams—we want every team to have a drone that maps out surface contamination before anyone sets foot into a minefield. We hope we can make the world safer for these teams, and significantly speed up the pace of releasing land back to the communities living with remnants of war. The best possible outcome will be if someday our services are no longer needed, because explosive devices are no longer scattered across fields and roads. In the meantime, we’ll work every day to put ourselves out of business.
Stephen Cass: Hello and welcome to Fixing the Future, an IEEE Spectrum podcast where we look at concrete solutions to tough problems. I’m your host,Stephen Cass, a senior editor at IEEE Spectrum. And before I start, I just want to tell you that you can get the latest coverage of some of Spectrum’s most important beats, including AI, climate change, and robotics, by signing up for one of our free newsletters. Just go tospectrum.ieee.org/newsletters to subscribe. We’ve been covering the drone delivery companyZipline in Spectrum for several years, and I do encourage listeners to check out our great onsite reporting from Rwanda in 2019 when we visited one ofZipline’s dispatch centers for delivering vital medical supplies into rural areas. But now it’s 2024, and Zipline is expanding into commercial drone delivery in the United States, including into urban areas, and hitting some recent milestones. Here to talk about some of those milestones today, we haveKeenan Wyrobek, Zipline’s co-founder and CTO. Keenan, welcome to the show.
Keenan Wyrobek: Great to be here. Thanks for having me.
Cass: So before we get into what’s going on with the United States, can you first catch us up on how things have been going on with Rwanda and the other African countries you’ve been operating in?
Wyrobek: Yeah, absolutely. So we’re now operating in eight countries, including here in the US. That includes a handful of countries in Africa, as well as Japan and Europe. So in Africa, it’s really exciting. So the scale is really impressive, basically. As we’ve been operating, started eight years ago with blood, then moved into vaccine delivery and delivering many other things in the healthcare space, as well as outside the healthcare space. We can talk a little bit about in things like animal husbandry and other things. The scale is really what’s exciting. We have a single distribution center there that now regularly flies more than the equivalent of once the equator of the Earth every day. And that’s just from one of a whole bunch of distribution centers. That’s where we are really with that operation today.
Cass: So could you talk a little bit about those non-medical systems? Because this was very much how we’d seen blood being parachuted down from these drones and reaching those distant centers. What other things are you delivering there?
Wyrobek: Yeah, absolutely. So start with blood, like you said, then vaccines. We’ve now done delivered well over 15 million vaccine doses, lots of other pharmaceutical use cases to hospitals and clinics, and more recently, patient home delivery for chronic care of things like hypertension, HIV-positive patients, and things like that. And then, yeah, moved into some really exciting use cases and things like animal husbandry. One that I’m personally really excited about is supporting these genetic diversity campaigns. It’s one of those things very unglamorous, but really impactful. One of the main sources of protein around the world is cow’s milk. And it turns out the difference between a non-genetically diverse cow and a genetically diverse cow can be 10x difference in milk production. And so one of the things we deliver is bull semen. We’re very good at the cold chain involved in that as we’ve mastered in vaccines and blood. And that’s just one of many things we’re doing in other spaces outside of healthcare directly.
Cass: Oh, fascinating. So turning now to the US, it seems like there’s been two big developments recently. One is you’re getting close to deploying Platform 2, which has some really fascinating tech that allows packages to be delivered very precisely by tether. And I do want to talk about that later. But first, I want to talk about a big milestone you had late last year. And this was something that goes by the very unlovely acronym of a BVLOS flight. Can you tell us what a BVLOS stands for and why that flight was such a big deal?
Wryobek: Yeah, “beyond visual line of sight.” And so that is basically, before this milestone last year, all drone deliveries, all drone operations in the US were done by people standing on the ground, looking at the sky, that line of sight. And that’s how basically we made sure that the drones were staying clear of aircraft. This is true of everybody. Now, this is important because in places like the United States, many aircraft don’t and aren’t required to carry a transponder, right? So transponders where they have a radio signal that they’re transmitting their location that our drones can listen to and use to maintain separation. And so the holy grail of basically scalable drone operations, of course, it’s physically impossible to have people standing around all the world staring at the sky, and is a sensing solution where you can sense those aircraft and avoid those aircraft. And this is something we’ve been working on for a long time and got the approval for late last year with the FAA, the first-ever use of sensors to detect and avoid for maintaining safety in the US airspace, which is just really, really exciting. That’s now been in operations in two distribution centers here, one in Utah and one in Arkansas ever since.
Cass: So could you just tell us a little bit about how that tech works? It just seems to be quite advanced to trust a drone to recognize, “Oh, that is an actual airplane that’s a Cessna that’s going to be here in about two minutes and is a real problem,” or, “No, it’s a hawk, which is just going about his business and I’m not going to ever come close to it at all because it’s so far away.
Wryobek: Yeah, this is really fun to talk about. So just to start with what we’re not doing, because most people expect us to use either a radar for this or cameras for this. And basically, those don’t work. And the radar, you would need such a heavy radar system to see 360 degrees all the way around your drone. And this is really important because two things to kind of plan in your mind. One is we’re not talking about autonomous driving where cars are close together. Aircraft never want to be as close together as cars are on a road, right? We’re talking about maintaining hundreds of meters of separation, and so you sense it a long distance. And drones don’t have right of way. So what that means is even if a plane’s coming up behind the drone, you got to sense that plane and get out of the way. And so to have enough radar on your drone that you can actually see far enough to maintain that separation in every direction, you’re talking about something that weighs many times the weight of a drone and it just doesn’t physically close. And so we started there because that’s sort of where we assumed and many people assume that’s the place to start. Then looked at cameras. Cameras have lots of drawbacks. And fundamentally, you can sort of-- we’ve all had this, you taken your phone and tried to take a picture of an airplane and you look at the picture, you can’t see the airplane. Yeah. It takes so many pixels of perfectly clean lenses to see an aircraft at a kilometer or two away that it really just is not practical or robust enough. And that’s when we went back to the drawing board and it ended up where we ended up, which is using an array of microphones to listen for aircraft, which works very well at very long distances to then maintain separation from those other aircraft.
Cass: So yeah, let’s talk about Platform 2 a little bit more because I should first explain for listeners who maybe aren’t familiar with Zipline that these are not the kind of the little purely sort of helicopter-like drones. These are these fixed wing with sort of loiter capability and hovering capabilities. So they’re not like your Mavic drones and so on. These have a capacity then for long-distance flight, which is what it gives them.
Wyrobek: Yeah. And maybe to jump into Platform 2— maybe starting with Platform 1, what does it look like? So Platform 1 is what we’ve been operating around the world for years now. And this basically looks like a small airplane, right? In the industry referred to as a fixed-wing aircraft. And it’s fixed wing because to solve the problem of going from a metro area to surrounding countryside, really two things matter. Your range and long range and low cost. And a fixed-wing aircraft over something that can hover has something like an 800% advantage in range and cost. And that’s why we did fix wing because it actually works for our customers for their needs for that use case. Platform 2 is all about, how do you deliver to homes and in metro areas where you need an incredible amount of precision to deliver to nearly every home. And so Platform 2—we call our drone zips—our drone, it flies out to the delivery site. Instead of floating a package down to a customer like Platform 1 does, it hovers. Platform 2 hovers and lowers down what we call a droid. And so the droids on tether. The drone stays way up high, about 100 meters up high, and the drone lowers down. And the drone itself-- sorry, the droid itself, it lowers down, it can fly. Right? So you think of it as like the tether does the heavy lifting, but the droid has fans. So if it gets hit by a gust of wind or whatnot, it can still stay very precisely on track and come in and deliver it to a very small area, put the package down, and then be out of there seconds later.
Cass: So let me get this right. Platform 2 is kind of as a combo, fixed wing and rotor wing. It’s like a VTOL like that. I’m cheating here a little bit because my colleague Evan Ackerman has a great Q&A on the Spectrum website with you, some of your team members aboutthe nitty-gritty of how that design was evolved. But first off, it’s like a little droid thing at the end of the tether. How much extra precision do all those fans and stuff give you?
Wyrobek: Oh, massive, right? We can come down and hit a target within a few centimeters of where we want to deliver, which means we can deliver. Like if you have a small back porch, which is really common, right, in a lot of urban areas to have a small back porch or a small place on your roof or something like that, we can still just deliver as long as we have a few feet of open space. And that’s really powerful for being able to serve our customers. And a lot of people think of Platform 2 as like, “Hey, it’s a slightly better way of doing maybe a DoorDash-style operation, people in cars driving around.” And to be clear, it’s not slightly better. It’s massively better, much faster, more environmentally friendly. But we have many contracts for Platform 2 in the health space with US Health System Partners and Health Systems around the world. And what’s powerful about these customers in terms of their needs is they really need to serve all of their customers. And this is where a lot of our sort of-- this is where our engineering effort goes is how do you make a system that doesn’t just kind of work for some folks, and they can use it if they want to, but a health system is like, “No, I want this to work for everybody in my health network.” And so how do we get to that near 100 percent serviceability? And that’s what this droid really enables us to do. And of course, it has all these other magic benefits too. It makes some of the hardest design problems in this space much, much easier. The safety problem gets much easier by keeping the drone way up high.
Cass: Yeah, how high is Platform 2 hovering when it’s doing its deliveries?
Wyrobek: About 100 meters, so 300 plus feet, right? We’re talking about high up as a football field is long. And so it’s way up there. And it also helps with things like noise, right? We don’t want to live in a future where drones are all around us sounding like swarms of insects. We want drones to make no noise. We want them to just melt into the background. And so it makes that kind of problem much easier as well. And then, of course, the droid gets other benefits where for many products, we don’t need any packaging at all. We can just deliver the product right onto a table in your porch. And not just from a cost perspective, but again, from— we’re all familiar with the nightmare of packaging from deliveries we get. Eliminating packaging just has to be our future. And we’re really excited to advance that future.
Cass: From Evan’s Q&A, I know that a lot of effort went into making the droid element look rather adorable. Why was that so important?
Wryobek: Yeah, I like to describe it as sort of a cross between three things, if you kind of picture this, like a miniature little fan boat, right, because it has some fan, a big fan on the back, looks like a little fan boat, combined with sort of a baby seal, combined with a toaster. It sort of has that look to it. And making it adorable, there’s a bunch of sort of human things that matter, right? I want this to be something that when my grandmother, who’s not a tech-savvy, gets these deliveries, it’s approachable. It doesn’t come off as sort of scary. And when you make something cute, not only does it feel approachable, but it also forces you to get the details right so it is approachable, right? The rounded corners, right? This sounds really benign, but a lot of robots, it turns out if you bump into them, they scratch you. And we want you to be able to bump into this droid, and this is no big deal. And so getting the surfaces right, getting them— the surface is made sort of like a helmet foam. If you can picture that, right? The kind of thing you wouldn’t be afraid to touch if it touched you. And so getting it both to be something that feels safe, but is something that actually is safe to be around, those two things just matter a lot. Because again, we’re not designing this for some piloty kind of low-volume thing. Our customers want this in phenomenal volume. And so we really want this to be something that we’re all comfortable around.
Cass: Yeah, and one thing I want to pull out from that Q&A as well is it was an interesting note, because you mentioned it has three fans, but they’re rather unobtrusive. And the original design, you had two big fans on the sides, which was very great for maneuverability. But you had to get rid of those and come up with a three-fan design. And maybe you can explain why that was so.
Wryobek: Yeah, that’s a great detail. So the original design, the picture, it was like, imagine the package in the middle, and then kind of on either side of the package, two fans. So when you looked at it, it kind of looked like— I don’t know. It kind of looked like the package had big mouse ears or something. And when you looked at it, everybody had the same reaction. You kind of took this big step back. It was like, “Whoa, there’s this big thing coming down into my yard.” And when you’re doing this kind of user testing, we always joke, you don’t need to bring users in if it already makes you take a step back. And this is one of those things where like, “That’s just not good enough, right, to even start with that kind of refined design.” But when we got the sort of profile of it smaller, the way we think about it from a design experiment perspective is we want to deliver a large package. So basically, the droid needs to be as sucked down as small additional volume around that package as possible. So we spent a lot of time figuring out, “Okay, how do you do that sort of physically and aesthetically in a way that also gets that amazing performance, right? Because when I say performance, what I’m talking about is we still need it to work when the winds are blowing really hard outside and still can deliver precisely. And so it has to have a lot of aero performance to do that and still deliver precisely in essentially all weather conditions.
Cass: So I guess I just want to ask you then is, what kind of weight and volume are you able to deliver with this level of precision?
Wryobek: Yeah, yeah. So we’ll be working our way up to eight pounds. I say working our way up because that’s part of, once you launch a product like this, there’s refinement you can do overtime on many layers, but eight pounds, which was driven off, again, these health use cases. So it does basically 100 percent of what our health partners need to do. And it turns out it’s, nearly 100 percent of what we want to do in meal delivery. And even in the goods sector, I’m impressed by the percentage of goods we can deliver. One of our partners we work with, we can deliver over 80 percent of what they have in their big box store. And yeah, it’s wildly exceeding expectations on nearly every axis there. And volume, it’s big. It’s bigger than a shoebox. I don’t have a great-- I’m trying to think of a good reference to kind of bring it to life. But it looks like a small cooler basically inside. And it can comfortably fit a meal for four to give you a sense of the amount of food you can fit in there. Yeah.
Cass: So we’ve seen this history of Zipline in rural areas, and now we’re talking about expanding operations in more urban areas, but just how urban? I don’t imagine that we’ll see the zip lines of zooming around, say, the very hemmed-in streets, say, here in Midtown Manhattan. So what level of urban are we talking about?
Wryobek: Yeah, so the way we talk about it internally in our design process is basically we call three-story sprawl. Manhattan is the place where when we think of New York, we’re not talking about Manhattan, but most of the rest of New York, we are talking about it, right? Like the Bronx, things like that. We just have this sort of three stories forever. And that’s a lot of the world out here in California, that’s most of San Francisco. I think it’s something like 98 percent of San Francisco is that. If you’ve ever been to places like India and stuff like that, the cities, it’s just sort of this three stories going for a really long way. And that’s what we’re really focused on. And that’s also where we provide that incredible value because that’s also matches where the hardest traffic situations and things like that can make any other sort of terrestrial on-demand delivery be phenomenally late.
Cass: Well, no, I live out in Queens, so I agree there’s not much skyscrapers out there. Although there are quite a few trees and so on, but at the same time, there’s usually some sort of sidewalk availability. So is that kind of what you’re hoping to get into?
Wyrobek: Exactly. So as long as you’ve got a porch with a view of the sky or an alley with a view of the sky, it can be literally just a few feet, we can get in there, make a delivery, and be on our way.
Cass: And so you’ve done this preliminary test with the FAA, the BVLOS test, and so on. How close do you think you are to, and you’re working with a lot of partners, to really seeing this become routine commercial operations?
Wyrobek: Yeah, yeah. So at relatively limited scale, our operations here in Utah and in Arkansas that are leveraging that FAA approval for beyond visual line-of-sight flight operations, that’s been all day, every day now since our approval last year. With Platform 2, we’re really excited. That’s coming later this year. We’re currently in the phase of basically massive-scale testing. So we now have our production hardware and we’re taking it through a massive ground testing campaign. So this picture dozens of thermal chambers and five chambers and things like that just running to really both validate that we have the reliability we need and flush out any issues that we might have missed so we can address that difference between what we call the theoretical reliability and the actual reliability. And that’s running in parallel to a massive flight test campaign. Same idea, right? We’re slowly ramping up the flight volume as we fly into heavier conditions really to make sure we know the limits of the system. We know its actual reliability and true scaled operations so we can get the confidence that it’s ready to operate for people.
Cass: So you’ve got Platform 2. What’s kind of next on your technology roadmap for any possible platform three?
Wyrobek: Oh, great question. Yeah, I can’t comment on platform three at this time, but. And I will also say, Zipline is pouring our heart into Platform 2 right now. Getting Platform 2 ready for this-- the way I like to talk about this internally is today, we fly about four times the equator of the Earth in our operations on average. And that’s a few thousand flights per day. But the demand we have is for more like millions of flights per day, if not beyond. And so on the log scale, right, we’re halfway there. Three hours of magnitude down, three more zeros to come. And the level of testing, the level of systems engineering, the level of refinement required to do that is a lot. And there’s so many systems from weather forecasting to our onboard autonomy and our fleet management systems. And so to highlight one team, our system test team run by this really impressive individual namedJuan Albanell, this team has taken us from where we were two years ago, where we had shown the concept at a very prototype stage of this delivery experience, and we’ve done the first order math kind of on the architecture and things like that through the iterations in test to actually make sure we had a drone that could actually fly in all these weather conditions with all the robustness and tolerance required to actually go to this global scale that Platform 2 is targeting.
Cass: Well, that’s fantastic. Well, I think there’s a lot more to talk about to come up in the future, and we look forward to talking with Zipline again. But for today, I’m afraid we’re going to have to leave it there. But it was really great to have you on the show, Keenan. Thank you so much.
Wyrobek: Cool. Absolutely, Stephen. It was a pleasure to speak with you.
Cass: So today on Fixing the Future, we were talking with Zipline’s Keenan Wyrobek about the progress of commercial drone deliveries. For IEEE Spectrum, I’m Stephen Cass, and I hope you’ll join us next time.
Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.
I think suggesting that robots can’t fall is much less useful than instead suggesting that robots can fall and get quickly and easily get back up again.
Sanctuary AI says that this video shows Phoenix operating at “human-equivalent speed,” but they don’t specify which human or under which conditions. Though it’s faster than I would be, that’s for sure.
This is the RAM—robotic autonomous mower. It can be dropped anywhere in the world and will wake up with a mission to make tall grass around it shorter. Here is a quick clip of it working on the Presidio in SF.
This year, our robots braved a Finnish winter for the first time. As the snow clears and the days get longer, we’re looking back on how our robots made thousands of deliveries to S Group customers during the colder months.
Adopting omnidirectional Field of View (FoV) cameras in aerial robots vastly improves perception ability, significantly advancing aerial robotics’s capabilities in inspection, reconstruction, and rescue tasks. We propose OmniNxt, a fully open-source aerial robotics platform with omnidirectional perception.
The MAkEable framework enhances mobile manipulation in settings designed around humans by streamlining the process of sharing learned skills and experiences among different robots and contexts. Practical tests confirm its efficiency in a range of scenarios, involving different robots, in tasks such as object grasping, coordinated use of both hands in tasks, and the exchange of skills among humanoid robots.
We conducted trials of Ringbot outdoors on a 400 meter track. With a power source of 2300 milliamp-hours and 11.1 Volts, Ringbot managed to cover approximately 3 kilometers in 37 minutes. We commanded its target speed and direction using a remote joystick controller (Steam Deck), and Ringbot experienced five falls during this trial.
As with every single cooking video, there’s a lot of background prep that’s required for this robot to cook an entire meal, but I would utterly demolish those fries.
Here’s everything you need to know about Wing delivery drones, except for how much human time they actually require and the true cost of making deliveries by drone, because those things aren’t fun to talk about.
This CMU Teruko Yata Memorial Lecture is by Agility Robotics’ Jonathan Hurst, on “Human-Centric Robots and How Learning Enables Generality.”
Humans have dreamt of robot helpers forever. What’s new is that this dream is becoming real. New developments in AI, building on foundations of hardware and passive dynamics, enable vastly improved generality. Robots can step out of highly structured environments and become more human-centric: operating in human spaces, interacting with people, and doing some basic human workflows. By connecting a Large Language Model, Digit can convert natural language high-level requests into complex robot instructions, composing the library of skills together, using human context to achieve real work in the human world. All of this is new—and it is never going back: AI will drive a fast-following robot revolution that is going to change the way we live.
Rapid and resourceful technological improvisation has long been a mainstay of warfare, but the war in Ukraine is taking it to a new level. This improvisation is most conspicuous in the ceaselessly evolving struggle between weaponized drones and electronic warfare, a cornerstone of this war.
Weaponized civilian first-person-view (FPV) drones began dramatically reshaping the landscape of the war in the summer of 2023. Prior to this revolution, various commercial drones played critical roles, primarily for intelligence, surveillance, and reconnaissance. Since 2014, the main means of defending against these drones has been electronic warfare (EW), in its many forms. The iterative, lethal dance between drones and EW has unfolded a rich technological tapestry, revealing insights into a likely future of warfare where EW and drones intertwine.
After the invasion of Crimea, in 2014, Ukrainian forces depended heavily on commercial off-the-shelf drones, such as models from DJI, for reconnaissance and surveillance. These were not FPV drones, for the most part. Russia’s response involved deploying military-grade EW systems alongside law-enforcement tools like Aeroscope, a product from DJI that allows instant identification and tracking of drones from their radio emissions. Aeroscope, while originally a standard tool used by law enforcement to detect and track illegal drone flights, soon revealed its military potential by pinpointing both the drone and its operator.
On both sides of the line you’ll find much the same kind of people doing much the same thing: hacking.
This application turned a security feature into a significant tactical asset, providing Russian artillery units with precise coordinates for their targets—namely, Ukrainian drone operators. To circumvent this vulnerability, groups of Ukrainian volunteers innovated. By updating the firmware of the DJI drones, they closed the backdoors that allowed the drones to be tracked by Aeroscope. Nevertheless, after the start of the conflict in Crimea, commercial, off-the-shelf drones were considered a last-resort asset used by volunteers to compensate for the lack of proper military systems. To be sure, the impact of civilian drones during this period was not comparable to what occurred after the February 2022 invasion.
As Russia’s “thunder-run” strategy became bogged down shortly after the invasion, Russian forces found themselves unexpectedly vulnerable to civilian drones, in part because most of their full-scale military EW systems were not very mobile.
During a training exercise in southern Ukraine in May 2023, a drone pilot maneuvered a flier to a height of 100 meters before dropping a dummy anti-tank grenade on to a pile of tires. The test, pictured here, worked—that night the pilot’s team repeated the exercise over occupied territory, blowing up a Russian armored vehicle. Emre Caylak/Guardian/eyevine/Redux
The Russians could have compensated by deploying many Aeroscope terminals then, but they didn’t, because most Russian officers at the time had a dismissive view of the capabilities of civilian drones in a high-intensity conflict. That failure opened a window of opportunity that Ukrainian armed-forces units exploited aggressively. Military personnel, assisted by many volunteer technical specialists, gained a decisive intelligence advantage for their forces by quickly fielding fleets of hundreds of camera drones connected to simple yet effective battlefield-management systems. They soon began modifying commercial drones to attack, with grenade tosses and, ultimately, “kamikaze” operations. Besides the DJI models, one of the key drones was the R18, an octocopter developed by the Ukrainian company Aerorozvidka, capable of carrying three grenades or small bombs. As casualties mounted, Russian officers soon realized the extent of the threat posed by these drones.
How Russian electronic warfare evolved to counter the drone threat
By spring 2023, as the front lines stabilized following strategic withdrawals and counteroffensives, it was clear that the nature of drone warfare had evolved. Russian defenses had adapted, deploying more sophisticated counter-drone systems. Russian forces were also beginning to use drones, setting the stage for the nuanced cat-and-mouse game that has been going on ever since.
The modular construction of first-person-view drones allowed for rapid evolution to enhance their resilience against electronic warfare.
For example, early on, most Russian EW efforts primarily focused on jamming the drones’ radio links for control and video. This wasn’t too hard, given that DJI’s OcuSync protocol was not designed to withstand dense jamming environments. So by April 2023, Ukrainian drone units had begun pivoting toward first-person-view (FPV) drones with modular construction, enabling rapid adaptation to, and evasion of, EW countermeasures.
The Russian awakening to the importance of drones coincided with the stabilization of the front lines, around August 2022. Sluggish Russian offensives came at a high cost, with an increasing proportion of casualties caused directly or indirectly by drone operators. By this time, the Ukrainians were hacking commercial drones, such as DJI Mavics, to “anonymize” them, rendering Aeroscope useless. It was also at this time that the Russians began to adopt commercial drones and develop their own tactics, techniques, and procedures, leveraging their EW and artillery advantages while attempting to compensate for their delay in combat-drone usage.
On 4 March, a Ukrainian soldier flew a drone at a testing site near the town of Kreminna in eastern Ukraine. The drone was powered by a blue battery pack and carried a dummy bomb.David Guttenfelder/The New York Times/Redux
Throughout 2023, when the primary EW tactic employed was jamming, the DJI drones began to fall out of favor for attack roles. When the density of Russian jammer usage surpassed a certain threshold, DJI’s OcuSync radio protocol, which controls a drone’s flight direction and video, could not cope with it. Being proprietary, OcuSync’s frequency band and power are not modifiable. A jammer can attack both the control and video signals, and the drone becomes unrecoverable most of the time. As a result, DJI drones have lately been used farther from the front lines and relegated mainly to roles in intelligence, surveillance, and reconnaissance. Meanwhile, the modular construction of FPVs allowed for rapid evolution to enhance their resilience against EW. The Ukraine war greatly boosted the world’s production of FPV drones; at this point there are thousands of FPV models and modifications.
A “kamikaze” first-person-view drone with an attached PG-7L round, intended for use with an RPG-7 grenade launcher, is readied for a mission near the town of Horlivka, in the Donetsk region, on 17 January 2024. The drone was prepared by a Ukrainian serviceman of the Rarog UAV squadron of the 24th Separate Mechanized Brigade.Inna Varenytsia/Reuters/Redux
As of early 2024, analog video signals are the most popular option by far. This technology offers drone operators a brief window of several seconds to correct the drone’s path upon detecting interference, for example as a result of jamming, before signal loss. Additionally, drone manufacturers have access to more powerful video transmitters, up to 5 watts, which are more resistant to jamming. Furthermore, the 1.2-gigahertz frequency band is gaining popularity over the previously dominant 5.8-GHz band due to its superior obstacle penetration and because fewer jammers are targeting that band.
However, the lack of encryption in analog video transmitter systems means that a drone’s visual feed can be intercepted by any receiver. So various mitigation strategies have been explored. These include adding encryption layers and using digital-control and video protocols such as HDZero, Walksnail, or, especially, any of several new open-source alternatives.
In the war zone, the most popular of these open-source control radio protocols is ExpressLRS, or ELRS. Being open-source, ELRS not only offers more affordable hardware than its main rival, TBS Crossfire, it is also modifiable via its software. It has been hacked in order to use frequency bands other than its original 868 to 915 megahertz. This adaptation produces serious headaches for EW operators, because they have to cover a much wider band. As of March 2024, Ukrainian drone operators are performing final tests on 433-MHz ELRS transmitter-receiver pairs, further challenging prevailing EW methods.
Distributed mass in the transparent battlefield
Nevertheless, the most important recent disruption of all in the drone-versus-EW struggle is distributed mass. Instead of an envisioned blitzkrieg-style swarm with big clouds of drones hitting many closely spaced targets during very short bursts, an ever-growing number of drones are covering more widely dispersed targets over a much longer time period, whenever the weather is conducive. Distributed mass is a cornerstone of the emerging transparent battlefield, in which many different sensors and platforms transmit huge amounts of data that is integrated in real time to provide a comprehensive view of the battlefield. One offshoot of this strategy is that more and more kamikaze drones are directed toward a constantly expanding range of targets. Electronic warfare is adapting to this new reality, confronting mass with mass: massive numbers of drones against massive numbers of RF sensors and jammers.
Ukraine is the first true war of the hackers.
Attacks now often consist of far more commercial drones than a suite of RF detectors or jammers could handle even six months ago. With brute-force jamming, even if defenders are willing to accept high rates of damage inflicted on their own offensive drones, these previous EW systems are just not up to the task. So for now, at least, the drone hackers are in the lead in this deadly game of “hacksymmetrical” warfare. Their development cycle is far too rapid for conventional electronic warfare to keep pace.
But the EW forces are not standing still. Both sides are either developing or acquiring civilian RF-detecting equipment, while military-tech startups and even small volunteer groups are developing new, simple, and good-enough jammers in essentially the same improvised ways that hackers would.
Ukrainian soldiers familiarized themselves with a portable drone jammer during a training session in Kharkiv, Ukraine, on 11 March 2024.Diego Herrera Carcedo/Anadolu/Getty Images
Two examples illustrate this trend. Increasingly affordable, short-range jammers are being installed on tanks, armored personnel carriers, trucks, pickups, and even 4x4s. Although limited and unsophisticated, these systems contribute to drone-threat mitigation. In addition, a growing number of soldiers on the front line carry simple, commercial radio-frequency (RF) scanners with them. Configured to detect drones across various frequency bands, these devices, though far from perfect, have begun to save lives by providing precious additional seconds of warning before an imminent drone attack.
The electronic battlefield has now become a massive game of cat and mouse. Because commercial drones have proven so lethal and disruptive, drone operators have become high-priority targets. As a result, operators have had to reinvent camouflage techniques, while the hackers who drive the evolution of their drones are working on every modification of RF equipment that offers an advantage. Besides the frequency-band modification described above, hackers have developed and refined two-way, two-signal repeatersfor drones. Such systems are attached to another drone that hovers close to the operator and well above the ground, relaying signals to and from the attacking drone. Such repeaters more than double the practical range of drone communications, and thus the EW “cats” in this game have to search a much wider area than before.
Hackers and an emerging cottage industry of war startups are raising the stakes. Their primary goal is to erode the effectiveness of jammers by attacking them autonomously. In this countermeasure, offensive drones are equipped with home-on-jam systems. Over the next several months, increasingly sophisticated versions of these systems will be fielded. These home-on-jam capabilities will autonomously target any jamming emission within range; this range, which is classified, depends on emission power at a rate that is believed to be 0.3 kilometers per watt. In other words, if a jammer has 100 W of signal power, it can be detected up to 30 km away, and then attacked. After these advances allow the drone “mice” to hunt the EW cat, what will happen to the cat?
The challenge is unprecedented and the outcome uncertain. But on both sides of the line you’ll find much the same kind of people doing much the same thing: hacking. Civilian hackers have for years lent their skills to such shady enterprises as narco-trafficking and organized crime. Now hacking is a major, indispensable component of a full-fledged war, and its practitioners have emerged from a gray zone of plausible deniability into the limelight of military prominence. Ukraine is the first true war of the hackers.
The implications for Western militaries are ominous. We have neither masses of drones nor masses of EW tech. What is worse, the world’s best hackers are completely disconnected from the development of defense systems. The Ukrainian experience, where a vibrant war startup scene is emerging, suggests a model for integrating maverick hackers into our defense strategies. As the first hacker war continues to unfold, it serves as a reminder that in the era of electronic and drone warfare, the most critical assets are not just the technologies we deploy but also the scale and the depth of the human ingenuity behind them.
Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.
RoboCup 2024: 17–22 July 2024, EINDHOVEN, NETHERLANDS
Enjoy today’s videos!
We present Human to Humanoid (H2O), a reinforcement learning (RL) based framework that enables real-time, whole-body teleoperation of a full-sized humanoid robot with only an RGB camera. We successfully achieve teleoperation of dynamic, whole-body motions in real-world scenarios, including walking, back jumping, kicking, turning, waving, pushing, boxing, etc. To the best of our knowledge, this is the first demonstration to achieve learning-based, real-time, whole-body humanoid teleoperation.
Legged robots have the potential to traverse complex terrain and access confined spaces beyond the reach of traditional platforms thanks to their ability to carefully select footholds and flexibly adapt their body posture while walking. However, robust deployment in real-world applications is still an open challenge. In this paper, we present a method for legged locomotion control using reinforcement learning and 3D volumetric representations to enable robust and versatile locomotion in confined and unstructured environments.
Improving the safety of collaborative manipulators necessitates the reduction of inertia in the moving part. We introduce a novel approach in the form of a passive, 3D wire aligner, serving as a lightweight and low-friction power transmission mechanism, thus achieving the desired low inertia in the manipulator’s operation.
Robot Era just launched Humanoid-Gym, an open-source reinforcement learning framework for bipedal humanoids. As you can see from the video, RL algorithms have given the robot, called Xiao Xing, or XBot, the ability to climb up and down haphazardly stacked boxes with relative stability and ease.
More than 80% of stroke survivors experience walking difficulty, significantly impacting their daily lives, independence, and overall quality of life. Now, new research from the University of Massachusetts Amherst pushes forward the bounds of stroke recovery with a unique robotic hip exoskeleton, designed as a training tool to improve walking function. This invites the possibility of new therapies that are more accessible and easier to translate from practice to daily life, compared to current rehabilitation methods.
DJI drones work to make the world a better place and one of the ways that we do this is through conservation work. We partnered with Halo Robotics and the OFI Orangutan Foundation International to showcase just how these drones can make an impact.
The aim of the test is to demonstrate the removal and replacement of satellite modules into a 27U CubeSat format using augmented reality control of a robot. In this use case, the “client” satellite is being upgraded and refueled using modular componentry. The robot will then remove the failed computer module and place it in a fixture. It will then do the same with the propellant tank. The robot will then place these correctly back into the satellite.
This video features some of the highlights and favorite moments from the CYBATHLON Challenges 2024 that took place on 2 February, showing so many diverse types of assistive technology taking on discipline tasks and displaying pilots’ tenacity and determination. The Challenges saw new teams, new tasks, and new formats for many of the CYBATHLON disciplines.
Small drones for catastrophic wildfires (ones covering more than [40,470 hectares]) are like bringing a flashlight to light up a football field. This short video describes the major uses for drones of all sizes and why and when they are used, or why not.
DARPA’s Learning Introspective Control (LINC) program is developing machine learning methods that show promise in making that scenario closer to reality. LINC aims to fundamentally improve the safety of mechanical systems—specifically in ground vehicles, ships, drone swarms, and robotics—using various methods that require minimal computing power. The result is an AI-powered controller the size of a cell phone.
In July 2021, Florida deputies arrested a man for shooting down their drone. Last week, a judge sentenced him to four years in federal prison, but not for destruction of property: He was sentenced merely for possession of a firearm as a felon.
According to a criminal complaint filed by an agent with the Department of Alcohol, Tobacco, Firearms and Explosives, deputies with the Lake County Sheriff's Office responded to reports of a possible burglary at a large commercial property in Mount Dora, Florida. Upon arrival, they found a front entry gate that had been "damaged by impact," as well as "unsecured doors on some of the buildings at the facility."
Thinking there could still be intruders present on the 10-acre site, deputies flew an unmanned aerial vehicle over the property to check it out. As it flew, however, they heard two gunshots ring out and saw the drone "slowly spinning downward and emitting smoke" before it "crashed onto the metal roof of an outbuilding, became suspended on a rain gutter, and caught fire."
Deputies investigated the source of the gunshots and encountered Wendell Goney, who lived nearby. Goney first denied shooting the drone but admitted to it after learning the craft was equipped with cameras. Goney claimed this was the first time he had ever shot at a drone, but he told deputies that he had been "harassed" by people flying them over his property. He said he had bought a .22 caliber rifle to put a stop to it and hadn't known that this particular drone belonged to law enforcement.
When deputies asked, Goney admitted that he was a convicted felon and therefore ineligible to own a firearm under federal law. They arrested Goney, who gave them permission to search his house and told them exactly where to find the rifle.
The criminal complaint noted that Goney "has approximately 23 prior felony convictions. He has been sentenced to more than a year in state prison on multiple occasions, including 1997, 2007, 2009, and 2013." His offenses included forgery, burglary, and multiple instances of grand theft, including the theft of a rifle in 1995. Most recently, he was sentenced in 2013 to two years and eight months in prison on charges including simple battery, aggravated assault of a law enforcement officer, and possession of cocaine.
Goney pled guilty in October 2023. Last week, U.S. District Court Judge Gregory Presnell sentenced Goney to four years in prison, followed by three years of supervised release during which he would have to participate in both drug and mental health treatment programs. He would also be required to pay more than $29,000 in restitution to the Lake County Sheriff's Office to replace the drone.
While Goney was initially charged with both destruction of law enforcement property and possession of a firearm, prosecutors dropped the former charge in exchange for his plea. Noting in their sentencing memo that federal guidelines "call for a recommended sentencing range of between 77–96 months," prosecutors asked for "a sentence at the low end of the applicable sentencing guidelines—77 months." The memo noted that "such a sentence would be slightly more than twice the length of the longest of the prison sentences the defendant has served to date."
But is that a just sentence? Goney's previous longest sentence was two years and eight months for crimes involving assaulting other people, including police; in this case, on the other hand, Goney's only aggressive act was toward an unmanned flying drone, and his only charged offense was possessing a gun without the government's permission.
While it seems sensible to bar felons from owning guns—especially those with such unsavory records as Goney's—the act presents uncomfortable questions of both liberty and constitutionality.
For one, the federal ban on firearm ownership necessarily sweeps up people who have committed nonviolent offenses, even those not involving firearms. In 1995, Bryan Range fraudulently obtained $2,458 in welfare benefits; when caught, Range repaid the money, paid a small fine, and served three years of probation. But under federal law, Range also lost the right to own a firearm as a result of his state misdemeanor conviction. In June 2023, the U.S. Court of Appeals for the 3rd Circuit ruled that Range's ban violated the Second Amendment's guarantee of a right to keep and bear arms.
Unlike Range, Goney's prior convictions did involve the use of violence. But who is supposed to be protected by keeping Goney from owning a firearm? At the time of his arrest, Goney had not been in trouble with the law for several years, and his only act of aggression was aimed at a drone he thought was "harassing" him.
The original criminal complaint notes that Goney is a felon and "has never had his civil rights restored by executive clemency following these felony convictions." Indeed, the only way for Goney to legally regain his Second Amendment right to own a firearm is through a pardon by the governor.
People convicted of crimes—even felonies, even those involving violence—do not cease to be citizens, or human beings, as a result of their sentences. Once their sentences have been served, they should have the ability to regain the rights they've lost—including the rights to vote, to hold public office, and to serve on juries. The right to bear arms is just as essential.
It’s kind of astonishing how quadrotors have scaled over the past decade. Like, we’re now at the point where they’re verging on disposable, at least from a commercial or research perspective—for a bit over US $200, you can buy a little 27-gram, completely open-source drone, and all you have to do is teach it to fly. That’s where things do get a bit more challenging, though, because teaching drones to fly is not a straightforward process. Thanks to good simulation and techniques like reinforcement learning, it’s much easier to imbue drones with autonomy than it used to be. But it’s not typically a fast process, and it can be finicky to make a smooth transition from simulation to reality.
New York University’s Agile Robotics and Perception Lab in collaboration with the Technology Innovation Institute (TII) have managed to streamline the process of getting basic autonomy to work on drones, and streamline it by a lot: The lab’s system is able to train a drone in simulation from nothing up to stable and controllable flying in 18 seconds flat on a MacBook Pro. And it actually takes longer to compile and flash the firmware onto the drone itself than it does for the entire training process.
ARPL NYU
So not only is the drone able to keep a stable hover while rejecting pokes and nudges and wind, but it’s also able to fly specific trajectories. Not bad for 18 seconds, right?
One of the things that typically slows down training times is the need to keep refining exactly what you’re training for, without refining it so much that you’re only training your system to fly in your specific simulation rather than the real world. The strategy used here is what the researchers call a curriculum (you can also think of it as a sort of lesson plan) to adjust the reward function used to train the system through reinforcement learning. The curriculum starts things off being more forgiving and gradually increasing the penalties to emphasize robustness and reliability. This is all about efficiency: Doing that training that you need to do in the way that it needs to be done to get the results you want, and no more.
There are other, more straightforward, tricks that optimize this technique for speed as well. The deep-reinforcement learning algorithms are particularly efficient, and leverage the hardware acceleration that comes along with Apple’s M-series processors. The simulator efficiency multiplies the benefits of the curriculum-driven sample efficiency of the reinforcement-learning pipeline, leading to that wicked-fast training time.
This approach isn’t limited to simple tiny drones—it’ll work on pretty much any drone, including bigger and more expensive ones, or even a drone that you yourself build from scratch.
Jonas Eschmann
We’re told that it took minutes rather than seconds to train a policy for the drone in the video above, although the researchers expect that 18 seconds is achievable even for a more complex drone like this in the near future. And it’s all open source, so you can, in fact, build a drone and teach it to fly with this system. But if you wait a little bit, it’s only going to get better: The researchers tell us that they’re working on integrating with the PX4 open source drone autopilot. Longer term, the idea is to have a single policy that can adapt to different environmental conditions, as well as different vehicle configurations, meaning that this could work on all kinds of flying robots rather than just quadrotors.
Přihlásit se
This early version of the Hover flying camera generated a lot of initial excitement, but never fully took off.Zero Zero Robotics
Santiago Tenorio holds one of Lime Microsystems’ private 5G base-station kits.Vodafone
Arthur Erickson inspects the company’s largest spray drone, the AG-272. It can cover thousands of hectares per day.Hylio
Unexploded ordnance lies by the road in a Ukrainian town near the war’s front lines. John Moore/Getty Images
During a training exercise in southern Ukraine in May 2023, a drone pilot maneuvered a flier to a height of 100 meters before dropping a dummy anti-tank grenade on to a pile of tires. The test, pictured here, worked—that night the pilot’s team repeated the exercise over occupied territory, blowing up a Russian armored vehicle. Emre Caylak/Guardian/eyevine/Redux
On 4 March, a Ukrainian soldier flew a drone at a testing site near the town of Kreminna in eastern Ukraine. The drone was powered by a blue battery pack and carried a dummy bomb.David Guttenfelder/The New York Times/Redux
A “kamikaze” first-person-view drone with an attached PG-7L round, intended for use with an RPG-7 grenade launcher, is readied for a mission near the town of Horlivka, in the Donetsk region, on 17 January 2024. The drone was prepared by a Ukrainian serviceman of the Rarog UAV squadron of the 24th Separate Mechanized Brigade.Inna Varenytsia/Reuters/Redux
Ukrainian soldiers familiarized themselves with a portable drone jammer during a training session in Kharkiv, Ukraine, on 11 March 2024.Diego Herrera Carcedo/Anadolu/Getty Images
