A fleet of 100 small drones lifts off and murmurates like a flock of electronic starlings over a nighttime festival crowd of 100,000 gathered on the banks of the Danube. The stage is Linz, Austria, home to the annual Ars Electronica Festival, the world’s largest festival celebrating the intersection of art, science, and technology.
Each quadcopter carries a colored LED light. For 10 minutes, the swarm performs a complex aerial dance, forming 3D wireframe pictures of abstract shapes, moving images, and even words in constantly shifting colors. Then the quads make their exit, settling back on the launch ground in quiet unison.
This drone performance echoes the natural phenomena known as swarm intelligence—the ways insects like bees, fireflies, and ants function as a collective—an emerging focus in artificial intelligence (AI) research. In the Ars Electronica project, each drone represents a point of light—a pixel. But these are SPAXELS, pixels in 3D space.
In 2012, the producers asked Ars Electronica’s R&D unit, Futurelab, to contribute a novel performance to the Klangwolke (sound cloud), one of Europe’s largest outdoor events, happening annually in Linz, that had previously featured fireworks and laser shows. That invitation led to the premier SPAXELS performance, says Horst Hörtner, Futurelab’s senior director.
That drone swarm dazzled the crowd, despite its divergence from Futurelab’s original inspiration: The original concept for the Klangwolke involved actual fireflies. “Our first idea was to navigate fireflies through the sky using pheromones,” Hörtner says. “Actually, we got pretty far along using the pheromones to control the bugs. But then we realized that European fireflies only light up in June; the event is in September, so that wasn’t going to work.”
So Hörtner and his team turned to pigeons. “We tried to equip pigeons with LEDs and then teach them to fly in formation,” Hörtner says. “But those animals are crazy. They don’t learn. When you attach something to their feet, they try to fly away from it. So we had to abandon pigeons. That led us to drones. They do whatever we tell them to do, we thought.”
Hörtner and his team started working with off-the-shelf drones but soon realized they’d need to customize the drones and equip them with LEDs. They use Autodesk Fusion 360 and EAGLE for the hardware design and prototyping while also developing Ground-Control, the software necessary to navigate a swarm safely through the sky.
SPAXELS maintain constant contact with Ground-Control, running on a computer at the airfield, while performing choreography designed with Autodesk 3ds Max animation and rendering software. Only a few “flight controllers” are needed to direct the entire swarm—no one is actually piloting the drones; that task is handled by Ground-Control.
A typical show runs from four to 10 minutes. “We want the drone batteries at least half-full when they are landing, for safety reasons,” Hörtner says. “There are practical limits, especially in outdoor shows, where the drones have to fight wind conditions to maintain their positions. That consumes energy quickly. Ten minutes is a practical maximum performance length.”
Futurelab´s first commissioned SPAXELS show came in 2013 for the Paramount Studios London premiere of Star Trek: Into Darkness, adjacent to the Tower Bridge. The largest SPAXELS show to date happened in 2015, using 100 drones for the giant chipmaker Intel. Both Intel and a Chinese company have since flown more than 1,000 drones simultaneously. “Many companies now use the concepts we have published to fly drone shows,” Hörtner says. “The whole concept of SPAXELS in the sky is freely available to anyone.”
Hörtner’s team has developed a library of programmed SPAXELS shows, in which a specific segment of animation can be lifted from one show and incorporated into another, like recombining ballet steps into original choreographies. So far, however, most clients have preferred to develop shows from scratch.
And while it still receives commissions, SPAXELS’ thrust is no longer to perform airborne drone ballets. “We are trying to provide solutions for organizations who want to design their own drone shows,” Hörtner says. To that end, in March, SPAXELS will release its new Swarm3D flight planner.
Swarm3D will allow designers to create collision-avoiding paths for multiple drones. “Whatever your purpose—signage or entertainment or whatever—the software manages the safe navigation of the drones,” Hörtner says. “You can design your preferred formations, and the tool will optimize the paths of all drones, through all your formations.” The idea is that, with Swarm3D, anyone can design and operate a drone show because the paths and behaviors are mathematically secured. One just needs to enter the maximum speed of each drone type and the average failure of the drone-tracking system, and Swarm3D automatically produces valid flight paths.
In addition to being great technology for entertainment (not to mention a sustainable alternative for fireworks), the swarm intelligence behind SPAXELS could have far-reaching benefits beyond light shows—for example, construction-site monitoring, agriculture, utility inspection, logistics, and much more.
In working with swarms, Hörtner and his team realized that the communication within a swarm of drones or robots is quite different than communication between humans and robots. In a recent collaboration with Mercedes-Benz, Futurelab researched the interaction between “an autonomous agent with its own agenda” (think self-driving car) and a pedestrian coming toward it. It’s often difficult enough to avoid running into other humans on the sidewalk, Hörtner says. So how does a human encountering an unmanned vehicle avoid a collision? The collaboration provided important insights into the psychology of that human-vehicle interaction.
At Futurelab, human interaction occurs among 47 people from all different disciplines: artists, architects, sociologists, physicists, psychologists, 3D modelers, industrial designers, mechatronic engineers, and software engineers. What they’ll be working on next is anyone’s guess—but they won’t be training pigeons.