Hello, Robot!

Hello, Robot!

By Ahti Heinla, co-founder and CTO at Starship Technologies

I see robots every day. I see them gliding down the sidewalks at pedestrian speed, stopping to make sure it’s safe to cross the road. Sometimes I even catch them talking with pedestrians. It’s a glimpse into the fantasies of the technologically minded — an AI wonderland. But this is no hallucination, no dream, it’s a reality that our team of dedicated visionaries have built over the past 5 years; we’ve brought the future into the now.

As recently as a couple of years ago, these robots needed a bit of human support and were accompanied on their journeys, much like the format autonomous car makers follow, who test their cars in public using ‘safety drivers’.

Starship became the first robotics team to begin operating regularly in public spaces about 18 months ago, without the use of safety drivers; we let our robots explore the world on their own. We now operate our network of robots every day in several cities around the world, bringing people their dinner, parcels and groceries.

Knowledge shared is knowledge gained

It’s exciting to be the first.

Back when I was a founding engineer at Skype, we were the first to make Voice over IP accessible in a practical way; we are now working to do the same with robots in public spaces. For four years, our engineering teams have worked behind closed doors on what has been a significant breakthrough and an amazing experience.

I want to share some of the details from our technical journey with you. Over the coming weeks and months, other members of the Starship engineering team will share aspects of their journey too.

Along this journey we have worked with computer vision, path planning and obstacle detection — subjects that are well-researched within the area of academic robotics. Indeed, Starship started as a research project, but soon transitioned into a functioning, practical delivery operation.

This means that in addition to fine-tuning the Levenberg-Marquardt algorithm for non-linear optimization, we’ve had to develop software to:

• Automatically calibrate most of our sensors — after all, we don’t want to spend hours calibrating them by hand; we have manufactured hundreds of robots and are currently preparing for a larger scale operation.
• Predict how much energy each trip will draw from a robot’s battery — so we can orchestrate which robot to send, based on their battery status.
• Predict how many minutes it takes for a restaurant to prepare the food — so the robot shows up just in time!

Most autonomous robots that exist in the world today are expensive, they are built as technology demonstrators or research vehicles and not used for commercial operations. A sensor package alone for an autonomous device can cost upwards of $10,000. This simply won’t work in the delivery space, it’s not a luxury industry where you can charge a premium.

Autonomous driving research vehicles often have 3 kilowatts of computing power in the trunk; impractical for a small, safe delivery robot. Therefore, part of our engineering journey has been about designing for lower unit economics. Here are some topics we had to consider:

• Advanced image processing on a lower end computational platform.
• Working around hardware issues in software.
• Tracking how often robots need maintenance, and why.
• Developing advanced route planning systems, to make sure we’re using our network of robots efficiently.

It has been quite a journey in visual design too, involving hundreds of sketches, drawings and surveys before we manufactured our robot’s first plastic body.

Back in the early days when we were still in stealth mode, we did not want to reveal how our robots looked. Regular public testing required the creative use of a trash bag, taped over the body of the robot as a disguise!

Building practical robotics is a blend of science, systematic engineering and hackery. This mix of various disciplines is the core characteristic of Starship. Nothing is ever simple in robotics. All your knowledge of the situation is probabilistic; all sensors have failure modes and faults, and even a seemingly simple task such as making the robot stop at obstacles can become its own small research project.

Starship is a fast-moving startup business and it’s important not to become just a large research project. The engineers who get excited about Starship are often not pure scientists, not pure hackers, not pure engineers; they have several of these traits and can use them as is appropriate to the task at hand. We need complicated technical solutions to be implemented quickly and within the resource constraints of low-cost hardware.

Ingenuity and resourcefulness are valued skills.

A week is a long time at Starship

At the beginning of the week our team will implement a new algorithm to detect curbs from point clouds and get it back-tested against a full test case database overnight, they’ll have it tested live on our private test-ground by the end of the week.

It’ll be on the streets the next Monday, with the team already reporting on their progress during our Monday Engineering Meeting. Most Mondays some of the engineering team are reporting a 300%+ gain on at least one of the metrics that was achieved, just the week before.

Data as the result and facilitator of scale

Metrics and data have become a big part of Starship engineering.

You see, back when we were just starting out we had no data — we hadn’t driven much yet. Every day we modified our robot (yep, just the one back then), took it to the sidewalks and saw how it performed. We now have many, driving around autonomously every day — too many for engineers to observe directly.

Thanks to the data, we can now see how our robots perform, hundreds of them. We can organize weekly ‘data dive’ seminars, where engineers share findings and watch random deliveries to stay in touch with their work in action.

When we are working on making our robots drive more smoothly, we analyze the data in our Data Warehouse’s ‘acceleration events’ table; there are at least 1 billion rows in that table. Other tables include ‘road crossing events’, our maps, every command each robot has ever received from our servers, and obviously data collected from each delivery they perform.

Four years ago, we had none of this. Back when we were just starting out — and were not running commercial deliveries yet — I often had to convince people that robotic delivery really works. People found it difficult to believe and were quick to point out a variety of reasons why.

Do skepticism and fear always accompany new technology?

Several years back, I landed at New York’s JFK airport with a robot in my luggage. The customs guy obviously asked: “What is this thing?” I explained that it was a sidewalk delivery robot, to which he replied: “Dude, this is New York! It will get stolen within minutes!”

Indeed, back then almost everyone thought these robots would get stolen — I’m sure they probably will be (postal delivery vans get stolen, even if rarely). To date our robots have driven over 200,000km (130,000 miles) and we have yet to see that problem.

There are of course security features in place. The robot has a siren and 10 cameras, it is constantly connected to Internet and knows its precise location with an accuracy of 2cm (thanks to the above mentioned Levenberg-Marquardt algorithm, and the 66,000 lines of automatically-generated C++ code that enables our robots to use it).

People also thought pedestrians might be fearful of robots on the sidewalk or might not accept their presence. Will people call the police? To be honest, we weren’t sure about this either! However, once we put one of the robots out there onto the sidewalk, we were in for quite a surprise.

What happened next surprised us: people just ignored it. The vast majority of the public didn’t pay any attention whatsoever to the robots, even those seeing it for the first time, and people certainly weren’t scared. Others would pull out their phones and post to Instagram about how they’ve seen the future.

And that’s what we wanted.

We want people to pay just as much attention to our robots as they do to their dishwashers. This pattern of silently accepting robots as if they’d always been with us has repeated itself in every city around the world we have operated in.

It gets better. Once people learn that these robots provide a useful service to the neighborhood, they develop an affinity with them. Kids even write letters thanking the robots, we have a ‘wall of thank you letters’ to prove it!

Automating last-mile delivery was never going to be easy, and we knew it would be an audacious project. We also knew all along that there would be more than one fundamental roadblock that would need solving — there turned out to be hundreds of roadblocks! But we realized long ago that all these problems are solvable — they just required ingenuity and persistence.

Some startups begin like running a sprint, throwing together a Minimum Viable Product in 3 months. For Starship it’s more like a marathon — a large consistent effort is needed, but the end result brings huge benefits to the world.

Last mile delivery is one of the world’s industries that has seen little technological disruption since the adoption of the automobile. The team at Starship are on a quest to change that, and with more than 20,000 deliveries under our belt, we are well on our way.

If you’re interested in learning more, check out our second Engineering blog post on Neural Networks and how they power our robots here — https://medium.com/starshiptechnologies/how-neural-networks-power-robots-at-starship-3262cd317ec0