How Amazon One used generative AI to enable payments with your palm

Global tech giant Amazon has announced a new breakthrough in convenient payments that doesn't require a smart watch or any kind of under-the-skin chip as envisaged by sci-fi movies. Just the palm of your hand will suffice.

AWS Applications vice president and distinguished scientist Gerard Medioni says generative artificial intelligence (AI) has been leveraged to "reimagine the future of convenience in shopping, entertainment, and access".

"My team and I like to build experiences for customers that feel like magic. Amazon One, which combines cutting-edge biometrics, optical engineering, generative AI, and machine learning, fits the bill perfectly," says Medioni.

"Amazon One is a fast, convenient, and contactless experience that enables customers to leave their wallets (and even their phones) at home. 

"They can instead use the palm of their hand for everyday activities like paying at a store, presenting a loyalty card, verifying their age, or entering a venue."

Amazon One is being rolled out to more than 500 US locations of Whole Foods Market, a high-end retailer that Amazon purchased in 2017 for $13.4 billion, as well as in dozens of third-party locations including travel retailers, sports and entertainment venues, convenience stores, and grocers. 

"At each location, you will see a small Amazon One device—a type of scanner that uses infrared light—to recognize the unique lines, grooves, and ridges on your palm, and the pulsating network of veins just under the skin," notes Medioni.

"The system uses this information to create your palm signature, or a unique numerical vector representation, and connects it to your credit card or your Amazon account.

"But the magic and ease of the Amazon One experience belies the complexity of the problem we had to solve. Amazon One comes down to one simple point: the system can’t make a mistake."

He explains that generative AI is a subset of traditional machine learning powered by models trained on billions of data points from books, articles, pictures, and other sources. Large language model (LLM) like ChatGPT are examples of this, but in Amazon One's case the process involved producing millions of synthetic images of palms to create a "palm factory" to train the AI model.

"The industry term for this computer-generated information is “synthetic data,” new data created by the AI to replicate the breadth and variety of real data as closely as possible," he says.

"This was pioneering work that happened years before the current generative AI craze started dominating conversations around the world."

The model was trained with various subtle changes such as illumination conditions, hand poses, and even the presence of a Band-Aid.

"But that’s not all. The images were also automatically “annotated,” which is normally a long and laborious process," he says.

"This saved time and allowed us to move faster, as we didn’t have to label the pictures and tell the computer that it was looking at the pattern lines of your palm, a scar, or a wedding band.

"We also trained our system to detect fake hands, such as a highly detailed silicon hand replica, and reject them.

"The combination of the palm surface and subcutaneous images allowed us to build a system that’s 100 times more accurate than two irises."

He clarifies that Amazon One was also designed to protect customer privacy, operating beyond the normal light spectrum and cannot accurately perceive gender or skin tone. 

"Amazon One also does not use palm information to identify a person, only to match a unique identity with a payment instrument," he says.