Mohammed Khan spent the last decade building a career at Amazon’s fulfillment centers, helping get everything from books to bookshelves shipped to customers. Now as an assistant general manager of a fulfillment center in Tracy, California, Khan knows what it takes to pick, sort, and pack thousands of packages every day. And Khan understands how to make sure that products arrive undamaged.
“When I joined Amazon, I started off picking products for customer orders and gaining a hands-on understanding of what goes into getting the right product to our customers,” Khan said.
Khan explained that as a product goes through fulfillment center operations, up to five different employees use a six-point visual check to assess whether products it’s damaged. He said it’s a time-consuming task that’s often hard to keep top of mind, because employees rarely find damaged items in Amazon’s inventory.
That’s why a team of scientists at Amazon Fulfillment Technologies in Berlin, Germany, are working hard to help Khan and his colleagues. The researchers are developing advanced artificial intelligence (AI) capabilities that can spot irregularities and flag defective products before they ship.
AWS announces Amazon Bedrock and 3 more generative AI innovations
Services from AWS combine decades of machine learning experience across Amazon with breakthrough technologies poised to transform just about everything.
Read more
Like other sophisticated AI tools, the damage-detecting technology relies on software and lots of data. But damaged products are very rare, which makes the data needed to train the AI scarce. It’s that scarcity of data and the vast scale of Amazon’s diverse inventory that has made AI-driven damage detection so challenging—until now. Last year, the research team determined that they could supply a machine learning model with reference images to teach it how to compare the product it’s “looking at” to an image of what the product should look like. To accomplish this, they use computer vision to scan every item that passes through their warehouse just outside of the German capital. Next, a machine learning model analyzes the scans to discover hidden patterns and continuously improve the system’s ability to detect damage. This approach to machine learning and computer vision gives AI the capability to make the types of subjective decisions about damage that humans make all the time.
“We started by training our machine learner to build its own mental model of item appearance by looking at millions of example images of damaged and undamaged items,” said Jeremy Wyatt, director of Applied Science at Amazon Robotics. “Then, in operation, we show it an image of the specific ‘query’ item and a previous ‘gallery’ image of the same product and the AI model compares the images. This was one of several approaches we took to unlock a big performance improvement.”
Christoph Schwerdtfeger, applied science manager at Amazon Fulfillment Technologies, said that the AI system is three times more effective than manually identifying damaged products. Due to the success, plans are being made to bring the system to other facilities.
“Of course, this technology is only focused on this task, while our operations employees have so much going on each and every day. We plan to expand the use of this technology beyond our testing sites,” said Schwerdtfeger. “We want to deploy our damage-detection software at a dozen operations in North America and Europe before the holiday season. Once in place, the technology will help scan for damage on over 40 million customer products every month, and it will be part of how we make sure people get undamaged gifts this holiday season.”
While the machine learning model will continue to improve as its dataset grows, Schwerdtfeger was quick to point out that the true power of the technology is in the way it helps humans work.
“These models are helpful, but sometimes, they get fooled,” Schwerdtfeger added. “If that happens, we receive immediate feedback from operations employees in Amazon’s fulfillment network. Our damage experts directly teach the AI how to make better decisions in the future. It is this collaboration between humans and machines that leads to much better results for our customers.”
What generative AI means for businesses and how AWS can help
The new AWS Generative AI Innovation Center helps customers successfully build and deploy custom generative AI products and services.
Khan is keen to see how this technology will help drive efficiency in fulfillment centers.
“This tool has the potential to streamline tasks and help us manage cost and delivery time for our customers,” said Khan. “It’s also exciting because of how it can help free up operations employees to stay focused on other core tasks and activities, especially important responsibilities like safety.”
In addition to installing the system in more locations, the Amazon Fulfillment Technologies team is planning to expand the system’s capabilities.
“As we look to future applications of this technology, one possibility is to move beyond the detection of damage before we ship an order,” said Wyatt. “For example, we may be able to identify when and where the damage occurred in the first place.”