Amazon Lookout for Vision
Amazon Lookout for Vision is a computer vision machine learning service that spot product defects to automate quality inspection at scale. Customers can upload a set of product images to Lookout for Vision and train ML models, then the model can detect anomalies from the production line. For more about the service:
Users & Pain points
The users of Lookout for Vision are those who build and train the ML models for their specific types of products. The first type of user is a Data Scientist. Data scientists have ML knowledge, but many are not familiar with the CV process, since the CV technology and its adoption are in their early stages yet.
Some customers don’t have a data science team in their organization. In this case, developers in the IT team take the charge. Most of them have a very low level of understanding of ML overall.
* LFV users and their pain points
Design goal & tenets
With the insights of users & pain points, I set the design goal of LFV as “Help customers train accurate ML models regardless of their level of ML or CV expertise.”, and could derive 4 design tenets from the goal.
Simple & Linear flow: Help users understand ML & CV processes easily
No prior ML experience is needed: Don’t overwhelm users
Predictable: Keep users informed along the way with inputs
Rich guidance framed around usee cases
Lookout for Vision user journey
Below is an end-to-end user journey within Lookout for Vision. 1/ Create project: A project is a container of datasets and trained models for a specific use case. 2/ Create dataset: Dataset consists of product images. 3/ Label dataset: User needs to classify product images in the dataset as ‘Normal’ or ‘Anomaly’. These labeled images are used as ingredients for ML model training. 4/ Model training & evaluation: After each model training, the user can look into the performance of the model - basic metrics and test results. 5/ Feedback: Usually to get satisfactory model accuracy, a few iterations of dataset enhancement and re-training are needed.
Design tenet 1. Simple & Linear flow
I put self-guidance materials on major pages that explain the ML training processes step by step. Each step also displays the corresponding status indicator or actions, so it intuitively lets customers know what to do next.
* Ideation, iterations of How it works design
Design tenet 2. No prior ML experience is needed
Dataset is a set of customer’s product images and a crucial ingredient for ML model training. The dataset consists of two different types of datasets, a training dataset, and a test dataset. The ML model trains itself with the images in the training dataset, then evaluates itself with the images in the test dataset. The reason for separation is to prevent biased or overfitting ML training results.
This is one of the basic concepts of ML, and most data scientists are well aware of it. But it could be hard to follow for the other type of users, IT developers without ML knowledge and could be a blocker of using the service.
I suggested providing an option that users create and label a single dataset, and Lookout for Vision separates the dataset into training and test datasets in the backend on behalf of customers. Also, I convinced the team to put this option as a default setting.
Through the usability test I found out the majority of customers - including data scientists - were using the single dataset option as the start point. Customers felt it easier and more convenient.
Design tenet 3. Keep users informed
Lookout for Vision supports model training with a small number of labeled images. Customers can train the first model fast, check how the model performs, and continue enhancement iteratively. I put a progress indication panel during the labeling process, kept customers informed until they reach the minimum required number of labeled images.
Design tenet 4. Rich guidance
Precision, Recall, F1 score are a basic set of ML model performance metrics. Previously AWS AI services displayed these metrics with plain texts. Since Lookout for Vision was supposed to be a service for non-ML experts, I added visualization to each metric and an explanation of how each metric was calculated. This small update also got a lot of positive feedback from customers.
Where we are now
“This is one of the most compelling end to end ML applications. ML contains a considerable amount of complexity, but Lookout for Vision is so simple as user experience.” (Principal developer)
“Easy to follow and user friendly. I like it.” (Data scientist)
Lookout for Vision announced General Availability in February 2021. As of now, a number of large manufacturing customers are going through the POC(Proof of Concept) and pilot stage, and we are expecting many of those will convert to the production stage starting from this year.
