Adventures at Amazon

Information Architecture, Research, Analytics • June 2012 - March 2015

I was an Information Architect in Amazon's Seller Support division from 2012 to 2015. Seller Support runs the sellercentral.amazon.com platform, where more than two million third-party merchants manage and sell their inventory to Amazon customers around the world. This division also operates global contact centers, where call center associates provide support to sellers. 

Important Note: Due to the sensitive nature of projects and in accordance with my Non-Disclosure Agreement, project specifics have been withheld, and images have been redacted or obscured.  Below, I've provided high-level overviews of a few big initiatives I worked on.

Setting the Stage

During those three years, I worked on a wide variety of projects. As is typical, Information Architecture in Seller Support fit between Amazon's users, content, and the overarching context of the business.
Occasionally, projects would require that I dabble more heavily in one of these areas, like UX Design, Content Strategy, and Process Optimization. Given my quantitative skill set, many of these projects had an analytical bent; and from 2014-2015, I managed the analytics program for the content development team in Seller Support, including capacity and demand planning.

Case study 1:
Patenting a way to gauge content quality

I have a patent, y'all! Check it out.

Some background... because I really like this stufF

As mentioned in other case studies, throughout my career I've relied on my education in mathematics and statistics to provide much-needed balance to the more qualitative aspects in the universe of user experience design.

As an undergrad, I received a grant to do independent research outside of the curriculum offered by my university, and chose to study recommendation algorithms.  I focused my research on book sales on Amazon.com (I was a big nerd and was working at a rare book store at the time).

Over the course of about three months, I developed a salability equation and salability model useful in predicting book sales on Amazon using logistic regression and other statistical techniques. See more here (search for my name). 

As luck would have it, I went on to work at Amazon just a few years later, and was able to apply this knowledge to help my team and contribute to Amazon's body of intellectual property. 

The Problem: help content

Amazon sellers around the world access help articles over 40 million times each year, seeking vital information they need to run their businesses. Amazon has grown tremendously, and with it, the number of help articles has increased at a rate of about 67% each year on a historical basis, and the cost of managing this content-- more than 41,000 help articles localized into 16 languages-- is not insignificant.

Furthermore, this glut of content sometimes led to a poor seller experience, if they were ever to come across an article that was out-of-date, or struggle to find the right article amid a sea of search results.

Luckily, Amazon is a very data-rich environment, and I had access to a multitude of information about how often this web content was being used, how recently it had been edited, seller feedback on the articles, etc.

It was time for a content audit, a common technique employed by Information Architects and Content Strategists. As the department's only information Architect, I was just the right person to take on this task!

The goal of this content audit was to inventory and evaluate the quality and effectiveness of the help articles, and to determine what could be archived. The department was gearing up to move these help articles into a new content management system, which was a daunting task, and one that couldn't be automated. The legacy HTML documents would be moving into a highly structured XML environment governed by a DITA schema in the new CMS. 

Given the manual nature of this type of content migration, it was important to reduce the number of help articles as much as possible. 

Solution: Research and Design

Obviously with this volume of content, a manual audit was out of the question. I collated a number of the data points listed above and categorized the content into performance groups and scored according to a confidential rubric.

Worst-performing articles were reviewed by global content development staff to determine their disposition. I checked in with staff regularly to determine throughput and progress toward our goal. 

result

By the end of the 9-month project timeframe, we successfully identified more than 10,000 help articles to be archived, about 25% of the total. This reduction in content had huge implications for the cost of our manual content migration, and also enabled more efficiencies in localization, search engine performance, and increasing seller success.

I was able to develop my scoring model into a content quality algorithm based on categorical and continuous predictor variables, ensuring that content auditing efforts in the future would be machine-assisted, and much more efficient. 

The value and extensibility of this algorithm did not go unnoticed. I partnered with a few colleagues across Amazon to identify other applications where this algorithm could be used, and later met with patent attorneys at Amazon and signed patent disclosure documents (and received a highly coveted Amazon patent cube!). 

Patent #US 9760607 B1, Calculating document quality was finally granted in September 2017.

Case study 2:
Workflow-Driven Applications

The Problem

Sellers contact Amazon call centers when they encounter problems running their online businesses. Certain types of contacts tend to take longer to resolve, or consistently result in low customer satisfaction metrics.

Many of these contacts are highly complex and require a lot of research and data collection by call center associates (CSAs), which cost our organization money, and provided a less-than-optimal experience to sellers.

This process needed to be streamlined and improved.

Solution: Research and design

In order to better understand the problem, I traveled to a call center in the Midwestern United States in order to conduct user research-- contextual inquiry, observation, and interviews-- with CSAs for one week. 

My main takeaways were:

1) CSAs have an incredibly difficult job, often needing to juggle tens of complex internal systems while engaging and calming frustrated customers on the phone. 

2) Existing documentation and troubleshooting guides were outdated, incomplete, or poorly written.

Upon returning to Seattle, I helped to organize a team of content developers to rewrite existing documentation in a structured, consistent format based on policy and best practices.

I worked with these content developers to develop logical user flows for each process, and they would go on to rewrite documentation. 

I validated the logic for each process with business SMEs and the CSAs, and then built very simple workflows in Axure and tested them with CSAs. Once feedback had been integrated, I worked with an engineering team to develop and implement the workflow-driven applications into the existing contact management framework.

result

For the types of contacts where CSAs could use these workflow-driven applications, average time to resolution metrics were cut nearly in half, and customer satisfaction metrics increased by more than 8%.

There were growing pains, however. Rewriting documentation was neither a quick nor easy undertaking, especially for processes that required global input.

It also became clear that creating workflow-driven applications for end users would eliminate the need for sellers to contact call centers in the first place, thereby saving Amazon money and improving the seller's experience.

At this point, we redoubled our efforts to move these applications upstream, and expose these workflows to sellers, so they wouldn't need to create contacts for certain issues.