Enterprise Engineering
Meta
AI personal sales assistant to improve effective engagement rate
My role
Product Design Intern
@Store discovery team
Timeline
4 months
May 2023 - Aug 2023
Keywords
Web design / Mobile design / Conversational AI chatbot design
Asset design / Accessibility / Scalability
My scope
My overall mission working under store discovery team is to inform and excite people about owning Meta's VR products on Meta.com
In 2023 summer, I worked as a product design intern on The Stores team, aiming at building the end-to-end shopping experience for Consumer Engineering. Our team is responsible for all systems involved in bringing our VR/AR products to market - from product setup and consumer-facing e-commerce storefronts, to the platforms that manage the demand, production, testing, and shipping across the entire supply chain.
My metrics - Impact
Primary metrics
Conversion rate
Effective engagement rate and total time spent
Adoption rate
Tracking metrics
General engagements
Quality
Guardrail metrics
Bounce rate
Performance
01 /
Meta Store
conversational AI personal sales assistant
Jun 16 - Aug 2
Mobile & web UI/UX
Conversational AI
See my design
About
Project brief
This genAI chat assistant presents a tailored solution to enhance the user experience by responding to their questions at all shopping phases 24/7, with goals to 1) educate the Meta skeptics, 2) monetize existing customers, and 3) reduce human agent costs
Painpoints
Meta.com is facing challenges of increasing the add to cart rate and conversion rate with no effective engaging sales experience
Most of our web pages are long and required a lot of scrolling - PDPs showed a bounce rate of %. Only % of page viewers effectively engaged with page contents and % added items to cart.
Questions from Meta.com visitors aren't being efficiently resolved.
Important info like ‘what’s included’, ‘ how to dive into Metaverse using the headset’, ‘perks and discounts’ are buried under images and texts.
Human agents are expensive - Reduce the cost of sales
Meta.com currently only adopt real people agent to respond customers' questions through all the shopping phases 7/24. However, humans agents are expensive and have limited bandwidth, which can lead to long lead times when we need support
It’s difficult to generate high-quality, original content simply using GenAI
Current GenAI can respond with inaccurate and distorted answers, which dilutes mutual trust with Meta's service quality. Furthermore, GenAI answers are typically lengthy with no key points and can result in excessive auto-scrolling, posing a significant drawback in user experience.
The Solution
Build a unified conversational agent, Virtual Assistant (Chatbot), to provide assistance and support to users in the overall E2E user journey on Meta Store.
Outcome
Delivered scalable and reusable GenAI chatbot asset repository to improve productivity of engineers
By identifying user touchpoints and pain points on Meta.com through pre-purchase, purchase, post-purchase, and loyalty enhancement phases, I designed multiple user flows with the introduction of interchangeable and scalable AI chatbot assets leveraging the Meta Design System applicable on both web and mobile.
Enhanced customer experience on Meta.com and improve efficiency
The chatbot boosts conversion rates by helping customers make purchase decisions instantly, providing accurate responses to inquiries, and reducing response times. It enhances customer support, automates tasks, and gathers insights on user preferences.
Paved the way of the standalone GenAI chatbot being an unmentioned part of a bigger enterprise product
During this transitional phase of widely define GenAI touch points in enterprise scenarios, my work contribute to the collective exploration to test known conventions and uncover opportunities.
Context
Meta.com
Meta.com is our 1st party site to discover Meta's revolutionary technology from virtual reality to social experiences. It's a e-commerce shopping site for Meta Quest, Ray-Ban Stories, games, and VR accessories. It's also a crucial medium to educate and convert VR enthusiasts and skeptics.
User personas
The Shopper: High intent visitors, taking up only 3% - 10% of MDC (Meta.com) traffic
The Browser: Low to mid intent visitors, taking up 90% - 97% of MDC traffic
Both data and user insights have indicated that the Shopper and Browser needs are not being met by pages on meta.com due to the content low discoverability.
Business needs of balancing the use frequency of human agents VS AI
Due to the inherent unreliability of LLMs in providing accurate and up-to-date answers, we need to hand over the conversation to a human agent at some point.
However, how do we define when and at which trigger point the human agent should take over the user flow, perfectly balancing the cost and the service quality?
Research + High Level Specifications
Toward narrowing the AI chatbot functions, I adopted a two-step research process:
1) User flow touch points across all shopping phases
2) GenAI (LLMs) framework - the functional mechanism of the data flow
1) 5-Phase user flow on MDC- Awareness, Consideration, Payment, Support, and Loyalty
Quantitative data insights:
Low to mid intent browsers : get them excited and increase purchase intent
High intent shoppers: remove purchase barriers and convert them
2) MDC GenAI (LLMs) framework design
Framework analysis Insights:
After diving into the framework, I identified there're three key touch points that can improve the chatbot functionality - Prompt quality, Domain specific data, and Feedback data.
It's challenging to input efficient prompts - minimize the user textual input needs
User feedback data is crucial for model improvement flywheel
Key features for MVP
To surface the key design features according the 1) user motivation 2) LLMs technical break points, I came up with this opportunity sheets and shared it on the weekly meeting with my manager, engineers, pm, and data scientists.
The key design opportunities listed in this chart will ingest into the user flow to connect the dots!
Guidelines of the high level AI chatbot interaction design specifications
Interaction type
Q&A interaction
Default prompt questions
User input questions
CTA, eg., add to cart, go to cart to checkout.
Escalation and live agent handover
Content
Tech Specs, Discounts, Product content, Marketing pages, Help center pages, blog, Apps/Games, high quality UGC content from Meta.com…
Persistent Context
For logged-in users, if the same customer returns to meta.com/help at a later date, the bot should recognize the customer’s identity and be able to retrieve previous interactions. Guest user should be able to see the chat history in the current general session.
Support type
Help purchasing a product: provide more information about a product
Order help: check the status of an order you placed, or make changes to it?
Device related post sales support
User feedback for the experiences
Thumb up or thumbs down on each interaction;
General satisfaction feedback and suggestions.
Business Configuration
For prescriptive answer flows (where a single, known path to resolution exists) business admins should have tools to edit, add, or remove bot flows that do not require engineering intervention.
Design Development
To start my design process, I broke the solution into 3 steps of consideration:
1) User flow
Pre-purchase / post-purchase / proximal problem spaces
2) Entry points spec
Mobile / web / vibe / visual icon / spacing spec
3) Handoff doc spec
Errors, Warnings, and Edge Cases / asset specs / motion / animation / visual
Design Development -User flow
Pre-purchase AI chatbot decision tree architecture
This decision tree model user flow illustrates 5 steps as a full interaction loop to solve customer questions, which shows the logic of how the chatbot fetch the answer from the prescriptive answer pool
1) Discover the chatbot
2) Identify the interested products / Manual input
3) Identify the interested services within a certain product
4) Possible interactions
5) Questions resolved - close the loop
Post-purchase AI chatbot decision tree architecture
The post-purchase AI chatbot focuses on
1) Solve users' post-purchase problems
2) Provide attentive post-purchase service
2) Monetize existing customers. Recommend complementary accessories, device upgrade plans, and new product recommendations.
User flow - pre-purchase
Pre-purchase AI chatbot UX flow
1/
Identify the entry point - as a floating button or show up on a sticky tab.
Define the default prompt card content.
Define the extreme case - only clicking on the default prompt without manual input; what would it be like?
Define the extreme case of the handover point - after three rounds of answer generation, if the user is still not satisfied, the chatbot will hand over the session to a human agent.
2/
Define the scenario of no user reaction.
After 20 seconds of not detecting any user reaction, we will provide a follow-up question to close the loop and guide users to provide feedback or offer an editable prompt to further explore their needs.
3/
We provide the key attributes of Meta headset and VR games in pills as a straightforward way to excite and engage users.
Introduce the option to 'view other questions' to generate more prompts and save users the effort of manual input.
Limit the answers as short as possible, e.g., 200 words and append relevant links to direct traffic to PDPs and other pages.
Provide product cards instead of simple links to offer multi-layered rich information.
4/
Break the purchasing barrier and excite users - Provide several selected discount cards & links upon the user's request.
Option 1 - All discounted items are placed in cards in a carousel enabling scrolling or the 'see all' expansion. The cards are info-rich and persuasive but take up too much interface real estate.
Option 2 - Only list one product in a card, and all the rest in links in a chatbox. This design is simple, doesn't take up too much space, and prioritizes according to our business needs. The downside is its less intuitve and visually refreshing.
4-1/
Expand the discounts and other important attributes toward a specified product
5/
Close the loop
1) Save logged in users chat history to better understand users.
2) Logged-out browsers are able to see their current session chat history but unable to trace back their former ones.
User flow - post -purchase
Post-purchase AI chatbot UX flow
Beyond the key features mentioned above, I dive into the top questions bring up by users in their post-purchase shopping phase as for the context. The most frequently asked questions are 1)direct number to call the human agent, 2) questions around the order - check the status, resolve issues, etc…
Here're a few key design interventions that I brought up in the post-purchase flow:
1) Default prompt with 3 case scenarios to direct customers to either making a purchase again, check an existing order, or getting support
2) Log-in to process the order details
3) Auto surface the most recent order and ask customers if they need any help on that
4) Provide relevant answers in both text and links
5) Recommend relevant accessories, games, discounted products
1/
Provide default options for customers back to the chat flow
Guide them to login or signup to track their order
2/
After users log in, the bot automatically provides them with the latest order info.
If this is what users want, it saves them effort from going back and forth providing detailed info.
If users have other questions, they can simply type them in manually.
3/
Provide order details with links
Recommend relevant accessories, games, discounted products
4/
User manually input their order number. We not only provide with the status and link
We also provide with default prompts to expedite the solution
5/
If a customer's issue remains unresolved after two rounds of interaction with the bot, we will hand it over to a human agent.
Entry points spec + handoff
Entry points explorations 1) floating action button 2) Sticky entry on nav bar
1) Floating action button
Pros:
1. Discoverability - visible and accessible throughout the web pages
2. A sense of real-time support
3. Space for motion design and color pattern exploration
Cons:
Cognitive load
Takes up extra mobile real estate
Accidental engagement
1-1) Desktop Expanding status exploration
2) Sticky entry on nav bar
Pros:
1. Not visually disruptive as it stays parallel with the shopping cart and account icons.
2. Doesn't take up extra mobile real estate as it stays on the nav bar.
Cons:
1. Not catchy enough as its color style aligns with the shopping cart and account icons.
2. Static, lacking potential for motion/color exploration.
3. Lack of a sense of real-time intelligent support.
3) Entry points handoff specs
Handoff asset specs
Handoff documentation
1) Assets overall look
2) Chatting page specs - Mobile + Desktop
3) Product + Game cards
4) Chatting page assets
5) Mobile mockup
Proximal problem spaces
There're 3 proximal spaces that are nice to have for the next phase development - 1) Chat history 2) Shopping bag 3) Voice input
1) Chat history
It can benefits the logged-in users to trace back to their past queries
Nudge the unregistered browsers to create their own meta account
Allow users to delete and manage their own data: Respecting users' privacy and giving them control over their data is crucial for building trust and enhancing the overall user experience.
2) Shopping bag - in the chat input section
The shopping bag enables users to access all the Meta products all at once without shifting to the mainpage,
which provides them with a comprehensive context to review and make comparison in a easier and holistic way
This clears the blocker of composing a detailed question in mind by just sending the product cards to trigger more product details
This can improve the adoption rate
3) Voice input
As mobile forms the major source of the traffic to Meta.com the UX of input of the mobile version is highly priotrized
This can avoid accidental touch and facilitate an easier input experience
Impact
1) The AI chatbot expedites the search and questions answer speed that impact the user experience of over 500k visitors on Meta.com per day
Benefits over 360,000 searches every year - 1-2% of these visitors search i.e. there are about 5k-10k searches per day. This adds up to as much as 30,000 searches every month, and over 360,000 searches every year.
Increase the adoption rate on apps and games related content - 28% of the searches - which is more than 100,000 searches - are for apps and games
2) AI chatbot as a mean of receiving important signals to improve the web experience
Outcomes from these subtle in-chat experiments won’t just help us in making a better, and more persuasive meta.com, but also give us valuable signals about ways to improve the content and layout of various pages on meta.com. For example, if we notice that framing responses in certain ways lead to conversion, we’d know what sequence of information can work harder in persuading the visitor to convert
Further, the kinds of questions that visitors ask will also tell us about which parts of the web experience need to be addressed for them to give the most relevant information for the visitor
Over time, all of this will help us in designing a better experience on meta.com
To wrap up this 3-month design internship
Photo of the Burlingame office, shoot on Jun 2023
Some takeaways…
1/ Gained the capibility of getting comfortable being uncomfortable
It's common that even when working within an established system, we are constantly dealing with ambiguity as we identify the next correct user needs and determine technical boundaries. In this navigation process, the ability to advocate for our design with careful and logical thinking becomes a must-have skill.
2/ Being hypersensitive to the problem is one of the most valuable, intangible, and long-lasting qualities
The ability to identify the right problems and provide feasible solutions is an advanced trait that can only be cultivated in business settings where real-world problems are dealt with.