Enterprise Engineering
Meta
AI personal sales assistant to improve effective engagement rate
My role
Founding designer for the AI chatbot on Meta.com
Timeline
4 months
May 2023 - Aug 2023
Keywords
AI agent / system design / chatbot components
My scope
My overall mission working under store discovery team is to inform and excite people about owning Meta's VR products on Meta.com
In 2023 summer, I worked as a product design intern on The Stores team, aiming at building the end-to-end shopping experience for Consumer Engineering. Our team is responsible for all systems involved in bringing our VR/AR products to market - from product setup and consumer-facing e-commerce storefronts, to the platforms that manage the demand, production, testing, and shipping across the entire supply chain.
Project metrics - Impact
Primary metrics
Conversion rate
Effective engagement rate and total time spent
Adoption rate
Tracking metrics
General engagements
Quality
Guardrail metrics
Bounce rate
Performance
About
Project brief
This genAI chat assistant presents a tailored solution to enhance the user experience by responding to their questions at all shopping phases 24/7, with goals to 1) educate the Meta skeptics, 2) monetize existing customers, and 3) reduce human agent costs
Painpoints
Low add-to-cart and conversion rates
Pages are too long and lack engaging sales experiences, leading to high bounce rates and low add-to-cart rates.
User questions not efficiently resolved
Key information is buried under images and text, making it hard for users to find answers quickly.
High cost of human agents
24/7 human agents are expensive and have limited capacity, resulting in slow response times.
The Solution
Build a unified conversational agent, Virtual Assistant (Chatbot), to provide assistance and support to users in the overall E2E user journey on Meta Store.
Progress
This project was at the 0-1 stage when GenAI’s capability was ambiguous in the 2C field during the summer of 2023. I took on the design and technical research as the founding designer from day one, aiming to provide more direction and context for the team.
Design
User flow & Logic - from entering the agent chatbot to user shopping problem resolved
Chatbot components — since this project was in a very early stage and the UX was actually uncertain, I identified the main use case and shipped the components early to benefit future implementation and testing.
Context
Meta.com
Meta.com is our 1st party site to discover Meta's revolutionary technology from virtual reality to social experiences. It's a e-commerce shopping site for Meta Quest, Ray-Ban Stories, games, and VR accessories. It's also a crucial medium to educate and convert VR enthusiasts and skeptics.
User personas
The Shopper: High intent visitors, taking up only 3% - 10% of MDC (Meta.com) traffic
The Browser: Low to mid intent visitors, taking up 90% - 97% of MDC traffic
Both data and user insights have indicated that the Shopper and Browser needs are not being met by pages on meta.com due to the content low discoverability.
Business needs of balancing the use frequency of human agents VS AI
Due to the inherent unreliability of LLMs in providing accurate and up-to-date answers, we need to hand over the conversation to a human agent at some point.
However, how do we define when and at which trigger point the human agent should take over the user flow, perfectly balancing the cost and the service quality?
Research + High Level Specifications
Toward narrowing the AI chatbot functions, I adopted a two-step research process:
1) User flow touch points across all shopping phases
2) GenAI (LLMs) framework - the functional mechanism of the data flow
1) 5-Phase user flow on MDC- Awareness, Consideration, Payment, Support, and Loyalty
Quantitative data insights:
Low to mid intent browsers : get them excited and increase purchase intent
High intent shoppers: remove purchase barriers and convert them
2) MDC GenAI (LLMs) framework design
Framework analysis Insights:
After diving into the framework, I identified there're three key touch points that can improve the chatbot functionality - Prompt quality, Domain specific data, and Feedback data.
It's challenging to input efficient prompts - minimize the user textual input needs
User feedback data is crucial for model improvement flywheel
Key features for MVP
To surface the key design features according the 1) user motivation 2) LLMs technical break points, I came up with this opportunity sheets and shared it on the weekly meeting with my manager, engineers, pm, and data scientists.
The key design opportunities listed in this chart will ingest into the user flow to connect the dots!
Guidelines of the high level AI chatbot interaction design specifications
Interaction type
Q&A interaction
Default prompt questions
User input questions
CTA, eg., add to cart, go to cart to checkout.
Escalation and live agent handover
Content
Tech Specs, Discounts, Product content, Marketing pages, Help center pages, blog, Apps/Games, high quality UGC content from Meta.com…
Persistent Context
For logged-in users, if the same customer returns to meta.com/help at a later date, the bot should recognize the customer’s identity and be able to retrieve previous interactions. Guest user should be able to see the chat history in the current general session.
Support type
Help purchasing a product: provide more information about a product
Order help: check the status of an order you placed, or make changes to it?
Device related post sales support
User feedback for the experiences
Thumb up or thumbs down on each interaction;
General satisfaction feedback and suggestions.
Business Configuration
For prescriptive answer flows (where a single, known path to resolution exists) business admins should have tools to edit, add, or remove bot flows that do not require engineering intervention.
Design Development
To start my design process, I broke the solution into 3 steps of consideration:
1) User flow
Pre-purchase / post-purchase / proximal problem spaces
2) Entry points spec
Mobile / web / vibe / visual icon / spacing spec
3) Handoff doc spec
Errors, Warnings, and Edge Cases / asset specs / motion / animation / visual
Design Development -User flow
Pre-purchase AI chatbot decision tree architecture
Post-purchase AI chatbot decision tree architecture
User flow - pre-purchase
Pre-purchase AI chatbot UX flow
1/
Identify the entry point - as a floating button or show up on a sticky tab.
Define the default prompt card content.
Define the extreme case - only clicking on the default prompt without manual input; what would it be like?
Define the extreme case of the handover point - after three rounds of answer generation, if the user is still not satisfied, the chatbot will hand over the session to a human agent.
2/
Define the scenario of no user reaction.
After 20 seconds of not detecting any user reaction, we will provide a follow-up question to close the loop and guide users to provide feedback or offer an editable prompt to further explore their needs.
3/
We provide the key attributes of Meta headset and VR games in pills as a straightforward way to excite and engage users.
Introduce the option to 'view other questions' to generate more prompts and save users the effort of manual input.
Limit the answers as short as possible, e.g., 200 words and append relevant links to direct traffic to PDPs and other pages.
Provide product cards instead of simple links to offer multi-layered rich information.
4/
Break the purchasing barrier and excite users - Provide several selected discount cards & links upon the user's request.
Option 1 - All discounted items are placed in cards in a carousel enabling scrolling or the 'see all' expansion. The cards are info-rich and persuasive but take up too much interface real estate.
Option 2 - Only list one product in a card, and all the rest in links in a chatbox. This design is simple, doesn't take up too much space, and prioritizes according to our business needs. The downside is its less intuitve and visually refreshing.
4-1/
Expand the discounts and other important attributes toward a specified product
5/
Close the loop
1) Save logged in users chat history to better understand users.
2) Logged-out browsers are able to see their current session chat history but unable to trace back their former ones.
User flow - post -purchase
Post-purchase AI chatbot UX flow
Beyond the key features mentioned above, I dive into the top questions bring up by users in their post-purchase shopping phase as for the context. The most frequently asked questions are 1)direct number to call the human agent, 2) questions around the order - check the status, resolve issues, etc…
Here're a few key design interventions that I brought up in the post-purchase flow:
1) Default prompt with 3 case scenarios to direct customers to either making a purchase again, check an existing order, or getting support
2) Log-in to process the order details
3) Auto surface the most recent order and ask customers if they need any help on that
4) Provide relevant answers in both text and links
5) Recommend relevant accessories, games, discounted products
1/
Provide default options for customers back to the chat flow
Guide them to login or signup to track their order
2/
After users log in, the bot automatically provides them with the latest order info.
If this is what users want, it saves them effort from going back and forth providing detailed info.
If users have other questions, they can simply type them in manually.
3/
Provide order details with links
Recommend relevant accessories, games, discounted products
4/
User manually input their order number. We not only provide with the status and link
We also provide with default prompts to expedite the solution
5/
If a customer's issue remains unresolved after two rounds of interaction with the bot, we will hand it over to a human agent.
Entry points spec + handoff
Entry points explorations 1) floating action button 2) Sticky entry on nav bar
1) Floating action button
Pros:
1. Discoverability - visible and accessible throughout the web pages
2. A sense of real-time support
3. Space for motion design and color pattern exploration
Cons:
Cognitive load
Takes up extra mobile real estate
Accidental engagement
1-1) Desktop Expanding status exploration
2) Sticky entry on nav bar
Pros:
1. Not visually disruptive as it stays parallel with the shopping cart and account icons.
2. Doesn't take up extra mobile real estate as it stays on the nav bar.
Cons:
1. Not catchy enough as its color style aligns with the shopping cart and account icons.
2. Static, lacking potential for motion/color exploration.
3. Lack of a sense of real-time intelligent support.
3) Entry points handoff specs
Handoff asset specs
Handoff documentation
1) Assets overall look
2) Chatting page specs - Mobile + Desktop
3) Product + Game cards
4) Chatting page assets
5) Mobile mockup
Proximal problem spaces
There're 3 proximal spaces that are nice to have for the next phase development - 1) Chat history 2) Shopping bag 3) Voice input
1) Chat history
It can benefits the logged-in users to trace back to their past queries
Nudge the unregistered browsers to create their own meta account
Allow users to delete and manage their own data: Respecting users' privacy and giving them control over their data is crucial for building trust and enhancing the overall user experience.
2) Shopping bag - in the chat input section
The shopping bag enables users to access all the Meta products all at once without shifting to the mainpage,
which provides them with a comprehensive context to review and make comparison in a easier and holistic way
This clears the blocker of composing a detailed question in mind by just sending the product cards to trigger more product details
This can improve the adoption rate
3) Voice input
As mobile forms the major source of the traffic to Meta.com the UX of input of the mobile version is highly priotrized
This can avoid accidental touch and facilitate an easier input experience
Impact
1) The AI chatbot expedites the search and questions answer speed that impact the user experience of over 500k visitors on Meta.com per day
Benefits over 360,000 searches every year - 1-2% of these visitors search i.e. there are about 5k-10k searches per day. This adds up to as much as 30,000 searches every month, and over 360,000 searches every year.
Increase the adoption rate on apps and games related content - 28% of the searches - which is more than 100,000 searches - are for apps and games
2) AI chatbot as a mean of receiving important signals to improve the web experience
Outcomes from these subtle in-chat experiments won’t just help us in making a better, and more persuasive meta.com, but also give us valuable signals about ways to improve the content and layout of various pages on meta.com. For example, if we notice that framing responses in certain ways lead to conversion, we’d know what sequence of information can work harder in persuading the visitor to convert
Further, the kinds of questions that visitors ask will also tell us about which parts of the web experience need to be addressed for them to give the most relevant information for the visitor
Over time, all of this will help us in designing a better experience on meta.com