In this first part of a two-part blog series, we demonstrate how generative AI coupled with customer data can help marketing teams generate personalized marketing content.
The holiday season is a key time for US retailers and consumer brands, with many businesses earning 25% of their annual revenue in November and December alone. For specialty brands in toys, games, and hobbies, it can rise to as much as 34.5%. This makes it essential to capture consumer attention well in advance of the critical shopping window. But with so many organizations competing for this attention, how does one stand out and win over customers bombarded with options?
Increasingly, the response is to lean into personalization. Personalized marketing content tailored to the needs, interests, and preferences of individual consumers is widely recognized as an effective way to cut through the noise and grab shoppers’ attention. But while it can improve results compared to other mass marketing approaches, the cost in time and effort of developing content has often constrained how truly personalized it can be.
People often talk about one-to-one marketing, but the truth is that most content is still targeted to segments of hundreds of thousands or even multiple millions of similar consumers. This is simply because marketing teams only have so much capacity to develop content around a campaign.
However, generative AI informed by comprehensive customer data can help marketing teams reach smaller, more specific consumer groups with highly personalized messaging while still aligning with broader campaign goals. While cost and the need to review output from generative technologies may not yet enable true one-to-one marketing, it does have the potential to unlock the next level in personalized marketing, helping organizations better stand out from their competition with more tailored content.
Demonstrating AI-Driven Content Creation
Returning to the holiday season theme, we want to demonstrate how to use generative AI to create a personalized holiday gift-giving guide. These guides, long a staple of holiday marketing, often address broad swaths of consumer interests, so potential customers need to browse for the sections that interest them. This gives us an excellent opportunity to narrow the focus of the content to the needs and interests of a specific gift-giver, making it much more likely the guide will resonate with them.
In this blog series, we will re-imagine the creation and delivery of a digital guide for a fictional sporting goods retailer. Instead of one monolithic guide delivered to customers deemed likely to purchase holiday items from this business, we’ll use information about each recipient’s preferred gift-giving categories and price sensitivity based on their prior browsing and purchase history to produce customized guides for each of the 21 intersections of 7 recognized categories and 3 levels of price sensitivity.
In this first entry in our series, we will address the challenge of getting a generative AI model to produce reliable, high-quality results. In the second blog, we will develop an end-to-end workflow to automate content creation and email delivery.
Using the AI Playground
Developing a set of instructions that will trigger a generative AI model to produce the content we require is known as prompt engineering. Prompt engineering is a highly iterative process, one that requires a lot – and we mean a lot! – of trial and error. Because of the sizable time investment that most organizations will make in this activity, Databricks has released an interactive user interface known as the AI Playground to simplify the process (Figure 1).
As we write this, the AI Playground is pre-configured to make use of the Meta Llama 3.1 70B Instruct model. Using the drop-down at the upper center of the playground screen, you can select other models pre-installed within the Databricks environment or even models pre-configured by your Databricks administrators. You can also choose multiple models to run side-by-side, allowing you to compare the output from a shared prompt. But for this exercise, we will focus on the one, default mode (Figure 2).
In addition to selecting a model, we can also configure a number of options controlling each model’s behavior (Figure 3). We’d suggest leaving these to their defaults during initial prompt development and then later playing with these values to see how they affect your results.
Turning to the prompts, you should note there are two types of prompts typically employed during content creation. You can see options for these two prompts, known as the system prompt and the user prompt, toward the bottom of the AI Playground screen (Figure 4).
The system prompt is typically the overarching instruction about how the LLM should behave. It sets the tone, context and constraints for the tasks the generative AI model is to perform, such as how our holiday guide should be structured. This differs from the user prompt where we often provide the specific details of a particular request, such as the audience for the specific holiday guide we need the model to produce. If this is your first time prompt engineering and it isn’t yet clear how these two types of prompts differ (and how they interact), it should be easier to see as we work through a few examples.
Defining the Prompts
With the orientation to the AI Playground out of the way, let’s use it now to develop our prompts. For the system prompt, let’s specify the following, clicking the Save button to persist the instruction:
Write a piece of creative with a title and a body that uses the user’s preferred price point and preferred holiday gifting subcategory. The focus should be on gifting during the holidays.
For the user prompt, we can provide instructions specific to one of our variants:
Create content for a low price point shopper with a holiday preferred subcategory of Ski.
When we send this instruction to the model, we get back a pretty interesting result (Figure 5).
To make the content more easily readable, here’s the output in plain text, truncated for brevity:
Ski into the Holiday Season without Breaking the Bank
Are you a ski enthusiast on a budget? Do you want to show your loved ones that you care about their winter adventures without overspending? Look no further! We've got you covered with our top picks for affordable ski-themed gifts that won't break the bank.
Under $20:
* A warm and cozy beanie to keep their head and ears toasty on the slopes ($15)
* A set of ski-themed stickers or decals to decorate their water bottle or laptop ($10)
* A gift card to their favorite ski resort or coffee shop, so they can treat themselves to a hot chocolate or a lift ticket ($20) ....
While the response looks coherent, it’s got a few problems. First, it’s way too long. We didn’t put any restrictions in the system prompt on content length but we clearly need to. Second, the model suggested a number of products (with prices!) when it doesn’t in fact have any knowledge of the items in our actual product catalog. We need to either provide information about specific products or ask it to elevate the content to address relevant classes of items we carry.
With that in mind, we updated our system prompt to read as follows:
Write a piece of creative with a title and a body that uses the user’s preferred price point and preferred holiday gifting subcategory. The focus should be on gifting during the holidays. Focus on product categories in the output. Never include specific products in the output. Also, never include prices. Outputs should be brief and concise, no more than approximately 150 words.
The user prompt reads as before:
Create content for a low price point shopper with a holiday preferred subcategory of Ski.
Note: The AI Playground is built to be conversational and will retain memory of prior prompts within the session. To clear the session, you might want to refresh the page before submitting these two new prompts.
With these two new prompts, we have a very different model response:
"Ski-tastic Stocking Stuffers"
Hit the slopes with a smile this holiday season! If you're looking for affordable gifts for the skier in your life, we've got you covered. Our collection of ski-themed treasures is sure to delight, without breaking the bank.
Browse our selection of cozy accessories, such as soft hats, warm gloves, and colorful scarves, perfect for adding a pop of personality to their ski gear. Or, opt for functional favorites like ski-themed water bottles, phone cases, and luggage tags. We also offer a range of fun and quirky ski-inspired decorations, like snowflake ornaments and mountain-themed magnets.
Whatever you choose, it's sure to be a gift that's "peak" perfection!
While the output was better, it was a bit too generic and seemed to be throwing out suggestions for random stuff like phone cases. To address this, we updated the prompt to emphasize that the gift guide pertains to a sporting goods retailer and therefore the focus should be on sporting good products:
System Prompt:
You are writing copy for a sporting goods retailer. Outputs should focus on sporting good products. Write a piece of creative with a title and a body that uses the user's preferred price point and preferred holiday gifting subcategory. The focus should be on gifting during the holidays. Focus on product categories in the output. Never include specific products in the output. Also, never include prices. Outputs should be brief and concise, no more than approximately 150 words.
User Prompt:
Create content for a low price point shopper with a holiday preferred subcategory of Ski.
With this, the response is starting to look better, though there are still some quirks like opening with “Hey thrill-seeker!” suggesting the reader would be shopping for themselves:
Hey thrill-seeker! Are you looking for affordable gifts for the skier in your life? Look no further! Our selection of ski accessories is the perfect way to show you care without breaking the bank. From warm and cozy hats and gloves to functional ski socks and goggles, we've got everything you need to keep your loved one on the slopes all season long. Plus, our ski bags and backpacks are perfect for carrying all their gear in style. Give the gift of a great ski season without sacrificing your holiday budget. Shop our affordable ski accessories today and make this winter one to remember!
And so we continued whittling away at the system prompt, letting the model know the gifts were for someone else, not the recipient of the email. We also found that providing a sample email around which the model could structure its content also helped, and providing that email with HTML formatting helped us produce a result that could easily be transformed to meet our delivery needs. So after several more rounds of iteration, we finally arrived at these prompts:
System Prompt:
You are writing copy for a sporting goods retailer. Outputs should focus on sporting good products. Write a piece of creative with a title and a body that uses the user's preferred price point and preferred holiday gifting subcategory. The focus should be on gifting during the holidays. The person receiving the content is the gifter. They might not be personally interested in the featured subcategory. Focus on product categories in the output. Never include specific products in the output. Also, never include prices. Outputs should be brief and concise, no more than approximately 150 words. Reference the examples when crafting content.
Example low price point with a holiday preferred subcategory of ski
<title>
Gift Guide for Skiers this Holiday Season
</title>
<body>
Planning gifts for the skiers in your family? We've hand picked new gloves, goggles, and all the other ski essentials to stuff your stocking this holiday season.
**Ski Goggles**
Designed for clarity and protection, these goggles shield against snow glare and offer a snug fit for all-day comfort.
**Ski Gloves**
Stay warm with insulated gloves that provide excellent grip and waterproof protection for those chilly mountain runs.
...
</body>
Example high price point with a holiday preferred subcategory of hiking
<title>
Gift Guide for Hikers this Holiday Season
</title>
<body>
Planning gifts for the high-performance hikers in your family? We've hand picked the top-tier backpacks, hiking shoes, and all the other hiking gear to stuff your stocking this holiday season.
**Backpacks**
Durable and spacious, these backpacks are designed to carry all the essentials, featuring ergonomic straps for long treks.
**Hiking Shoes**
Engineered for challenging terrains, these shoes combine superior traction with weather-resistant materials for unmatched performance.
...
</body>
User Prompt:
Create content for a low price point shopper with a holiday preferred subcategory of Ski.
Response:
<title> Stocking Stuffers for Skiers on a Budget</title> <body> Give the gift of warmth and comfort to the skiers in your life without breaking the bank. Our affordable ski accessories are perfect for stuffing stockings this holiday season.
Ski Socks Keep toes warm and dry with moisture-wicking ski socks that provide a snug fit and excellent arch support.
Ski Hats and Headbands Stay warm and stylish with our collection of ski hats and headbands, designed to keep the cold out and the warmth in.
Hand and Toe Warmers Keep hands and toes toasty on the slopes with our convenient and portable hand and toe warmers, perfect for a quick pick-me-up.
</body>
We were very pleased with this result. It is clearly oriented around holiday gift giving and speaks to our shopper’s gifting preferences. To get the model to return this result, we had to provide a lot of detail in the instruction but the most important element seems to be giving the model some sample copy for it to modify. That seems to have constrained its behavior very nicely, providing us with content that’s near ready to go out the door. We then tested the prompt with the other 20 variations of price and activity preference and saw that we’ve got reasonably good results, allowing us to finalize our prompt engineering efforts.
Evaluating the Results
As was mentioned before, prompt engineering requires a lot of trial-and-error. Over time, you begin to be comfortable with different models, understanding the kinds of details needed in the instructions to deliver the results you want, but the need for experimentation never fully goes away.
A key question that often comes to mind: is there a way to evaluate the model outputs so that we might more reliably and consistently recognize good and bad results? While there are some metrics used by those in the field of generative AI to evaluate model outputs, these metrics don’t often satisfy our needs when we are putting models into use for practical applications.
So, we might reflect upon the process of manual content creation our organizations have been using up to this point and take some cues from there. When we are manually creating marketing copy, there’s typically a lengthy process of the author wrestling with the language and the constraints of the campaign. This is often followed by reviews, edits, etc. that ensure that the copy meets the needs of the organization. While a very human-driven process, the fact that so much time and so many eyes have been directed towards it means we have relative confidence the final result aligns with our needs.
If we embrace this process, we can still feed our generative outputs into a human-in-the-loop review process. If we keep the number of variations within reason, such as the 21 variations examined here, there’s no reason we couldn’t have folks on the team review and edit each one and still achieve significant gains in terms of efficiency. If we took the additional step to feed the corrected outputs back into our model (through a process known as fine-tuning or by simply including good and bad output examples in the system prompt), we might arrive at a custom model better able over time to deliver our requirements with less and less additional effort.
We can also make explicit the criteria we frequently use to judge marketing content. By incorporating into our workflow code-driven checks for misspellings, inappropriate language, sentiment, tone, etc., using popular natural language processing libraries and techniques, we can relieve our marketers of many of the more mundane elements of the review process. But we can go a step further still.
If you look at the bottom of the output produced within the AI Playground, you will notice a set of evaluations performed by an AI Judge (Figure 6).
An AI Judge is nothing more than an independent, often equally powerful or more powerful model, that is asked to examine content to see if it meets sometimes subtle requirements. For example, in the AI Playground, there are as of the time of writing two questions regarding the safety and relevance of the response. You could easily engineer numerous other criteria for your own AI Judge to evaluate as part of your content creation workflow, better ensuring that content meeting your criteria requires minimal human intervention before being sent to customers.
Next Steps: Building the Creation & Delivery Workflow
With our prompts defined, we can now explore how to build an automated workflow for creating copy at scale and then connecting this copy with a delivery mechanism. In the next blog in this series, we will show how this is done, connecting Databricks with a CDP in order to access customer details, generating and evaluating copy based on prompts defined like we’ve done here, and then passing the approved copy to an email delivery service in order to reach our target customers. In the meantime, we invite you to explore the Databricks AI Playground and try your hand at prompt engineering for content creation.
Source link
lol
