Gen-AI Doesn’t Understand Your Audience: A Cautionary Tale
Learning about and experimenting with gen-AI tools has helped us understand something that happened to a client a while back.
This client had purchased AI-generated user personas, with the intent of using them to guide content strategy. To produce the personas, the client provided a vendor with access to customer surveys and other customer data and the company's own social media accounts. The vendor’s ‘secret sauce’ included additional, mystery-meat data.
We were engaged as part of a small team to review the personas. We determined there was little worth salvaging and ultimately developed new personas based on user research with the client’s own audiences.
Why did we need to start over with the personas?
The gen-AI-created personas were angry, insecure, jealous, and political in a way that was irrelevant to real audiences' interactions with the company. In other words, these personas seemed like a reflection of THE WHOLE INTERNET, not just the company's own data.
One current deficiency of gen-AI tools is a lack of transparency. Generally speaking, we don’t know what these tools are doing, and we don’t know what data they are considering in answering the questions we pose to them. So, a problem with using gen-AI tools to analyze a data set and create a deliverable based on its findings is, we don't really know for sure that it is ONLY going to consider the data we give it. Good prompt engineering can help define the data set, but we cannot be assured that gen-AI tools won't "hallucinate" a likely but inaccurate answer from a much larger data set—one that does not represent our companies or audiences.
Considering the lesson of our client's angry personas, we would not trust gen-AI to tell us what any given group of people thinks or feels. Nor would we rely on an AI tool to reach conclusions about any research data by itself. Controls on data set scope are getting better, but are not reliable yet. (Note: We cannot yet comment on services that offer interviews and other research with “synthetic users” as proxies for real research subjects. We would need to learn more about exactly what data they employ, and many other factors.)
Moreover, AI does not interpret, has no real understanding of context or relevancy, and may not be able to discern what information is relevant to your audiences and what is extraneous, and even misleading.
We developed accurate personas by conducting a survey and interviews of the actual audience base, finding patterns, segmenting the audiences, and representing them using authentic quotations, demographic details, challenges, and motivations.
One of the many deficiencies of the AI-generated personas was a lack of representation of audience members who are valuable to our client but under-represented in its industry—professionals who are Black, female, and/or from emerging economies in African and South America. Bias is a big problem with AI and its training data set, with potentially cascading harms. Despite some recent improvements, we as product users will need to be vigilant about looking for bias whenever we use AI tools to create written or visual products.
Gen-AI tools are useful in many contexts, and we are learning daily how to use them wisely and safely. But unless and until we can know what data a gen-AI tool is looking at to arrive at its conclusions, and how it determines what is important and what is not, we would not recommend relying on it to understand your audiences, or conduct research. AI tools can assist researchers but are no substitute for human insights and judgment.
We researchers, content strategists, and user experience professionals are still needed as "humans in the loop."
