Qualitative market research in the age of AI

Ines Imdahl wrote a psychological plea for humanity in times of increasing automation and AI, and the associated relevance of qualitative market research, for the anniversary issue of planung&analyse for its 50th year of publication. Read the original article here. You can access the magazine and e-paper here.

Speaking, listening, finding meaning and context. Why qualitative market research is important in the age of AI.

Does qualitative research have a future? Don't Google, ChatGPT and AI already know everything?
That's exactly what I was recently asked at an event by the manager of a band. "Do you still survey customers directly? Isn't that completely old school?"

The manager - like other companies - has a lot of data at his disposal. He knows exactly how many people buy which songs by his band, how many are streamed or how many see the 'product' live.

However, he cannot recognise any meaningful connections in the large amounts of data. And above all, he cannot develop any derivations or good forecasts that will make his decisions for future activities more reliable. And here he admitted that it would be good to ask people directly. "When do you prefer to see the band live? Why is that? Why do you stream a single song all day long? And why do you buy this album but not that one?" Simple answers based on the 'like' of certain songs do not provide conclusive explanations for sales, concert visits or streams.

In fact, however, customers are being asked less and less directly or even in detail. Market research-affine companies, for example from the FMCG sector, have also drastically cut their budgets more than once in recent years. At first glance, this has 'logical' reasons:\

1. Companies today have direct access to their target groups, which used to be reserved for market research. They can get in touch with people directly and get feedback via social media and DIY platforms.\
2. Digitalisation, automation, deep learning and AI are making market research faster and cheaper. Surveying many people for little money and getting answers within a few hours? That's possible.

The detour via the possibly long and personal (in-depth) interview seems superfluous, complicated and outdated. Especially as almost nothing has changed in the field of qualitative research in the last decade. Still open questions and personal discussions, in the best case evaluation and interpretation according to a theory. New theories are hard to come by. Cognitive, semiotic, psychological and even one of the youngest, morphological psychology, are really no longer fresh. In contrast, there is the disruptive technology development of quantitative research. From pencil-paper surveys via CATI to mobile, online and automated surveys, big data analytics, AI and deep learning, there is a self-fueling change that everyone is trying to keep up with.

Does this not actually make qualitative research superfluous?

Collecting customer data alone is not enough. It must be meaningfully classified.

More data and more direct access to people should ultimately create greater customer proximity. The latest developments show: Although more and more corporate decision-makers believe they know the needs of their customers, fewer and fewer customers see it that way. (Source Harvard Business Manager August 2023)

From a psychological perspective, various (paradoxical) developments can be observed which not only underline the added value, but even the necessity of conducting conversations and listening and working out meaning and context, as is customary in qualitative research. Let's take a slightly deeper look into the crystal ball at the future of qualitative research.

First paradox: Because 'everything' is obvious, the 'real thing' is heard less and less.

In a time of data abundance and new, uncomplicated ways of reaching consumers, companies are turning to people less and less directly. They believe they know their target groups and customers because data is available en masse and it is seemingly 'obvious' what motivates people. This obviousness is questioned less and less. The sheer quantity and the mere possibility of access create a feeling of security and control. 'Everything' is there.

This 'everything' is scanned and sorted by extensive automation processes and the increasing use of AI tools. At best, it is also systematically and regularly evaluated. Ideally, a system is developed that functions independently and independently of further - human - influence. The fascination that these technologies radiate can be felt almost everywhere. Shining eyes, flaming speeches when it comes to the boundless world of digitalisation and AI - it can not only evaluate, facilitate and accelerate 'everything'. At the same time, it seems to relieve the annoying confrontation with the disruptive factor of humans, who, in their behaviour and experience, their sometimes senseless and changeable wishes, their complex inner life, do not fit so smoothly into the company's plans. It is like a secret wish fulfilment: Through the perfected processes, people's needs can be met with pinpoint accuracy and - at least in everyday business - ignored.

Customers want to be heard & understood. AI cannot do this.

Many customers also feel served in the sense of annoyed. It is becoming increasingly difficult to contact companies directly or to get through with important concerns outside of standardisation. Whereas customer service was recently outsourced to call centres, bots are now increasingly being used. More and more frequently, customers are not only stuck in waiting loops, but also in feedback loops. Instead of discussing with a direct contact person, they discuss with the AI or despair at standardised answer options.

This also increasingly applies to the way people are interviewed in research. If they are interviewed. Questions are considered at the green table that often miss the needs of the people. It is hardly ever checked whether the topics being surveyed are the ones that 'actually' move people, whether a product or service is a real solution or adds value to people's everyday lives. It goes straight to the comparison: what is better, what is worse: A or B? As in the well-known 'personality tests' from magazines, none of the answer options actually fit: What is chosen is perhaps the least unsuitable. But you don't feel comfortable with it. Research becomes even 'smarter' without people, in which the AI simulates personas and participants. Here, the 'disruptive factor' of humans can be completely eliminated and no one has to take the sensitivities or availabilities into account.

So the 'real thing' is hardly heard in the huge amounts of data. But it cannot be kept out in the long run, it seeks its space. More and more people - also on social media - are turning to psychological coaches, empathy, emotionality and a positive mind-set. They want to study psychology themselves (hardly any other subject is in such demand), they want to get to the bottom of their dreams, wishes and sensitivities. Intensely. A huge trend. Where they feel particularly 'unheard', they go even further: onto the streets or into radicalism. Often as a last resort to make themselves heard. (Source "Media between respect and ostracism", rheingold salon on behalf of the Stiftervereinigung Presse, the BDZV, the Stiftung Presse-Haus NRZ). Empathy has also become an important leadership skill for corporate management, which they like to use to adorn themselves. Only in customer and target group surveys is it primarily technological and 'freed' from the human element. The 'real thing' is often left out here.

Companies can listen & understand. Qualitative market research supports this.

This is all the more problematic as people currently place a lot of hope in companies. Strictly speaking, they are the last remaining hope for solving social challenges. Politics has had its day here. At least in the western world, neither politics nor religion are able to provide clear guidelines and orientations. In order not to lose the confidence placed in them, companies should not only focus on optimising their processes - and see the customer merely as a "means of money" . They must at least give him another meaning and lend him their ear. In the form of qualitative surveys.

Our experience shows: If success fails to materialise or dwindles despite optimally digitised processes and feedback loops, and it becomes clear that man and machine function quite differently, the door often opens for qualitative research.

Direct conversations and in-depth psychological surveys lead to aha moments that show where the real needs of customers lie and when they need a real person in addition to AI. Companies that regularly listen to their customers have a real advantage, also when using AI and digitalisation. Because these can only be as human as the people who feed them and readjust them again and again. And even if it is tempting: companies cannot avoid dealing with the human element. It may be arduous not to believe all statements directly and to have to follow up again and again. However, there is only one way to better understand the human, the psychological and often seemingly nonsensical: talk to people to understand them. Directly and personally.

Second paradox: The desire for comprehensive calculability demands a firm belief in the 'unknown' and promotes the revival of the black box.

Enabling understanding. Qualitative market research & the question of why.

The comprehensive surveying and calculation promise through AI and digitalisation has its price. It goes hand in hand with a firm belief in an unknown. Because AI and deep learning are not only intransparent from a certain point onwards, but can no longer be understood by humans. (Source: "Die KI wars" Katharina Zweig). Understanding and the 'why' are dispensed with. Both remain outside or inside: in the black box. It is the uncertain, the imponderable that must be lived with.

Qualitative research is also increasingly encountering a lack of interest in deeper understanding. A pharmaceutical company can measurably polish up its scandal-ridden image with advertising on social media? That's enough. Why that is is of no interest. The advertising works. At least in the supposedly right direction. You can show your face with that, to the boss, to the stakeholder, to the investor.
For most people, the 'why' or 'how' of artificial decision-making processes is also irrelevant to their personal everyday lives. If a doctor or an AI had to decide who should be given preference in a transplant, many would even want the decision to be made by the AI. In the blind trust that it will decide more neutrally.
Why the AI decides how is only irrelevant until you personally no longer feel that a decision is justified: "Why am I not getting the loan or the flat? Why was my social media account blocked?" What sometimes sounds abstract can possibly be traced back to the Schufa information. The Schufa rating is often perceived as opaque and unfair. If machines classify us into certain pigeonholes, this can have a major impact on our lives. Blind faith in the correctness of the decisions of algorithms and AI throws us back into the age of fatalism.

The same applies to companies. Only when the advert, which supposedly improved the image, does not prevent the shitstorm, will the judgement of the new 'god' perhaps be doubted and looked at more closely. Out of necessity, the sacrifice of the personal survey is made.

Understand connections & find meaning. Qualitative market research in times of upheaval.

Qualitative research, when understood correctly, is not a nuisance stopgap when you no longer know what to do. If we want to be masters of the spirits we have called, then the why and the understanding of human behaviour and experience is the key. We are thus opposing a fateful devotion to new god-like structures of arbitrariness. As long as we question meaning and uncover connections, we not only keep AI and the black box in check, but also use them in the sense of humanity. And this is the only meaning they should have.

Qualitative research helps to keep the 'real thing' in sight and make it heard. It offers understanding where we do not simply want to or are allowed to come to terms with circumstances and decisions. It creates connections and meaning that people are currently seeking more urgently than ever before. And with its reliable, proven methods, it provides an anchor of stability and security in times of uncertainty and the unknown. Because it knows how to deal with the human element, to assess it and interpret it meaningfully. Because it has an incredible amount of practice in this. And last, but not least: Qualitative and quantitative research are not only not opposites, but are mutually dependent in order to keep the meaning of research relevant in the future. Technological developments need meaningful classification more than ever.