Getting the Numbers Right

Primary “voice of the customer” research is a powerful tool for gaining in-depth insight and analysis on business customer needs, experiences, opportunities and pain points. In a business banking setting, the conduit of proprietary research has proven itself to be both reliable and effective in deriving meaningful insights for strategic planning, new product development and building brand awareness through effective thought leadership programs. 

Both valuable quantitative and qualitative data can be captured at interview when the process is designed well. For example, rating scales can often be used to capture a market sentiment in a quantitative format whilst open-ended questions can also be used to drive further qualitative insights adding greater “colour” to the black and white data-based insights
 
Not Getting the Numbers Right

One of the key focus areas for senior bank decision makers is the underlying emotional nuance driving corporates’ behaviour, including issues around trust and switching intentions. The Edelman Trust Index is a popular measure of trust between government, media, non-governmental organisations (NGOs), businesses and the public. The underlying methodology consists of online surveys across approximately 15 countries, with pe-selected participants asked to rate their level of trust for a number of institutions. 

Although Edelman’s index can provide a general insight into the changes in trust sentiment across various markets, issues have been raised with how the data is gathered and represented. For example, the index depicts the mass population as all interviewees excluding the “informed public”. This creates problems when attempting to generalise the sample data as a population outcome as an important section of the public has been excluded. 

Edelman’s index also uses an unrepresentative 1 – 100 scale where 1 – 49 equals ‘distrust’, 50 – 59 equals ‘neutral’ and 60 – 100 equals ‘trust’ noting other scales such as BCG’s Net Promoter Score also implement a skewed scoring and scaling approach.  Generally, a score of 49 on an index scaled to 100 would be best represented as neutral as opposed to outright distrust.  

Forrester Research has also been criticised for recently rating Australia’s Commonwealth Bank (CBA) as the best online retail banking platform for functionality and user experience despite not acknowledging the bank’s recently poor outage record relative to the Big Four majors. An area that has been identified as a significant source of antagonism for many customers.

Primary field research is a powerful tool to capture sentiment nuances and help banks better understand their customers needs and expectations however it is a tool that is open to misinterpretation and misrepresentation. Where banks are looking for guidance from their research partners on where to allocate scarce resources for the strongest return on investment, innovation and opportunity while also assessing competitive threats and key rivals, it is critical the insights they rely on stand up to scrutiny.

These issues reveal the importance of using best practice research principles when conducting field research to ensure data is robust, reliable and representative. The four guiding principles for successful fieldwork are:

    Representative Sample Frame 

A random sampling methodology with a sample size large enough to make inferences of the population. When executing a research project a rigorous process is used to define the target population based on location, industry sector and size which delivers sample results referenced to the population at large. This means that once the field work is done, there are no imbalances in the sample which then need re-weighting.

   Direct Sample Interviews 

Direct interviews are a core part of best practice methodologies because it produces in-depth data gathering and allows interviewers to ensure that participants understand all questions and provide appropriate responses. It also helps to guarantee are very high completion rate. 

An engaged interviewee will always deliver better quality information and insights than an interviewee engaged under sufferance. 

   Anonymous and non-attributable 

Interviews conducted on an anonymous and non-attributable basis minimises the risk of biased responses from interviewees, driven by their desire to ‘please’ the interviewer or commissioning client who may have control over their access to credit.

Similarly, when seeking a market wide representative overview, soliciting client lists often results in positively-skewed outcomes as only the happiest and well serviced clients are put forward for interview.

Knowing that the information they provide will not be directly linked to their business helps ensure open and honest interviewee engagement.

   Structured Closed-ended Questionnaire 

Structured closed-ended interview questionnaires provide powerful quantitative results driving client decisions around needs such as new product design, competitive positioning, channels to market, cross sell and wallet share improvement – all designed to deliver reliable business case decisioning.

While Interview transcripts often read more as conversations, using a structured questionnaire as the guide ensures individual interviewees are responding to the same questions, without alteration. 

These principles drive best practice market intelligence and data gathering programs and represent an ‘open book’ approach implementing direct interviewing only, structured sample framing, independent interviewee populating of samples, an industry-experienced research team and low rejection rates.


Sampling errors can undo any good work generated by an excellent questionnaire or compelling research theme directive. Although businesses may provide compelling responses and answer the questions fulsomely, ultimately, an unrepresentative sample will deliver skewed findings. Building an appropriate sample frame is arguably the most important stage for conducting best practice research and analysis.

Primary research grounded on ‘bullet proof’ principles that ensure initiatives do not become a case of ‘garbage in, garbage out’ alleviate the risk of flawed commercial decisions that flow from poorly executed analysis.