Market research agencies are uniquely poised to take advantage of advances in NLP and AI text analytics; In this case study, we explore how Touchpoint Group Client augmented their standard practices using our AI text analytics tool, Ipiphany.
Leverage Customer Feedback To Gain Greater Insights Into Mobile App Performance
Apps have become one of the leading customer-facing assets for many brands and the outcomes, both emotional and functional, that customers achieve with the app reflect their overall impression of your brand. Ipiphany™, Touchpoint Group’s AI customer analytics tool, offers a comprehensive analysis of mobile app feedback against similar brands in the banking sector to uncover insights in market positioning, app performance and customer perception resulting in key opportunities to optimise your app.
The following use case details how Ipiphany™ can be leveraged to help a Tier 1 bank gain greater insight into the performance of their Mobile App through publicly available customer feedback data.
Use Case Business: JP Morgan Chase & Co
JP Morgan Chase & Co is a Tier 1 bank in the United States, and one of the largest banking institutions in the world. Their app is used by thousands of customers every day to manage transactions, investments, and trading worldwide. This use case analyses;
- Publicly available anonymised review data
- 31,920 reviews containing unstructured text comments posted in a 12 month period*
- Data from two sources: 21.7% Apple App store, and 78.3% Google Play Store
- The reasons behind the combined overall score of 3.9 out of 5**
Using the AI Analyst, Ipiphany™ - Touchpoint Group was able to extract insights from unstructured text data, readily available through public review platforms. The objective was to understand in detail the insights customers were sharing around preference, usability, issues and their overall perception of a product they rely on for daily use.
Using public data captured by third parties gives a business a rich insight into what customers really think about a product as they’re rating it on an unbiased platform with no incentive. Better yet, it gives us [Touchpoint Group] open access to a large repository of data and the ability to benchmark organisations against their direct competitors within the industry, locally and internationally, without having to undertake expensive market research initiatives.
With the supervised AI training of Ipiphany™ to read and categorise customer comments specific to the banking industry, we were able to extract insights from the text data focusing on;
Benchmarking: Directly comparing JP Morgan Chase & Co against competitors to determine their positioning in the market
Competitor Analysis: Compare product performance and app capabilities against key competitors and leveraging market insights to drive future improvements and features
Measure against the key S.U.R.F Pillars: Learn app performance and perceived user value against the main four areas of customer feedback; Security, Usability, Reliability and Functionality focusing on remediation opportunities that return the greatest value using the ratings as the proxy for RoI
Granular Detail: Delving into the 'why' behind their 3.9 rating by analysing customer feedback and determining the root cause
The following use case introduces a broad idea as to what is possible with Ipiphany™, with examples drawn from the data analysed for JP Morgan Chase & Co.
By benchmarking the overall score of JP Morgan Chase & Co against competitors - Wells Fargo, Citibank, and Bank of America, we are able to gain a high-level overview of customer app perception and market positioning.
JP Morgan Chase & Co scored 3.9 out of 5* overall with the next closest competitor - Bank of America - at a significantly higher 4.5. We then used the AI Analyst, Ipiphany™, to read the text and apply sentiment analysis to the data. Two specific aspects influencing the rating were noted as;
- Reviewers discussing the spend analysis and budgeting features tool rank the app at 4.2 - these features are regarded positively, and are responsible for driving the overall score up.
- Reviewers discussing ‘login features’ rate the app significantly lower than average, with a score of 2.1. Driving the overall score down.
Understanding the impact that these two features have on customer perception and the overall score of 3.9 gives JP Morgan Chase & Co specific opportunities for improvement against a major competitor.
Comparing the strengths and weaknesses of competitors in the market allows for a detailed analysis of positioning and reveals market insights, as well as key areas of focus that will provide the highest impact on JP Morgan Chase & Co’s ranking against its competitors.
By focusing on competitors’ strengths, we gain a better understanding of market positioning, and by focusing on JP Morgan Chase & Co’s weaknesses we can pinpoint clear avenues for improvements that will drive the most impact. By analysing the text data by customer segment, we are able to drill down to a granular level to uncover these insights,
- Reviewers frequently lament the absence of a “Search Transactions” feature - this functionality is commonly available in the market and drives the overall score up in competitor’s reviews, but it is absent in the JP Morgan Chase & Co app.
- All banks in the analysis group have a check deposit feature which, when mentioned, is responsible for driving the overall score down. However, JP Morgan Chase & Co’s cheque deposit feature appears worse than the others, netting an average score 1.3 points lower than competitors in reviews where this feature is mentioned.
- While JP Morgan Chase & Co has account management and budgeting features that are on par with the average score, all three competitors we analysed score significantly higher in this area. The high frequency of reviews mentioning this suggest that this feature is important to app users.
We understand from this data that there are pain points across the board with app users wishing to deposit cheques, but JP Morgan Chase & Co has an opportunity to drive impact by improving this feature in their app. As an identified weakness in the market overall, this improvement would have the most significant impact on the overall ranking.
The Touchpoint Ipiphany™ app benchmarking tool uses four classifications to categorise analysis into the way users interact with and regard an app. These are the S.U.R.F (Security, Usability, Reliability, and Functionality) pillars. By analysing these pillars and the way they relate to each other, we are able to gain a complete understanding of issues plaguing users.
Analysing the JP Morgan Chase & Co App review data against each SURF Pillar at a high level allows for an easy understanding of where to focus first.
From the chart above, we can surmise the following:
- Security: Average 2.4, decreasing this quarter
- Usability: Average 4.0, remaining steady
- Reliability: Average 2.5, decreasing this quarter
- Functionality: Average 2.9, decreasing this quarter
The SURF Analysis quickly reveals a hierarchy we can use to direct focus. Usability ranks at a steady 4.0, so although investigation may reveal small improvements, it’s not a major concern for immediate focus. However, Security and Functionality, and reliability saw significant decreases this quarter, presenting an immediate opportunity for investigation and improvement.
Detailed analysis reveals one major issue that is connected to all three of these pillars. There is a recent and significant problem with the Touch ID functionality of the JP Morgan Chase & Co app, which is leading to an overall decrease in trust in the security of the app. By investigating this in parallel, we are able to perform a granular analysis and identify the root cause of the problem.
Root Cause Analysis
In the high-level SURF analysis, we used Ipiphany™ to uncover a pain point leading to a significant decrease in the score, relating to both the Security and Functionality pillars. The next step is to investigate the detail of this pain point in a Root Cause Analysis.
A high frequency of users detail intermittent but consistent issues with the Touch ID functionality in the JP Morgan Chase & Co app. These users report that the touch function often works once before shutting down from within the app, requiring users to re-install the app, reset their settings, or in some cases causing the phone to crash. By comparing user segments within the app review data, we are able to use Ipiphany™ to provide enough information on the cause of this issue in order to provide actionable recommendations to fix this pain point.
- This issue began approximately 6 months ago
- 91% of users experiencing this problem report that they are using an android phone
- The issue is more than twice as likely to occur in Version 4.005 of the Android OS or newer, with the frequency of the issue increasing with newer versions of the Android OS.
This issue will continue to increase in frequency as users update their operating systems, so addressing this pain point becomes increasingly important. 78.3% of those reviewing the JP Morgan Chase & Co app are Android users, so this issue affects a large portion of the user base. The distress users are experiencing is leading to an overall decrease in the perception of reliability and security, and it should be addressed promptly in order to maintain a competitive position in the market. The level of detail achieved by the Ipiphany™ AI provides enough information for JP Morgan Chase & Co to hand over to its app development team for further investigation and resolution.
Read more about root cause analysis to discover another insight hidden in this data.
*12month date refers to May 2019 - May 2020. The data does not include structured feedback or rating feedback with no comments.
** Score will differ to the score found on App stores. This is a combined score from both Google Play & Apple App stores, any reviews not containing comments have been removed from this data set.
Get A Demo
To learn more about our mobile banking app performance analysis, the findings from our study, or the other 30 banks we investigated worldwide, get in touch with our team of experts.