Linking AI Initiatives to Real-World Outcomes

Use Case



Project Summary
  • Collect and analyze predictive MNO data, (3 primary data sources GSM, Network transaction and Airtime credit, 12 months data)
  • Cleanse, transform, enhance and integrate predictive, MNO data into Analytical Data Warehouse (ADW)
  • Load to Analytical Data Warehouse
  • Combine and tune with existing Models Library (ML) credit risk and other models
  • Combine with models and data from comparable emerging markets and use a proxy
  • Blend scores from multiple ML credit models to create optimized credit/risk, limits management, and fraud scores
  • Implement using Decision Engine (DE) and Acquired Insights’ platform
  • Benchmark against equivalent industry performance
  • Execute Mobile Financial Services (MFS) lending decisioning in pilot model
  • Review MFS pilot lending results
  • Model and test Minimum Viable Product (MVP) based on comparable MFS lending pilot experience
Project Goals

The main goal was to create and benchmark a pilot, automated, consumer lending platform to integrate with the client’s Mobile Financial Services (MFS) architecture, including,

  • Intelligent, Mobile Financial Services (MFS) customer pre-selection and credit scoring based on the MNO customer base
  • Enable MFS lending, based on MVP (minimum viable product), for a subset of pre-selected and scored MNO customers
  • Machine learning and subsequent MFS model tuning based results
  • Pilot implementation roadmap and MFS platform
  • Demonstrate the credit performance and viability of the pilot platform

The diagram below shows a general approach to tuning to the ML models selected:

Minimum Viable Product (MVP) Methodology
  • The data was uploaded to Acquired Insights’ Big Data, Analytics Environment Analytical Data Workbench (ADW)

  • After initial data discovery, the data was compared to the Models Library MFS credit data model templates and the data quality measured using Acquired Insights 4I, (Invalid, Incomplete, Inconsistent, Inaccurate), data quality methodology and tool set

  • Following data discovery, the data was cleansed and enriched. This step was required for MNO data segments found to have sub-optimal data quality. Additional data domains were added to the raw MNO data to fill the predictive data gaps discovered

  • A range of Acquired Insights’ existing emerging market models were used, along with standard tools, (Decision Engine (DE), Behavior Data Warehouse (BDW), Models Library (ML) and Analytical Data Workbench (ADW)

  • Data discovery, data quality, data enrichment and Acquired Insights data model template comparison were conducted iteratively in order to achieve desired predictive data quality

  • Business metrics and target variables for the Mobile Financial Services (MFS) were defined, e.g. Default rate, fraud rate, customer retention, loan-to-value (LTV), customer limits

  • As the client digital lending product was not already in production, the Acquired Insights team used a number of emerging market proxy targets to measure accurately and estimate the underwriting and portfolio profitability results
Data Transfer

A comprehensive set of data domains was provided by the client and loaded into the ADW. Data extraction, cleansing, integration, fuzzy matching, enrichment, feature engineering and ML model tuning were performed by the Acquired Insights MFS Data Science and Risk Management team, based on an iterative MVP methodology as shown here:

Data Sharing

Data sharing considerations included,

  • Legislative - certain data needed to be obfuscated
  • Accessibility - certain data was not available for the MVP
  • Time limitations - MVP project timescales limited the time to extract and transfer all predictive, MNO data. MVP deployment was successfully accomplished utilizing only the most predictive MNO data.

A range of data domains were used for decisioning, robo-underwriting workflow development and initial, ML MFS model tuning. Data volumes included:

  • 39m prepaid GSM subscribers with 1 year of monthly and daily aggregates
  • 750,000 prepaid GSM subscribers with 6 months monthly and daily aggregates;
  • 1.5m of MVP data and Mobile Wallet data customers with 6 months detailed transactions and daily aggregates (1m active customers with15m transactions per month).
  • 4 months’ credit payment data
Data Discovery and Preprocessing

During predictive data discovery, raw data was uploaded into Acquired Insights’ analytics environment. Numerous, Acquired Insights AI/ML tools were used, including, enrichment/data transformation queries, data integration scripts and advanced data visualization techniques etc. These were applied to understand the data, detect data issues and support the feature engineering and Models Library (ML) Mobile Financial Services (MFS) model tuning process. Additional data sets were uploaded to assist with data queries/issues. A range of Acquired Insights data discovery and data mining tools and methodologies were used to understand MNO MFS business processes, calculate MFS business KPI’s and discover data additional fields that could be used as inputs for the ML MFS models tuning phase.

The feature engineering process was started after data discovery, data quality assessment and data cleansing. Features were aggregated to customer level using numerical features, categorical features, timestamps, strings, etc. The model tuning phase was started once clean, complete data sets were obtained with predictive features and target variables.

Models Library Mobile Financial Services Model Tuning

Prior to MVP implementation, the client’s Mobile Financial Services (MFS) platform had not been used for lending products. In absence of actual data on MNO MFS loan performance, Acquired Insights used the following approach:

  • Used existing ML MFS models from similar markets
  • Model tuning using “proxy” targets
  • Compared output, retuned, against proxy targets
  • Produced initial MVP output results measured against proxy targets
  • Compared, tuned MVP MFS models output to MFS models output statistics in similar markets

Three primary ML models were tuned and transferred using available MNO data:

  1. Airtime credit (bills delinquency) model - predicting bill payment delinquency, (a “proxy” target,) for postpaid subscribers based on their MNO data usage - tuned and tested on MNO data.

  2. GSM Usage model - predicting customer default events for a live loan product based on subscriber MNO data.

  3. Transaction model - predicting customer default event for a live loan product based on mobile wallet financial transactions.

Models 2 and 3 were tuned, tested and compared to a number of ML emerging market models and data. Features were mapped to MNO’s datasets and ML models were transferred and adapted from similar markets.

After the models were tuned, they were blended into a single credit score to obtain a more robust and stable prediction of subscriber’s probability of default for digital, microloan products.

This ML credit score achieved good predictive result, sufficient to be taken to the next stage for initial pilot use and tuning.

The credit default prediction model was tuned using client and other emerging market data.

The diagram to the right shows the results of the tuned ML model. This was predictive for credit default events for all three geographies, thus providing a cross geography credit scorecard.

A further round of model retuning was conducted in order to obtain an optimal, blended model, leveraging data from multiple similar geographies.

Such an approach made it possible to have viable underwriting platform to support a pilot micro-lending product within a short period of time. The Models Library (ML) Mobile Financial Services (MFS) models demonstrated good predictive power, automatically rejecting high-default and high fraud segment loans.

Tuned Models Library MFS Credit Model Performance –  Area Under the ROC Curve (AUC) Metrics
  • Airtime credit model: 0.91 AUC
  • GSM model: 0.82 AUC
  • Transaction model: 0.78 AUC.

(0.5 AUC = bad (or random coin flip) prediction; 0.6 AUC = acceptable performance; 0.7 = good performance; 0.8 AUC and higher = excellent performance)

The pilot results showed, that the airtime credit model performance was strong and stable. GSM and Transaction model had a slightly lower performance, as these models were transferred from other geographies and not all features were mapped to MNO data, (NB some features were missing from the initial MNO datasets provided, so were excluded from the models).

The Airtime model was tuned with a “proxy” target. The two other models were ensembled and produced a stable, robust model for default event prediction for the MVP phase.

After applying each model to MNO customer data, Acquired Insights divided the MNO customers into three risk bins: High Risk, Medium Risk and Low Risk.

The graphs show performance of each model for the airtime credit “proxy” target.

The three models were blended into a single, combined credit score to provide robust default prediction, credit limit estimation and customer segmentation capabilities.

The blended score was compared with various business KPIs (e.g. revenue split) to conduct a “sanity check” of the customer default prediction score. An example of such a check is described below.

Customers were segmented by blended score and divided into 10 equal parts (bins). The 1st bin represents customer with the lowest score, i.e. high risk, high default level customers. The 10th bin represents customer with highest score, i.e. low risk customers. Diagram 3 shows correlation of customer risk level with customer’s monthly revenue – the higher score, the higher monthly revenue.

If we drill down revenue structure we can see the combined score model still produces a good, predictive curve of score distribution vs. services revenue.

As well as credit default, the following models were also tuned using the MNO data,

  • credit limit - based on blended credit score
  • fraud prevention - MVP tuned fraud detection, included fraud black list checks and anomaly detection model based on customer transaction data.

The diagram to the right shows the result of the transaction anomaly detection model.

Each dot on the graph represents the customer’s aggregated mobile wallet usage, the distance between dots shows customer transaction behavior similarity, i.e. similar customers are shown close to each other on the plot. Red dots show clusters of anomalous customer behavior that can be defined as fraud suspicious.

Decisioning Workflow Implementation

Following the data absorption and model tuning, Acquired Insights implemented the combined DE and analytics tool set as a running, MVP environment. This provided a scalable solution architecture that would allow rapid implementation and ease of customization for full scale production.

Acquired Insights also worked with the MNO team to help generate Mobile Financial Services (MFS) product definitions and functional system requirements.

Technical assistance was provided to help define, agree and implemented the integration communication protocols (APIs) between the MFS solution components.

The Decision Engine (DE) was an essential part of the MFS solution. Using DE, MFS decisioning workflows were rapidly created, tuned and credit model and credit decision outputs integrated. A pilot MFS loan product decisioning workflow was developed as part of MVP. The three tuned models, blended score, fraud list, credit limit and business rules were implemented as part of the DE workflow.

Prior to MVP launch, Acquired Insights produced back-end prototypes and developed a demonstration App of the MVP from the customer point of view (front-end). Given the low, smart phone penetration in the market, the requirement was to demonstrate this using USSD. The diagram below shows the illustrative customer journey for MVP using USSD capabilities.

The diagram HERE shows a sample decision workflow with ML Mobile Financial Services (MFS) models utilized in the DE.
Brief Summary of Mobile Application

On Screen 1, a customer enters the USSD code of a loan product (the digital loan product can be advertised via SMS/MNO website/multi-channel marketing campaigns/etc.) and requests information about his potential loan.

The customer is asked to enter his national insurance number (or equivalent) at the time of requesting a loan.

The Acquired Insights platform, (DE plus the models described above), then determines whether a customer is eligible for a loan and calculates the optimal loan amount, based on a set of customer prequalified loans, (scored in batch mode), using customer MSISDN (mobile account number or other reference).

On Screen 2, the customer is shown the output of the decisioning system with the eligible loan amount. A customer can accept this amount, enter a lower or a higher amount. If a customer enters a higher amount, the customer’s loan limit will be checked and a loan issued, provided the requested amount is lower than the maximum, dynamically calculated limit.

On Screen 3, the customer enters their MPIN to indicate their consent. The appropriate error messages is shown for any un-authorized action or unfulfilled user request.

If the customer accepts the loan terms and conditions, the loan amount is disbursed to the customer mobile account using the Acquired Insights platform. A corresponding record is created in the Acquired Insights Loan Management system (LMS) including customer details, loan details, loan schedule, decisioning data, verification data, credit scores, model scores, limits utilization, etc.

A pre-qualified customer/loan sample, processed in batch mode, was used for the MVP. Real-time loan decisioning was implemented using the Acquired Insights platform In addition to real-time functionality, further integration with the mobile/web/chat application was built to enable the MFS digital loan product to be provided across all MNO customer’s portfolio and channels.

The system was integrated with smartphones, feature phones, chat, USSD and web applications.

Results Summary
  • Acquired Insights platform and analytical toolset assembled to capture and enrich required MNO datasets
  • Data quality mechanism applied and MNO data issues discovered and fixed
  • The pilot demonstrated that the MNO had large variety of data that could support an automated lending platform and be used for the creation of new products, once cleaned and prepared
  • A production ready default prediction blended score was built based on three high performance models (AUC 0.78-0.91) tuned, tested and transferred using different, similar emerging markets geographies. Score results were good and sufficient for MVP deployment
  • Credit limit model and fraud prevention workflow were built and deployed using DE
  • Blended credit score was shown to be good, production strength and have a high correlation with a range of MNO KPIs, e.g. customer monthly revenue distribution
  • Production strength credit decisioning workflow was built based on tuned ML models and MNO MFS architecture and data
  • Digital consumer lending MVP successfully deployed, using tuned ML MFS models and DE based credit decision workflow on MNO MFS platform
  • Once data was made available, the above tasks were completed by our team in 8 weeks
Additional Recommendations
  1. Increasing predictive data capture and range of data, (e.g. enriching data with new sources such as social network data, additional behavioral data, etc.)

  2. Increasing the range of fraud models.

  3. Improving collections infrastructure to optimize payments from non-performing loans.

Overall Improvements:
  • MNO emerging markets project using GSM, Airtime Credit and transaction data
  • High predictive results for mobile lending functionality (AUC 0.78-0.91)
  • Project duration, 8 weeks from receipt of data

© Zoral Limited 2017 All rights reserved 

© Copyright