(ARTICLE 1 in the Series)

The Rise of Big Data

Big Data technologies made it possible for enterprises to capture and integrate diverse sets of data. They were no longer constrained to the data warehouses for analytics and reporting. Big Data allowed the integration of third-party syndicated data sets and social media data such as tweets and blogs. In addition, it helped break down silos between the various divisions within the enterprise, democratizing data access and help gain new insights from data.

The enriched big data sets can be used not just to understand the past, but make predictions about the future – which customers are likely to churn, which customers/equipment are most likely to generate new claims, which products are the most likely to succeed, etc.

AI-powered

We are now in the next wave of deriving value from data using AI-powered applications. The big breakthrough for this wave is the ability to use AI-powered neural networks to solve a wide variety of problems including autonomous driving vehicles, natural language understanding, image recognition, etc. Translating these technological advancements to real business use cases will result in significant operational benefits – reducing cost, providing faster customer service while creating new business models and sources of revenue.

Let’s look at some of the use cases for AI in insurance.

Underwriting

Underwriting or new application processing is the first pillar in any type of insurance – namely, processing applications for new insurance policies. The process can be complicated depending on the type, size, prior history and other components of the application to evaluate the risk and enroll the client. It involves communication among multiple parties – the client, agent, and underwriter. This is traditionally a manual process as it involves a review of many different types of documents from diverse carriers with no standardization that allows easy automation. Further, many carriers still receive paper documents that are faxed or scanned (or worse – sent via snail mail !)

AI-powered systems can help this step in multiple ways:

  1. Natural Language Processing (NLP) systems and chatbots and streamline communication between the parties
  2. AI-driven document extraction systems (Docu-AI) can automate the processing of the various documents using AI and Big Data
  3. Data from documents can then be analyzed by AI-powered analytics to help the underwriter assess risk

Claims Processing

Claims processing forms the core of the business for insurance carriers. When a claim is processed in a timely manner, it improves customer satisfaction and retention. Simultaneously, the processing has to minimize financial loss due to fraud or other factors to maximize profitability. Most companies have focused their energies on improving the claims process using technology.

Related image

Many software applications already automate workflows ensuring timely processing and smooth communication with all parties involved. Mobile apps allow users to easily submit claims along with documentation such as photos of the incident, claim form, invoices, etc.

Yet, main parts of the process are heavily manual. Claims adjusters have to frequently go out in the field to make assessments. Even in the case of smaller claims, the adjuster may manually review documents and photos.

 

How can AI-powered systems help claims processing?

  1. Image recognition algorithms can help identify and automatically categorize various pieces of information in claim evidence photos such as license plates of vehicles, insurance cards, various types of damages, etc.
  2. AI-driven document extraction systems (DocuAI) can automate analysis and categorization of line items in medical records, body shop estimates, contractor reports, etc. Using NLP and Deep learning techniques allows these systems to recognize a wide variety of content.
  3. Robotic Process Automation (RPA) can automate many parts of the processing workflow along with 1) and 2) above

Fraud Detection

Fraud detection is usually a part of claim processing to ensure that no opportunistic fraud has taken place. The biggest loss for insurance companies is due to fraud. Many larger carriers already use predictive analytics to help identify potential fraud in claims. These Machine Learning models use not just  a carrier’s own data but also shared databases across companies to flag potential fraud.

AI-powered systems can take this a step further. They can use the vast amounts of accumulated data and images to detect more subtle instances of fraud as well as previously intractable ones. With the cost of running these models dropping dramatically, even small claims can be analyzed to detect patterns.

Customer Service

Image result for customer service iconImproving customer service is the goal of every organization. With Big Data and AI, it is possible to automate the analysis of customer service calls and emails allowing customer service agents to proactively address complaints and issues.

AI-driven chatbots are now pervasive on websites and web portals. They provide an easy way of answering customers’ questions while reserving human interaction to handle more complex issues. Mobile apps with the ability to answer spoken natural language queries are now possible using technologies like Siri, Alexa and the same knowledge base used by chatbots and customer service agents.

New Business Models

With IoT enabling the gathering of fine-grained data (how many do I drive every day, what is the average trip, how many hours is the property unoccupied), insurance companies are seizing the opportunity to come up with new ways of underwriting policies. AI-powered systems can provide better risk analysis for determining premiums resulting in new personalized products. These new products can be provided at attractive premiums, driving new business.

 

orzotapartner

IoT and Big Data are becoming essential to market growth and customer success. Enter Orzota’s call for partners. There are initiatives in all major verticals like Manufacturing, Oil & Gas, Transportation, Retail, Financial, Insurance, Life Science/Healthcare, etc. Additionally, there are many pieces to the puzzle between, data architecture, technology and the necessary resources to deliver a successful Big Data & IoT program that will benefit business users.

At Orzota we seat at the intersection of IoT and Big Data. As a Silicon Valley based company we aim to provide solutions that can transform the way companies collaborate and derive value from Data whether being it from sensors, machines, ERPs, websites, industry boards, social media and beyond.  To do so we bring platforms for Big Data and IoT that are flexible and quickly accelerate the delivery of solutions while supporting it through our Managed Services models. We’re harnessing Open stacks and cloud technologies that provide the elasticity and economics to quickly generate ROI. Lastly, we augment projects with verified resources for Data Architects, Data Engineers and Data Science.

As a partner, you’re a technology or a consulting provider that serves the mid-market and is looking to augment their service portfolio while adopting the latest in Open stack technologies for Big Data and IoT.  Apart from an excellent solution, what you can expect is the support of an experienced team that has been in the technology side of this domain at Yahoo while harnessing project experience from companies like Netflix, Boeing, and Bank of America to name a few.

We’re all about making it easy so email us at partners@orzota.com

We recently worked with a leading Hi-Tech manufacturing company to design and implement a brand new scalable and efficient workforce analytics solution targeted the mobile workforce.

The solution is designed to raise the workers’ confidence bar, and to minimize the effort required to train the workers. The solution also improved the manpower utilization by optimizing inventory adjustments with higher accuracy while fulfilling orders. It also reduces the learning curve for workers resulting in substantial reduction in training hours.

Workforce Analytics Solution Overview

The Workforce Analytics solution was built on a Common Data Analytics Platform leveraging Hortonworks HDP 2.4 and used the following technologies: Kafka, Storm, HBase, HDFS, Hive, Knox, Ranger, Spark and Oozie.

The platform collects real time data from the application on mobile devices, stores it, and runs analytics with better performance and lower latency compared to their prior legacy system.

The HDP components at a glance:
Workforce Analytics Solution HDP Components

Workforce Analytics Architecture

The operational real-time data is collected using Kafka and ingested into HDFS and HBase in parallel using Storm (see diagram below). HBase acts as the primary data store for the analytics application. The data in HDFS is encrypted and reserved for other applications. Based on the business logic, the data stored in HBase is processed using Spark on a daily, weekly, monthly and yearly basis, and stored back into HBase as a feed for Spark Analytics (Spark SQL). Spark Analytics is used to run jobs to generate specific insights. The output from Spark Analytics in Hive as a temporary table. Hive Thrift Server is used to execute queries against Hive and retrieve the results for visualization and exploration using Tableau. Custom dashboards were also built for business users to help them track higher-level metrics.

Workforce Analytics - Architecture

To address security requirements, Apache Knox and Apache Ranger were used for perimeter security and access control, respectively. Both are included as a part of HDP 2.4 and are configured in the Access Node.

Workforce Analytics Physical Architecture

The figure below shows the physical layout of the services on the various servers used. The architecture comprises of Edge Nodes, Master Nodes and Slave Nodes. Each set of nodes run a variety of services.

Workforce Analytics Physical Architecture

Issues and Solutions

While implementing this solution, we ran into a variety of issues. We outline some of them here in the hope that it may help others who are designing similar architectures with the Apache Hadoop  or Hortonworks HDP eco-system of components. Table creation, user permission and workflows were the common focus areas.

HBase Table Creation

We ran into permission issues with HBase table creation.

Solution: In Apache Ranger, update HBase policy by giving appropriate read, write and create permission for the defined user.

Connection to hive thrift server

Another issue we ran into involved connections to Hive Thrift Server for a particular user “ABC”.

Solution: Ensure that the below properties are added to $HADOOP_CONF/core-site.xml

hadoop.proxyuser.ABC.groups=*

hadoop.proxyuser.ABC.hosts=*

Oozie workflow jobs submission

Permission errors continued to plague the project while creating workflows in oozie.

Solution: The following needs to exist in the section of the corresponding job definition in workflow.xml:

<env-var>

HADOOP_USER_NAME=ABC

</env-var>

within the

<shell xmlns="uri:oozie:shell-action:0.2">

oozie workflow job stuck in prep state

When re-running an Oozie workflow job after a period of time, it went to PREP state and did not execute. While trying to kill the job via CLI, the Oozie log shows the job was successfully killed.

USER [test] GROUP[-] TOKEN[-] APP[-] JOB[ABCDEF] ACTION[] User test killed the WF job ABCEDEF-oozie-oozi-W

However, in the Oozie UI, the job is still shown to be in PREP state.

Solution: Further research showed that the Oozie database at the backend (Derby by default) was corrupted, and was not representing the correct state of the jobs.

We decided, for longer term stability, to migrate from Derby to MySQL as the backend database for Oozie. After this migration, we did not run into this issue again.

Conclusion

Big data projects can grow and evolve rapidly. It’s important to realize that the solution chosen must offer the flexibility to scale up or down to meet business needs. Today, in addition to commercial platform distributions such as Hortonworks and Cloudera, higher level tools and applications simplify the process of developing big data applications. However, as seen by some of the issues we describe above, expertise in the underlying technologies is still crucial for timely completion of projects. Orzota can help. Please contact us.

Does your Analytics journey look like this?

analytic solutionsBig Data and Analytics is now moving at a faster pace than before to midsized businesses (MSBs). The potential “Value” of Analytics while once thought useful to larger global enterprises, has quickly moved downstream. As a consequence, the spend in this space is also expected to grow faster than the large-scale enterprises.

Midsize businesses across domains can now gain a significant competitive advantage, get valuable insights to identify potential markets, and form the basis to improve customer experience and operational efficiency.

However, there’s more money spent on efforts to cope with massive influx of available data than the applicability of “Analytical Value” that technology offers. To influence business results meaningfully, MBs must take a multi-dimensional view of the Analytics Value. They must consider the Value across the following three dimensions:

Intent, Commitment, and Clarity of Business Impact

It is imperative to understand the purpose and focus areas where Big data and Analytics have the most potential. Here are few key questions:

  • What do we want to use big data for? Strategic? Tactical?
  • How can we monetize the data streams in terms of customer loyalty, revenue growth and/or cost reduction?
  • What are the areas for Business Impact (Customer, Product/Service, Operations, Supplier/Partner, Finance, Risk)?
  • Where within these areas does Big Data and Analytics provide the most sustainable value?
  • How can we use data-driven customer intelligence to understand customer behavior?
  • What specific customer-centric and operational-centric KPIs or metrics provide insights into a particular component of our business? e.g.: Propensity to Buy, Customer Lifetime Value (CLV)
  • How will better insights and information help overcome the most pressing challenges in our business?

Solution Options Built on a Foundation of Analytics

Most medium sized businesses lack understanding of the various Solution Options and Tools available, and hence are not confident and hesitate to employ it. It is imperative to select diligently from a plethora of Analytics Solutions and Tools for cost efficiencies, process improvements, data governance, and technology.

From a process perspective, they must be able to collect enough internal data, normalize and combine this data with external data sources to identify patterns and behaviors.

A study by IDC revealed that organizations that use diverse data sources, analytical tools (e.g., predictive analytics) and the right set of metrics are five times more likely to succeed and exceed expectations for their projects than those who don’t use these big data strategies.

Skills Training, Gap Analysis, and Lessons Learned

Having a successful initial implementation or Proof Of Concept (more is better, timing is of essence) within 6-8 months is critical. Shorten decision paths and leveraging domain centric Big data and Analytics partners and solution providers is essential.

However, identifying and realizing Analytics Value for midsize businesses is also more than just working with external partners. They must know the critical questions to answer based on the data.The business must understand the Analytics terminology and technology, as well as possess some internal statistical knowledge. We see a few midsized businesses include a new role – a Data Strategist, to help with their growth strategies, streamline business operations and integrate technology to help the business operate more efficiently.

Please feel free to reach out to learn how Orzota has helped organizations across the above dimensions to:

  • Build models targeted to specific use-cases that can be implemented swiftly, with clear business focus
  • Select, deploy of targeted data-analytic solutions
  • Adopt the Analytical solutions and tools
  • Realize Analytics Value Faster

Last week, we announced the acquisition of big data solutions provider, the Houston based company, Avarida. It has been a lot of work (and a lot of fun) doing the integration and figuring out our combined strategy as we move into 2016.

Many people have asked what the synergies between Orzota and Avarida are and how will our strategy and offerings change. As mentioned in the Press Release, Avarida’s focus on building big data solutions to solve customer problems is a great complement to Orzota’s technology-focused expertise. Integrating some of these solutions wit the Orzota Big Data Management Platform will provide our customers with end-to-end big data solutions.

Secondly, the Avarida team brings great enterprise-class experience on all fronts. Founder and CEO, Naren Gokul has tons of experience working in and with large enterprises. His sales and solutions engineering team nicely complement our own.

Finally, Avarida’s locations in Houston and Chicago eases our access to customers in the Center and East Coast. We are already seeing the benefits of this expanded team’s reach with increased interest from prospective customers.

Over the coming weeks and months, we will be launching new solutions and offerings, so please watch this space!