What are Insights, why are they Rare and Elusive ? How are Insights different from data, information, analytics and wisdom ?

Insights are a rare phenomenon and there is a limited understanding around this area. In the first part of this post, we outlined the key differences between Instincts, Intuition and Insights. We also categorized Insights into Hindsight, Insights, Foresight as well as Strategic Insights. In this part of the post, we will outline the various differences between Data, Knowledge, Insights and Wisdom.

As we use the the three terms Instincts, Intuition and Insights interchangeably, we tend to use Data, Knowledge, Insights and Wisdom also interchangeably. There are significant differences between these terms. As we will show, they are all a part of the same “Information Prism”.

  • Data is the building block of information and we collect data in different ways in different domains or contexts. For most of us, data is information that is collected, derived or observed accordingly to an agreed standard. Same data can be interchangeably applied across different standards, however, most of the data is unique to each domain or context.
  • Analytics is the process of examining, structuring, sequence data to find existing or hidden patterns so as to create relevant information around which a strategy can be developed. Analytics is used to generate Information as well as Insights. As we discuss below, Insights are rarely generated through the Analytics process. What is generated is information which is packaged and sold as insights.
  • Information is data synthesized specifically for a given context and situation. Various people can use same type of data to integrate it differently and arrive at different conclusions. Same data can lead to different information and there are millions of possibilities once the data is shuffled and weighted. Neither data and neither synthesized information can be termed an insight.
  • Wisdom is cumulative sum total of our life experiences acquired through Intuition. Everyone acquires information as they move across their life span. Everyone will claim that they have wisdom, however acquisition of wisdom also creates bounded effects i.e we chose some information to acquire while deliberately or implicitly ignoring other areas of information. As a result, unless one has had diverse range of experiences in multiple contexts, acquired wisdom can limit our understanding of the contexts. Wisdom also have a predictive function i.e. wisdom should most of the time lead to accurate assumptions about the past, current and future. We can also say that Wisdom is the cumulative information we acquire through our life time. Our cultural and family of origin can severely constrain our wisdom.

What is Insight ?

We will now address the most interesting part of the discussion i.e., what is Insight and how is it different from data, knowledge and wisdom.

Insights are of three types Hindsight, Insights and Foresight. The most powerful form of Insight is “Prescriptive Insight”, which converts Data, Knowledge, Information and Analytics into Intelligence. Intelligence is about prediction of events as well as prediction of actions related to those events. Insights can either lead to a revelation or illumination. Revelation is uncovering what was already there while Illumination is exposure to the novel and new.

Insights can either lead to a revelation or illumination

Using an example from entrepreneurship and marketing, have you ever heard of “Product-Market Fit”. Product-Market fit is the elusive moment of singularity when a service or a product suddenly clicks and becomes successful with the customer. The lack of a product-market fit after cash flow burn are the two major causes of business failure. An entrepreneur or a business goes through many learning cycles to truly, deeply understand their consumer needs and preferences. They famously use the term “pivot” for every iterative correction made to arrive at the moment of singularity. Assuming that the journey will lead to success, at every step, the entrepreneur is generating insights on how to achieve the perfect market fit.

Entrepreneurs, scientists and marketing gurus and Data Scientists all strive to get a multi-dimensional and accurate understanding of the world. As our reality, market or world is a complex system made of multiple layers, it requires a lot of information before some sort of accurate understanding can be achieved. Insights bridge the gap between what we don’t know and what we want to know. On this basis, we can say that “Insights” are unique pieces of information which provide us accurate information about the true nature of reality.

“Insights” are unique pieces of information which provide us accurate information about the true nature of reality.

Although Insights are sold around every corner, Insights are rare. We instead have “Data Rich, Insight Poor” companies who spend millions of dollars on accumulation or useless analytics of consumer information. There are good reasons for this lack of creative insights. One is that Insights are a “neurological phenomenon” and existing psycho-neurological optimization of the subject who is searching for insights is a necessary condition. The second is that even a basic business scenario is a complex system marred by ambiguity, inaccuracy, deception and rapidly evolving domain. As such, “uncertainty” is inherent in the complex system at all the time. Uncertainty means “Quantum Indeterminacy” and in simple terms, is it almost impossible to know everything about an event or domain using reductionist approaches. Data Science or Analytics has both reductionist or holistic areas, however, holistic components are significantly less developed that the structured analysis based excavations.

Unsuccessful attempts have been made at commoditizing and commercializing Insights, however the reality is catching up fast. From failed analytics projects to completely inaccurate election forecasts. Even institutional investors are sounding the alarm on the hype. Data Science and the process of generating insights is both Art and Science. We have surely missed the “Art” part and moved to over rely on scientific models and accumulated data. To be clear, I am not saying that Data Science is complete bluff and has failed to generate valuable insights. All I am saying is that Data Science has not made good faith attempt to understand how our intuitive ability and neuro-psychological optimization is the key to generating better insights.

Neurological Basis of Insight

Insights provide us accuracy and multi-dimensional information about the domain or context that we are researching. It is important to understand the origin of Insights. As we have already discussed in the previous article, Insights emerge into our consciousness when Intuitive Processing has reached a particular activation threshold. Insights emerge into our conscious frames when we are ready to capture them.

As per Wikipedia, “An emergent behavior or emergent property can appear when a number of simple entities (agents) operate in an environment, forming more complex behaviors as a collective. If emergence happens over disparate size scales, then the reason is usually a causal relation across different scales“.

On this basis, let us outline the necessary and sufficient conditions for emergence of insights based on year of practice in this area. Insights emerge in our consciousness when the below necessary and sufficient conditions are met :

Necessary Conditions  

  • A Problem is well framed around which Insights are being sought.
  • The subject should have have some sort of prior domain knowledge or domain expertise around the area of insight generation.
  • The subject should be in active receipt of current information about evolving domain, changing trends and forces at work.
  • For true and Effective Insights to be generated, the insight should not be available in public domain.

Sufficient Conditions

  • The subject should have optimized psycho-neurological ability to enable their own intuitive processing, so, should have personalized and customized framing methods around this area.
  • Their Intuitive Pattern Recognition skills should be well honed to identify and capture relevant and salient insights.
  • They should have well developed ability to convert these insights into Relevant Strategy using “Sense-making”

As we can see, the sufficient condition is the within the mind of the subject who is trying to process or generate insights. Insight Generation is creative work and Data Science Analytics may attract creative and passionate individuals. However, it is also possible that driven by the employment potential, the domain attracts individuals who are motivated by other than creative reasons. If data science and insights generation is creative work, then it is important that organizational culture imbibe creativity and allow creative expression of key departments. However, it is usually seen that Data Science and Analytics is relegated to second tier functions leading to poor results. This is exactly why data science and analytics is facing the challenges that it is facing today.

Now, we can discuss a little bit about the neurological components of Insights.

Mark Jung-Beeman, John Kounios, Ed Bowden of Northwestern University studied the origin of insights in our brain. They term “Insight” as the sudden comprehension of a solution which emerges in the mind of an unprepared subject. The researchers also differentiate insights from non-insight solutions in routine problem solving. The research highlights that the problem is solved at an unconscious level processed by the right hemispheric processes before an insight emerges. They used EEG (Electro-Electroencephalograph) and fMRI to understand the electrical and neural correlates of Insight. They also found something very interesting that creative individuals who experienced Aha type of insights deploy their attention in a diffused manner rather than intense focus. The brain wave electrical activity of these individuals also exhibited more stable resting state activity. Overall, this research pointed out that there may be pre-disposed patterns of internal brain activity which leads to more creativity amongst individuals.

Assuming that necessary and sufficient conditions are met for processing, generation and capture of Insights, the next questions that arises is whether all Insights are alike ?

All insights are not alike and the quality of insights defines the value of an Insight. In the next part, we will discuss a simple, yet, powerful framework to evaluate the quality of insights

Quality of Insights

Some insights create more impact than others. The impact may be measured locally or globally, at a company level or an industry level. It may impact a small cluster of individuals in a remote village in Asia or may impact a large swatch of population. Insights are a catalyst for creating change and all catalysts are not alike. We have devised the VARIC Framework to measure the value and the potential impact of an Insight.

Insights - VARIC Framework

All high impact Insights must pass the VARIC Test i.e Value, Applicability, Rarity, Illuminating a Breakthrough Path and Customized/Uniqueness.

In summary, Insights are emergent properties of our Intuitive Processing which meet the VARIC Framework and can be validated with results. Every organization or person may not be pre-disposed to getting high value insights just because they have access to data and analytical tools. The necessary and sufficient conditions must be met to achieve success in this area.

True Insights are rare and elusive as emergence of Insights occurs at the cusp of Brain Function, Genetics, Cultural Environment, Domain Knowledge and Pattern Recognition.

In the next series of posts, we will discuss case studies how individuals and organizations generated high value Insights.

Case Study 1 – Absolute Vodka – Search for Insights