In statistics, the long tail of a given distribution of data set is the portion of the distribution having a cumulatively large number of occurrences far away from the “head” or central part of the distribution. The phrase The Long Tail became well known when in 2004, Chris Anderson penned an article with the same name in Wired magazine, eventually writing a bestseller in the context of marketing.

The concept was born partly due to an influential essay by Clay Shirky, “Power Laws, Weblogs and Inequality”. The essay highlights the power law and emphasises that Diversity plus the freedom of choice creates inequality, and the greater the diversity, the more extreme the inequality. Freedom of choice will make stars inevitable. People will intuitively gravitate towards this power law distribution, but all the future successes will come from counter-intuitively getting oneself out of the power law distribution trap.

No alt text provided for this image

Netflix is a perfect example to show the immense hidden value in the long tail. As most people know, Netflix used to stream popular, mainstream movies on demand (the hits), while streaming a much lower volume of non-mainstream content. Netflix used to operate in the head of the “The Long Tail” curve (red zone) till disaster struck.

In 2017, Netflix had an accumulative debt of $ 20 billion when Disney’s announced that by the end of 2019, it will remove all of its content from the platform. Netflix’s future looked bleak at this point. As the company only offered popular movies on demand, it was caught by surprise when the content creators such as Disney started competing with the platform.

No alt text provided for this image

Netflix survived by diversifying its services portfolio, aiming at cable channels and by streaming a larger content choice which historically attracted a smaller customer base. Netflix also decided to make its own content rather than solely rely on being a platform. Using the long tail strategy, Netflix increased its subscriber base and revenues where the volume of on-demand content with lower market demand/sales volume captured more market share rivalling or surpassing the bestsellers.

The massive potential at the long tail of the curve comes with a caveat i.e the cost required to serve the long tail should be lower or equal to the cost required to serve the head zone. With data being streamed from centralized cloud-based servers, the cost was lower and made commercial sense for Netflix.

If the cost to extract value from the long tail is higher or if the value extracted is lower, moving into the long tail can be a risky business.

I will use the long tail concept in the context of Open Innovation and this theory was first formulated by Alpheus Bingham and Dwayne Spradlin, founder and CEO of open innovation platform InnoCentive. Using the long tail theory, we can make a case that during innovation initiatives, more value is dispersed in the long tail (open innovation) than it is within the confines of a company (closed innovation). If you are not familiar with open innovation, please read a quick primer here before proceeding ahead.

Bill Joy - Quote

The long tail theory is built on two foundational theories, the first one being the “Joy’s Law” coined by Bill Joy, the co-founder of Sun Microsystems. Bill famously said that “No matter who you are, most of the smartest people work for someone else.“. Though most people ignored what Bill said at the time, his statement was prophetic in nature.

No alt text provided for this image

The second foundational element of the long tail is based on economist Friedrich Hayek’s theory of classical liberalism which claims that knowledge is unevenly distributed. Hayek stated that “the knowledge that we wish to grasp never exists in a concentrated or integrated form but solely as the dispersed bits of incomplete and frequently contradictory knowledge which all the separate individuals possess“. In short, he meant that only by aggregation of the sum of its parts, we can understand the whole.

No alt text provided for this image

Joy’s law builds in Hayek’s components and claims “that in any given sphere of activity most of the pertinent knowledge will reside outside the boundaries of any one organization, and the central challenge [is] to find ways to access that knowledge.” The smartest people are hence, always outside your organization.

Let us build on this hypothesis further using the figure below. Let us assume that your firm or organization is in Zone B as shown below. Zone A is clustered with smartest people, Zone B is nearly smart and Zone C progresses towards the tail from enthusiasts, nearly smart to average Joes and Janes.

Your competitors are always trying to poach the best and hence the smartest people are in Zone A or in the process of moving to Zone A. In my former naval career, we had a truism about the career progression of officers i.e “Either you are written off or about to be written off”. We can reverse this here and claim that “Either you have lost your best or are in the process of losing them”. It is just that you do not know it yet. It is also possible that as you are also actively poaching smart people from other companies, you are in Zone A for your targeted companies. Zone A is unstable, so to keep the argument simple, let us assume you are most stable in Zone B for now (which is closer to reality).

Top of the Zone A has more agility followed by Zone C. Zone B is slow to respond to the environment directly.

Long Tail of Smart People

As we can see in the figure above, the number of people in Zone C are cumulatively higher than the number of people in A and B together. Unlike shown in the image, the tail is so long that it actually never touches the X-axis and that makes the cumulative total number of people in Zone C even higher.

Hence, for a company that is trying to solve an R&D problem or innovate, the number of solvers are more in Zone C than Zone A and B combined. The concept of hackathons became famous due to the above philosophy.

Point 1: The smartest people are either working for someone else, about to work for someone else or dispersed unevenly in the long tail making it challenging to extract value from them.

Let us build another argument here using a personal example. While in the Navy and we went through a rigorous Services Selection Board (SSB) screening. Few very people got in and I was surprised, amused and proud when they selected me. When a panel selected from a group of 18-year-olds, they make some predictions about their current and future performance. Obviously, they saw some personality and professional traits that increased the confidence that the recruiters had chosen a good candidate.

The panel also ranked selected candidates during their assessment. After 10 years or so, the recruiters circled back to check how their predictions had shaped out. They surprisingly found over and over again that very often the top listers have become bottom farers and vice-versa. It seems that the predictions had an average success rate of around 50% which is the same if you or me unskilled in hiring for military roles would randomly pick a candidate from lottery picker and make a guess based prediction.

It seems that the recruiters got it wrong, but a more complex issue was at play.

The military recruiters were predicting the performance of the candidate in the future while they were also predicting the future.

The real problem was that the future rarely turned out as predicted. After all, the recruiters were skilled in hiring candidates and not predicting how future complexity would evolve. The above example throws another wrench in the work for companies. If companies are recruiting for a role based on their assessment of current and future performance and they think they are hiring the best, they could turn out to be spectacularly wrong a few years down the line. To make things worse, if any smart hires actually get inside the workforce, the competitors will eventually poach them. Basis the above, it is unlikely that you will be able to innovate using your existing resources and capabilities. Unless you fix the recruiting process or the retention policies or both, Zone B is where you find yourself most of the time.

Point 2: Moden day hiring methods are ill-equipped to hire for the evolving future. When disruption shows up on the front door, the closed innovation framework and processes blunted by short-sightedness may not work.

One last point and then we will move into a real case study.

If we take a large distribution of people across the long tail, given one single vertical skillset, some people are highly qualified (head), some less qualified (middle) and some way less qualified (tail).

The less qualified and the way less qualified are easy to exclude in the conventional hiring process, but, maybe more equipped to innovate as they are least likely to be poached, have a diverse and broader skill set and may have future perfect skills that are easily missed. 3M was able to generate plural breakthrough innovations tapping into these less qualified lead users and being ahead of the curve.

Why would someone hire a music enthusiast when a Dolby sound engineer with a PhD degree is available?

Most companies equate on job performance with symbolic degrees or precedential work experience. No wonder that most of the innovations emerged from the long tail zone as most of the recruiting processes discounted these seemingly lesser qualified people who eventually did very well in innovative contexts (Jack Ma).

These institutionally discounted nodes (people) exist in organizations, communities and larger ecosystems. Although they are not in the head zone of the skills you were looking for, they are in the long tail of the skills you will be looking for in the future. Due to this, the best way to innovate is to tap into existing resources before venturing out.

Final Point: The next breakthrough innovations will emerge from people in the long tail. In hindsight it would make sense but looking forward, these people would seem least qualified to fit into conventional closed innovation culture and would be institutionally discounted. These long tail nodes are obscure, hidden and escape our pattern recognition due to institutional biases. One day, they will emerge and move from obscurity to ubiquity, creating a prediction shock.

Basis the above, organizations that want to innovate or create breakthrough innovations that will change the basis of future competition have four options at their disposal.

a) Change the hiring policies to include people who may seem lesser qualified than the seemingly perfect candidates. Let us add one more dimension to inclusivity which is to be more accommodative of the picture perfect qualification credentials requirements.

b) Tap the existing resources of discounted long tail nodes within the organization kickstarting closed innovation.

c) Tap the open innovation marketplace using a meta-innovation level strategy to extract value from the long tail of gazillions of possibilities.

d) Use a good combination of a) and b)

Let us now take a real case study from NASA to see how this is empirically validated.

Case Study: NASA: In 2015, NASA’s R&D budget had been reduced by 45% directly negatively impacting the “Constellation Program” an ambitious human space exploration program, which would ultimately take humans back to the moon for months at a time.

As the program aimed even for higher benchmarks i.e to send humans to Mars, they had a sense of urgency to understand how to increase the survival rate for the astronauts. Due to budget cuts and the inability of their own internal R&D teams to solve the problem, NASA decided to tap the long tail of expertise.

They released a public challenge titled “DataDriven Forecasting of Solar Events” on InnoCentive’s website. The problem was finding a suitable method to more reliably predict the solar particle storms originating with solar events. These storms contain energetic and ionized particles and can represent a radiation exposure hazard to spacecraft and astronauts above the protection of the earth’s atmosphere. They also have the more terrestrial consequence of impacting weather. 579 solvers looked at the challenge and eventually, 14 solvers ended up submitting solutions.

NASA issued a success award to Bruce Cragin, a semi-retired radio frequency engineer. Cragin earned his BS in Engineering Physics and his PhD in Applied Physics. He has 15 years experience in plasma physics basic research and another 13 years of industrial experience as a Radio Frequency engineer. He’s also a licensed PE in Michigan. The challenge was “right in the ‘sweet spot,’” Cragin said, “Though I hadn’t worked in the area of solar physics as such, I had thought a lot about the theory of magnetic reconnection. Also, the image analysis skills I acquired in the 1980s while looking into something called the ‘small comet hypothesis,’ turned out to be very useful.

As with many novel ideas, the fusion of skills and specific experiences allowed Cragin to see the problem and propose a solution that had escaped others focused primarily on the discipline of solar physics.

Cragin would never have been hired at NASA as he was somewhat qualified on the long tail and the institutional hiring process would have missed him. Cragin saved NASA billions of dollars and it is not difficult to imagine the impact he would have created had he been working at NASA.

In conclusion, where most closed innovation initiatives have a high failure rate, open innovation could be used to improve organizations ability to deal with disruptive forces. The best way though is to reduce the failure rate in closed innovation by tapping into the inside long tail and then venturing out into open innovation space to extract maximum value from the eco-system rapid cycling between the two systems.