Hype Cycle & The Data Products

Urban Dictionary defines hype as ‘Wild and flashy valuing style over substance with a breakneck pace’. Another word is a fad. A strategy where a product is publicized as the thing everyone must have, to the point where people begin to feel missing out without it. History provides a good starting point to discern the present state. So, to determine whether Data Products are result of hype, it is essential to examine the evolution of the data and analytics industry, as history serves as a valuable tool for understanding the current state.

Big Data Revolutionizes Financial Markets – The Wall Street Journal, 2012
Investment Firms Embrace Big Data Analytics for Better Returns – Bloomberg, 2013

World of data

Over a decade ago, Big Data dominated headlines, but its popularity has since waned. As I reflect on this topic, memories resurface of the enthusiastic migration from IBM Netezza to Hadoop Big Data, as if it held the ultimate solution to all our problems. Such hype cycles often arise from excessive optimism about new technologies, leading people and companies to get carried away with the allure of the next big thing, driven by the fear of missing out.

However, the reality of any new hype cycle or technology lies somewhere in the middle; it cannot magically address all the challenges one hopes to overcome. When Big Data emerged, organizations made significant efforts to transition from existing tech stacks, like Teradata and SQL Server, only to find themselves grappling with longer query run times in the new environment. The newer technologies are not inherently flawed, but the real challenge stemmed from those who hastily moved to this new shiny thing without making heads or tails of it.

With the advent of better technology, faster internet, and acceleration of digital connected experiences aka Internet of things, caused for data deluge and resulted in the need for tools and techniques to synthesize this raw data and produce meaningful/actionable insights out of it. This gave birth to another hot new thing, which was data scientist. Our beloved statisticians turned out to be data scientists, then it turned to machine learning, and now AI.

The convergence of advanced technology, faster internet, and the proliferation of digital connected experiences, such as the Internet of Things (IoT), has led to “data deluge.” Consequently, this surge in data necessitated the development of tools and techniques capable of processing and extracting meaningful and actionable insights from this raw information. This marked the rise of a new and exciting field known as data science.

Initially, data science transformed the role of traditional statisticians, empowering them to adapt to the changing landscape and become data scientists. As the field evolved further, it incorporated machine learning, enabling data scientists to employ algorithms and models to discover patterns and make predictions based on data.

Today, we witness another transformative shift as data science integrates with artificial intelligence (AI). AI takes data analysis to new heights, allowing systems to learn from data and make autonomous decisions. This continuous evolution reflects the dynamic nature of technology and how the quest for unlocking the potential of data continues to shape the landscape of innovation.

In each new cycle , we observe that easily implementable aspects change rapidly. In the evolution of data science, one consistent element that has undergone frequent transformations is the job titles. The roles have evolved from statisticians to data scientists, then to machine learning scientists, and now to AI leaders. Now, we are talking about Prompt Engineers as its own field. I’m going to resist my urge to make any comments around that. As with any cycle, time will tell.

Data Scientists: The New Rock Stars of Tech – TechCrunch, 2016

Data Scientist Positions Among the Fastest Growing in the Job Market – CNBC, 2018

What doesn’t change and where should one direct focus?

Through all these advancements, the fundamentals of data science have remained constant. You still need to understand the concept behind Bayesian statistics, linear regression, and the probabilities vs possibilities. The newer toolkit makes it easier to build models or draw insights, but one must prioritize learning ‘first principles’. First principles help you reverse engineer and break the complex problems into basic elements and build it back from ground up. Elon Musk and Charlie Munger are renowned for their consistent application of first-principles in all their endeavors.

This brings to data products…

What is a data product?

Data product is a product that facilitates an end goal through the use of data – DJ Patil, former United States Chief Data Scientist.

While this definition captures the core idea, it may be too broad for our specific understanding. Let’s consider an example to illustrate this point. Take Meta Quest Pro, an advanced virtual reality headset that heavily relies on user data to deliver its ultimate goal: providing a virtual reality experience. Can we categorize Meta Quest Pro as a data product? Probably not. In this case, Meta Quest Pro is a consumer product designed primarily to offer virtual reality experiences, utilizing data as a means to enhance its functionality rather than being a data-centric product itself.

In the same example, while the Meta Quest Pro itself may not be considered a data product, the Meta team could indeed have separate analytics products or tools that gather and analyze user data to gain insights into user behavior, preferences, and overall device usage. These analytics products can be considered as data products since their primary function revolves around data collection, analysis, and generating valuable insights to improve user experience, refine product design, and make data-driven decisions.

My definition of data product (data science lens)

---

A product whose primary objective is to transform the raw data into meaningful and actionable insights for its users, while adhering to the principles and lifecycle of product management.

---

We must unpack two elements of this definition: 1) The MA Framework (Meaningful and Actionable), and 2) The principles and lifecycle of product management.

The MA Framework

1. 1. Meaningful insights should help users gain clarity, make informed decisions, and derive valuable insights from the data. Your data product should effectively answer specific questions related to the problem or objective at hand.
2. 2. Actionable insights takes it a step further by providing recommendations that users can act upon to drive positive outcomes. It goes beyond fact reporting or descriptive statistics.

Not to mention, the above should be supported by having accurate, reliable, and timely information. Without it, your data products are just a garbage.

Below is an example to differentiate Meaningfulness and Actionability.

Business : We would like to understand the customer segments for upcoming marketing campaign focused on movers to capture their share of wallet.

Data Practitioner 1 : Here is the raw excel export of all the customer data and demographic information.

Data Practitioner 2 : Here is the dashboard with collection of charts and graphs on different customer segments available; Doesn’t provide any context or annotations.

Data Practitioner 3 : Here is the data product (agnostic of tools) that analyzed all the existing customer data, and identified particular segments of the population that could benefit this marketing campaign based on their past purchase behavior. These visuals also represent new, and lapsed customer segments who could potentially benefit from this marketing event.

The above conversation is as real as it could get. As you may agree, the DP1 & DP2 are just data pulling machines, we don’t see any elements that help users of this data to gain clarity or nudge for meaningful decisions.

However, the #DP3 not only analyzed all the existing information, but also provided recommendations on what could be the potential next steps. We see guidance and persuasion of business teams to move in the right direction. It has both meaningful and actionable information.

One could argue to classify the #DP3 example as analytics or insights work, which is true if we consider just the first part of the definition. What truly qualifies it as a data product lies in the incorporation of the second element of the definition – ‘adhering to the Principles and Lifecycle of product management.’ This element distinguishes it as a fully-fledged data product rather than just an analytics output or insights presentation.

MA FRAMEWORK + PRODUCT FRAMEWORK = DATA FRAMEWORK

The MA framework works very well with data science and analytics problem statements and might not squarely fit to other areas like software applications, APIs, etc. which by the way are data products too.

Data Products → Data as a Product

The inherent confusion on what fits to be a data product comes from its broad scope and lack of specificity. By making a slight adjustment to the verbiage to ‘Data as a Product’ significantly narrow down the scope and better categorize the offerings that fall under this concept. So far, we have covered data science evolution, and touched on the first element of the ‘data product’ definition. Now, we are going to focus on the second element, ‘Principles and lifecycle of Product Management’.

The Principles and Lifecycle of Product Management

You’ll find a ton of material covering core principles of Product Management. So, I’m not going to belabor the boiler plate. In essence, the core philosophy of PM centers around understanding and fulfilling customer needs and wants, emphasizing continuous iteration and improvement, driven by data-driven decision making, and the value prioritization to deliver successful and customer-centric products. From a tactical perspective, this includes OKRS, roadmaps, and timelines to bring alignment, avoid scope creep and drive the focus to achieve stated objectives.

The true beauty of ‘data as a product’ comes when we embrace the principles of product management as a framework around data and analytics. While the first-principles of data and analytics remain valid, adopting a product mindset adds a valuable layer of perspective and focus to the process.

So, can we categorize all your 100+ Tableau dashboards and 50 different statistical models as data products? Fortunately not! To determine if something qualifies as a data product, we should begin with the MA Framework: Is the data meaningful and actionable for its intended audience? Secondly, we need to consider whether the product addresses user needs and wants, if it aligns with OKRs, if there is a roadmap in place, and if there are plans for iterative improvement and evolution from the MVP. These are essential questions to ask before labeling something as a data product.