Enabling Data Democratization with a Data Marketplace

A custom data marketplace platform can decentralize and maximize the power of data for an organization. We’ll explore the features and benefits of implementing a data marketplace.

The data democratization opportunity

The era of data-driven decision-making has transformed businesses across the globe. The organizations we work with at Credera have heavily invested in building modern cloud data platforms, leveraging lakehouse principles, and building on platforms like Databricks, Snowflake, Azure Synapse, GCP Big Query, and/or the AWS Data Platform.

Equipping data owners with the tools they need to build a modern data platform is crucial for delivering early value to the business, but what happens when demands for solutions using that data inevitably surge? Even the most formidable teams can be overwhelmed by the pace at which they are asked to build new data products to answer business questions. For many organizations, this is often a direct result of a disconnect between users and data resources. In the State of Analytics Engineering in 2023 report, 42% of analytics leaders said their top concern is “data isn’t where business users need it.” This bottleneck problem dealing low visibility of data further slows the process of tackling the growing list of data work needing to be done.

The best way to remedy this problem? Expand the number of people in the organization capable of building their own data products (aka data democratization).

Companies that effectively democratize data—the practice of enabling data access, use, and literacy across all roles in an organization—are reaping significant bottom-line rewards.

Studies have shown since at least 2011 that companies that successfully execute data democratization experience higher levels of innovation and performance:

• One study that used detailed survey data on IT investments among publicly traded firms found that companies that democratize data-driven decision-making have output and productivity 5-6% higher than what would be expected given their other investments and information technology usage.

• According to another study, companies that have democratized data-backed decision-making are more than twice as likely to be in the top quartile of financial performance within their industries.

When data owners can identify and capitalize on data that can drive key decision-making, the organization may reap significant financial benefits.

Building an internal data marketplace

The challenge in ensuring the right data is accessible to business users is that they often do not know where they can source the data, or which data they need, until the question arises. Teams have responded by making the delivery of data more efficient by sending data to business users as soon as they receive requests. These data platform investments can be compared to Amazon’s system of distribution centers, enabling next-day delivery when the business asks for data. Unfortunately, the realization of business value is limited when users are limited in their awareness of data accessibility.

To solve this problem, organizations have invested in data catalogs to index all data and make it searchable by the business. However, most of these catalogs offer a data discovery experience like the paper mail order catalogs of the past. Users are typically blocked by not knowing where to find the data they need without flipping through an entire data inventory.

Furthermore, business users often receive little guidance on how they can query for the data they need. Oftentimes, they source the wrong data and lose trust in the platform when it is used incorrectly to make poor business decisions. Additionally, trying to track down the appropriate contact and getting the proper access and setup can eat up valuable time that could be spent solving the actual business problem at hand.

Figure 1: Data catalog with difficult to navigate user interface.

Today’s business users demand a 2023 data discovery experience akin to today’s eBay for secondhand shopping. In other words, they need a data marketplace.

A data marketplace exists to simplify the experience of “shopping” for data. It resembles a real marketplace in that the (data) products you are looking for can be quickly located (and “purchased” by requesting access from the appropriate contact). It serves as a one-stop shop for anyone’s data needs, which allows for quick data identification, trust, and collaboration between business users and data teams. Flattening the hierarchy of data access makes it easy to scale data operations by prioritizing ease of discovery and availability to all users.

Key data marketplace benefits

Navigating the many complex layers of data can take significant time, and the data marketplace aims to speed up the process of locating the right data. It aims to be domain specific to allow users at any literacy level to browse and find what they need quickly.

• Easy search: Google Search-like functionalities enable data consumers to find exactly what they are looking for without having to consult a more knowledgeable resource.

• Faster delivery: When the data has been found, the platform can notify the point-of-contact immediately to expediate delivery of data from the storage location.

• Highlight common usage: If the data of interest has been visited frequently by others, the data marketplace can point to previous analyses that can serve as an immediate solution or at least a starting point to leverage.

• Minimize back and forth: Data producers can save significant amounts of time and increase workflow by enabling business users to reference their outputs more quickly without wasting time with back-and-forth exchanges.

Optimizing the data consumer user experience

The data consumers you are expanding to are not likely to accept a frustrating data sourcing experience. When building out the ideal data marketplace, it is important to proactively understand your customers and create an engaging experience that can adapt to all users. With the right approach, we believe the organization can arrive at a solution that can build trust in enterprise data.

We recommend the following four best practices to accelerate a data-driven marketplace:

Figure 2: Emphasizing a user-friendly design.

1. Ease of use

Users should see simple instructions on how to use search or browsing features. They should be able to better understand the marketplace as they navigate through the site and be drawn to revisit it later for similar situations. The process of “shopping around” should be simplified as much as possible to encourage widespread use and make it an enjoyable experience.

2. Recommendation engine

The layout should also be structured so data is consumable to both technical and non-technical users. They should be presented with relevant assets at their preferred level of granularity. This is achieved by designing versions that are accessible to customer personas (specific to your organization) like admins, data scientists, and different business professional roles. From there the marketplace can surface the most relevant metadata and offer recommendations that adapt to user behavior.

For example, the recommendation engine could utilize a user’s department and/or title information to yield the most used data products to which they have access. This would result in faster retrieval of data and direct users to custom data product details pages to facilitate understanding.

3. Measuring engagement

To avoid user dropout, feedback should be collected to ensure the marketplace satisfies user needs. After all, a product that no one plans to use after a trial run probably has some underlying issues that need to be addressed.

After a user’s first experience, the data organization should follow up with them to see if they were able to find exactly what they needed. Did they find the data product they were looking for, or did they quit because it was too frustrating to navigate the site? And if they did find it, did they find it quicker than if they had tried reaching out to someone offline? All these questions should be answered during this testing phase as we seek to continuously improve the “time to success” key performance indicator.

4. Network effects

One of the best things about a data marketplace is its ability to create a virtuous cycle of data usage and data stewardship (management and oversight of an organization's data assets), eventually leading to a data culture. Once users directly experience the benefits of a data marketplace in helping get the data access they need, they can, in turn, make their own contributions to the system by supplying their knowledge for the data products they manage and see how it can help others.

If the value of the data products can be realized and passed onward, the positive network effects can encourage others to become not just data consumers but data stewards as well.

Figure 3: Data harmony among users and producers.

A data marketplace case study

Credera partnered with NRG, a $30 billion energy company, to build a data marketplace over their Databricks and AWS data lake architecture to meet NRG’s goal to democratize their data among their many users. The custom user interface (UI) of the site helped drive the “one-stop shop” analogy to fuel data consumer trust and collaboration.

By surfacing data assets, ownership, data lineage and other metadata in an intuitive domain-specific UI, business managers and data analysts across the organization were better able to find and leverage data, and the NRG executive team grew their understanding of the data products that support their business.

See full details on the case study.

Credera’s data marketplace accelerator on Collibra & AWS

Based on Credera’s experience developing data marketplaces, we have learned that much of the upfront costs are tied up in making the metadata easily accessible to the front-end team. So we developed an accelerator that delivers a pre-built flexible and intuitive API layer for data marketplace teams to leverage.

The Credera data marketplace back-end accelerator enables teams to focus on the right approach to promote collaboration and build trust in their data consumer base. For clients who have already implemented a cloud data platform and are using the Collibra catalog, this code asset enables the rapid acceleration of the development of a custom data marketplace front end to enable data democratization.

Figure 4: Credera data marketplace accelerator.

Data democratization fueled by a data marketplace

Optimally leveraging data has remained a common pain point across organizations, with more and more business functions growing more reliant on quick access to relevant data sources. As a result, Credera has noted an increased need for modern governance practices and internal data marketplaces. Without them, organizations are at the mercy of mismanaged data silos while trying to meet the increased demand for data products.

To work through the data culture journey, organizations need to develop an efficient operation model by democratizing data resources to all, without relying on the efforts of subject matter experts. Increasing reach is the best way to expand both knowledge and expertise in data, by equipping people with a tool like the data marketplace to encourage faster business outcomes.

The effectiveness of internal data marketplaces has been well documented, with companies like NRG using it to create a data-driven culture that prioritizes sustainable and scalable data usage. Used responsibly in the hands of everyone, a data marketplace can encourage a path to stewardship that brings out the best of governed data.

If you’re interested in continuing this conversation about data democratization and the data marketplace, reach out to us at findoutmore@credera.com.

• Data & Analytics
• Data Clean Rooms
• Data Collection
• Data Marketing
• Data Marketplace