Future of Data Architecture

Data analytics architecture best practices have passed through a number of eras over the past decades, as digital transformation initiatives have highlighted the need to modernize the data strategy and leverage opportunities to use data. These eras include:

• Pre-2000 period — Enterprise Data Warehouse Era: Data architecture centered on the success of the enterprise data warehouse (EDW).

• 2000-2010 — Post-EDW era: This period is defined by fragmented data analysis, whereby data marts were dependent on the data warehouse. And depending on who you asked, you had a different version of the truth, as each data mart consolidation led to yet another data silo leading to fragmented and inconsistent analytics.

• 2010-2020 — Logical Data Warehouse (LDW) Era: This period saw more unified analysis of data via a common semantic layer, that enabled access to data warehouses, data marts and data lakes. This is current best practice.

• 2020-future — Active Metadata Era: The future will see augmented analysis of data using all the relevant data sources, accessed and enabled by advanced analytics, recommendation engines, data and AI orchestration, adaptive practices and metadata analysis. 

Democratizing data access and self-service analytics is motivating the current evolution underway from the LDW Era to Active Metadata Era. Chief data and analytics officers (CDAOs) likewise hope to expand the use cases for data beyond that which LDWs can handle. These include master data management, interenterprise data sharing, B2B data integration, partner data sharing, application data integration and others.

But what is metadata, and what role does it play in this evolution?

Metadata describes different facets of data, such as the data’s context. It is produced as a byproduct of data moving through enterprise systems. There are four types of metadata: Technical, operational, business and social. Each of those types can be either “passive” metadata that organizations collect but do not actively analyze, or “active” metadata that identifies actions across two or more systems utilizing the same data.

Data fabric is an emerging data management and data integration design concept (also see “Modernize Data Management to Increase Value and Reduce Costs”). Its goal is to attain flexible, reusable and augmented data integration to support data access across the business. 

Data fabric is a natural evolution for many organizations from their logical data warehouse models because it leverages existing technology and metadata in a modernized data architecture. There is no “rip and replace” with a data fabric design. Instead, it capitalizes on sunk costs while providing prioritization and cost control guidance for new data management spending.

Data fabrics deliver benefits across different perspectives:

• Business Perspective: Enables less technical business users (including analysts) to quickly find, integrate, analyze and share data

• Data Management Team Perspective: Productivity advantages from automated data access and integration for data engineers, and increased agility resulting in more closures of data requests per day/week/year

• Overall Organization Perspective: Faster time to insight from data and analytics investments; improved utilization of organizational data; reduced cost by analyzing the metadata across all participating systems and providing insights on effective data design, delivery and utilization

Data mesh is an architectural approach that allows for decentralized data management. Its goal is to support efforts to define, deliver, maintain and govern data products in a way that makes them easy to find and use by data consumers. Data mesh architecture is based on the concept of decentralizing and distributing data responsibility to people who are closest to the data and to share that data as a service.

The most common drivers for data mesh are: More data autonomy for lines of business (LOBs), less dependency on central IT and leveraging decentralization of data to break down silos (though some data centralization within a mesh architecture may be warranted). Despite its obvious appeal, be aware of the following prerequisites and challenges.

Data mesh architecture is not yet an established best practice.

The term is associated with varied approaches that differ by organizational model, management of the data and technology implementation. The organizational drivers also vary. They include removing IT as a bottleneck and rationalizing siloed datasets resulting from LOB-led data pipeline creation, or triggered by a cloud-modernization data-management initiative.

Data analytics leaders should not adopt data mesh architecture as a seemingly easy solution to their data management challenges. Although it formalizes common practices, it abdicates data accountability to LOB experts, which risks proliferating siloed data uses.

Data mesh success depends on the organizational model and data skills in LOBs.

If data literacy, autonomy and data skills vary greatly across departments, and if organizations lack the ability to operationalize data management activities, central IT will need to provide more support — at least at first. LOBs can evolve toward greater autonomy within a data mesh environment by creating new roles, such as data product owners, to manage the definition, creation and governance of data products. Organizations that lack commitment to building distributed data skills, however, should avoid data mesh.

Data leaders operating with on-premises, cloud, multicloud, intercloud and hybrid deployments will need to revise their existing data architecture strategy to support their present and future complexity. A carefully planned and robust data architecture ensures that new technologies cohere with existing infrastructure and can support future demands — including integration and interoperability across cloud providers, SaaS solutions and on-premises resource deployments, among others. Focus your planning around the following activities:

• Devise a strategy that addresses the whole data ecosystem. It's common even for organizations with initial cloud deployments to grow into a hybrid and multicloud environment over time. Establishing an overarching cloud strategy that prioritizes providers can govern additional cloud deployments. This will mitigate the risks that unsanctioned cloud deployments can pose to your data architecture.

• Align data requirements to use cases. Distributed and complex use cases are now driving newer innovations that deliver business value — in particular, by enabling self-service data access. Success in the cloud will depend on the ability to satisfy business consumer use cases, which are most likely distributed in nature, close to data sources and operating on edge networks and devices.