Contemporary data management ensures data can be reused seamlessly and as needed to equip the business.
- Gartner client? Log in for personalized search results.
Modernize Data Management to Increase Value and Reduce Costs
Decide today where data fabric fits in your data management strategy
The number of data and application silos has surged in the last decade, making it difficult to deliver integrated data quickly and easily. Data fabric can help but is still an emergent data management design. Download this essential guide to:
- Understand how data fabrics could eliminate manual data integration tasks and augment or automate data integration design and delivery
- Communicate its value
- Start mapping your way forward
Evolving your data management strategy to drive business value
Traditional data management defines data and then tries to integrate, qualify, master and fix it. Modern data management approaches are AI-enabled to capture value faster. Here’s what to know.
- Data Integration/Quality
- Data Infrastructure
- Management Challenges
- Applied Data Management
Embracing new data management strategy best practices
The continued shift in infrastructure toward the cloud requires new technologies and skills for implementing and evolving data integration, data quality and metadata practices.
Data integration
Today’s data integration approaches must enable distributed and augmented data management. The trends affecting this shift include:
Market leaders losing ground to smaller vendors. The combined market share of the top five vendors is shrinking as leaders concentrate more on integrated platforms and less on point capabilities.
Cloud data ecosystems driving growth. Vendors gaining market share by focusing on specific data integration styles, such as data virtualization or data replication.
Data fabric and the push toward augmented data integration. The market is demanding solutions that enable automated and augmented data integration and support.
Data mesh gaining traction. Organizations are seeking access to more decentralized data product delivery through agile capabilities that deliver data as a product to domains and business teams.
FinOps and financial governance for cost optimization. As data and analytics teams are placed increasingly in lines of businesses, leaders need a way to associate the cost of running integration workloads to the value associated with them.
Support for hybrid and intercloud data management. Nearly half of data management implementations use both on-premises and cloud environments, making hybrid management and integration pivotal in the market.
Independent data integration tools to prevent lock-in: This is especially important for organizations using applications with embedded data integration capabilities.
Data quality
Data quality is the top priority in data analysis initiatives for the impact it has on driving business value. Yet enterprises confront the following challenges related to data quality:
Limited internal capabilities and strategies. Organizations struggle to scale and unite disparate data quality efforts for enterprise success and benefits.
Data inconsistency due to data silos. Data silos make data standardization much harder, resulting in data inconsistency.
Lack of ownership and collaboration. Data quality practices that focus on the project or data domain level, not at the enterprise level.
Legal and reputational risk. Regulatory compliance and privacy laws restrict organizations on how to use and manage certain types of data, especially personal data.
Lack of smart and augmented data quality capabilities. Data quality practices require significant manual efforts and rely on SMEs to assess and remediate data quality problems.
Traditional data quality tools have limited impact on these issues. This has fueled an evolution toward augmented data quality solutions, available in unified data management platforms and stand-alone data quality solutions.
Metadata management
The term “metadata” describes facets of a data asset used to improve its usability throughout its life cycle. Metadata is the foundation of data management, since metadata management enables organizations to deliver key data benefits, such as:
Data reuse
Clear identification of data assets used in regulatory roles
Improved risk management and better assessment of the impact of change
Reduction of technical debt
Systemic coherence across applications and models consuming metadata
Assessment of data content relative to business purpose
Implementing operational and analytical data management infrastructure
The logical data warehouse (LDW) is a current data management infrastructure best practice, as evidenced by its current position in the Plateau of Productivity on the Gartner Hype Cycle for Data Management.
The LDW evolved from the enterprise data warehouse (EDW), which gathered an organization’s data in a single place in a common format so it was accessible and usable by all. However, the growth of big data, the Internet of Things (IoT) and social media, as well as the increased complexity of cloud, multicloud and hybrid environments made storing and processing every kind of data on a single relational DBMS server infeasible.
Data management strategy had to evolve to embrace a model in which a collection of analytical servers, data warehouses, data lakes and data marts are logically integrated with metadata and data virtualization — the LDW.
Pros and cons of logical data warehouses
The advantages of an LDW include:
Provides access to a variety of data, as well as local and remote data
Preintegrates multiple analytical components
Provides a single logical view of the data
Enables multiple exploratory views
LDWs have a major disadvantage, however — they require targeted, appropriate, but significant design and management effort. For that reason and others, organizations have already begun the next evolution in data management strategy toward the data fabric. LDWs provide a strong foundation to enable this shift, since they already have integrated metadata, established interfaces between multiple servers and multiple data integration methods.
The promise of the data fabric above and beyond the LDW is to provide an abstraction layer above the dispersed data systems it touches, and automate many tasks that data scientists must do manually for an LDW. Data fabric also will provide augmented features, which enable the system to advise users on what data to use and which datasets to combine. The data fabric will self-administer to a significant degree as well.
Pros of data fabric
The advantages of a data fabric include:
Provides access to distributed data
Learns behavior patterns of data usage
Supports operational and analytical use cases
Leverages use-case awareness and learning engines
Permits significant data management automation
Enables AI capabilities for data management augmentation
Despite these expected advantages of data fabric, it is still early in its development and subject to a great deal of hype. It will take between five and 10 years to reach maturity.
Anticipating data disruptions caused by new use cases and technologies
Despite the ongoing progression to the cloud and the anticipated changes in data management strategy, the current state is defined by a multitude of data silos, traditional people-led data management practices, and established technologies, roles and change management processes. These characteristics lead to ongoing issues, such as slow time to market, low productivity and poor self-service.
That future state will be built on a foundation of augmented data management capabilities for data storage, integration, metadata and governance. The characteristics of this future state will include:
Financial governance: Optimal use of data management resources
DataOps: Operational excellence in data delivery
Data fabric: Flexible access, integration and management of data
Transitioning from the current to the future state will require data analytics leaders to address the gaps that exist and the management challenges organizations face resolving them. These gaps fall into three broad categories:
Technology gaps
Cloud data management is technically different from on-premises data management, creating several gaps organizations need to address. For example, cloud applications have different approaches to: managing resources, including jobs, users, clusters; storing and processing data; and managing networks. These differences mean that organizations will not be able to move all existing applications as-is to the cloud. In some cases, you will have to reengineer the underlying architectures, solutions and operating models.
Operations gaps
Operations (ops) plays a critical part in ensuring that your organization realizes value from data management investments. There are various ops tasks, including monitoring, performance-tuning, housekeeping and ongoing maintenance. The timing of upgrades and renewals also plays a role. Migrating your data management to the cloud will not necessarily address these ops gaps without a conscious eye on continuous improvement. Ask: Which ops activities can we retire? Which ops activities will change? Which new ops activities will be required?
Skill gaps
Major changes like cloud migrations often expose skill gaps on the data management team and need for new roles like cloud architects, platform engineers and data engineers.
These gaps are interdependent. Prioritize the following three actions to start addressing them:
Financial governance planning. Organizations can avoid unpleasant budget surprises by engaging in robust financial governance practices. The ultimate goal is to adopt formal FinOps practices that connect the cost of workloads to their business value in an iterative cycle that adapts to the changing needs of the business.
Cloud data ecosystem planning. The three options for data management infrastructure on the cloud include using the native services from your cloud service provider, using the native services of an independent software vendor or combining features and functions from each in a blended approach.
- On-premises data ecosystem assessment. Plan migrations to the cloud by focusing on two core characteristics of the data management landscape: data gravity and entanglement. Data gravity is the force that couples applications that share/exchange data. Entanglement refers to how frequently components of a logical application exchange data with each other.
Optimize data management strategy for real-world use cases
The ultimate goal of data management is to enable business value through concrete use cases. While the overall trends in data management strategy point to modernization through cloud infrastructure and data fabric, modern use cases will also drive architectural decisions such as unified data lake and data warehouse patterns, as well as special-purpose data stores, to consolidate data storage and processing.
For example, marketers have been embracing customer data platforms (CDP) to enable their key use cases. A CDP is a marketing system that unifies a company’s customer data from marketing and other channels to enable customer modeling and optimize the timing and targeting of messages.
Marketers want increased autonomy over the management and use of customer data. However, not all CDPs provide the same capabilities toward that end, and the data needed to meet these needs often comes from systems outside of marketing. Moreover, many of the core functionalities of CDPs already exist in other tools that may be more suited for certain use cases. For example, CDP vendors reuse some of the terminology associated with master data management (MDM) like “golden record” or “single customer view,” but not all CDPs enable a persistent customer profile. For these reasons and others, data and analytics leaders and IT teams have become productive partners with marketing in CDP selection and deployment.
Expect to engage with other functions of the organization to guide their specialized data management strategy — such as in finance or the supply chain.
Experience Gartner conferences
Join your peers for the unveiling of the latest insights at Gartner conferences.
Related data management resources
Gartner clients: Log in for a complete suite of actionable insights and tools on data management.
Frequently asked questions on data management
What is master data?
Master data is the minimal set of data that describes and delineates those entities core to an organization’s existence — for example, customers, citizens, suppliers, patients, students, locations, products, parts and assets. Master data has specific classification and usage criteria that enables the business to identify candidate master data attributes.
What is master data management?
Master data management (MDM) is designed to increase consistency in the most important data at the center of the most important business decisions and processes. Organizations that benefit from a single, trusted enterprisewide view of this master data are far better positioned to make informed decisions directly impacting the risk, revenue, value and cost goals of the organization than those who do not.
What is data management and why is it important?
Data management addresses the complexity of state, store, access, quality and context of data to enable organizations to realize the enterprise’s data-driven aspirations. This capability is central to digital business success where being data-driven means using data, whatever its source, to make better business decisions.