By Steve Leeper, Datadobi
Artificial intelligence promises transformative outcomes — but only when it’s built on a solid data foundation. For many organisations, the real barrier to AI success isn’t algorithms or compute power; it’s the challenge of managing sprawling, complex datasets at scale.
Unstructured data, spanning everything from images and video to emails and collaboration content, now makes up the bulk of enterprise information. Without smarter approaches to organising and governing this data, AI initiatives risk being undermined before they even begin.
It’s not just about volume – it’s also about quality; an issue in some ways similar to the classic computing principle of ‘garbage in, garbage out’. Considering that AI models often struggle to work effectively with unstructured data, especially when it has been poorly managed, organizations can easily find AI outputs skewed by poorly managed data, leading to unreliable performance and raising serious business, security, and compliance concerns.
The issues are very real. Earlier this year, for example, Gartner said that 63% of organizations either do not have or are unsure if they have the right data management practices for AI, and that “through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data.”
The basic issue is that unstructured data files, which can exist in the billions, need to be identified, organized and visualized in a manner that can keep pace with rapid developments in AI systems. This can potentially be a highly complex and resource-intensive task, but there’s no escaping the fact that choosing the correct data is critical for producing accurate, actionable, and unbiased outputs.
Armed with these capabilities, however, organizations can accelerate the identification of the correct data to train AI models that address their performance improvement priorities.
The data management headache
Managing unstructured data is a significant challenge because it’s scattered and dispersed – literally. It doesn’t follow any specific format, so trying to categorize, search or make sense of it can feel like an impossible task, particularly as there is so much of it. It’s also a moving target – companies generate and collect unstructured data constantly, and without the right tools, it’s easy to get overwhelmed and miss out on valuable insights hidden in that mess.
At the heart of the matter is the need to gain insight into what data exists, where it resides and its value. Essentially, businesses need smarter data management policies that extend far beyond storing all data in perpetuity, not to mention the potential risk issues that grow commensurately with the capacity of the data being stored. Without the ability to automate classification, tagging and lifecycle management, IT staff spend increasing amounts of time firefighting storage and compliance issues rather than enabling innovation.
To cope with data growth, it has historically been far easier to simply add storage. As a result, accumulation rates have jumped sharply higher in recent years. What’s required instead is a shift in focus away from where the data is stored and towards how it is managed. Of course, if organizations were more cautious about the amount of data they collect, then these would be easier problems to address. The reality, however, is that we operate in economies where, rightly or wrongly, data has intrinsic value. As a result, the go-to approach is to collect it from just about every available source, even if the organization in question has no specific plans for it.
Embracing complexity
Instead of focusing on the device where unstructured data is stored, IT leaders should focus on how it is managed. What’s required is an approach that can span the entirety of the environment and provide a single point from which data can be monitored and managed.
Organizations preparing for AI implementations need to address their data strategy, with an emphasis on data management, governance and enterprise-wide visibility. In particular, data management technology can provide deep insights and actionable intelligence on unstructured data, enabling it to effectively train AI models to produce high-quality outputs.
A key part of this picture, of course, is the adoption of vendor-agnostic data management technologies that can seamlessly integrate unstructured data across diverse storage systems, applications, and cloud systems.
Effective data management technology can bridge the capability gap between raw unstructured data that contains latent value to a situation where teams can train AI models with high-quality data. Ideally, the data management solution used with AI systems will deliver seamless, reliable and efficient data migration, management, and protection across heterogeneous storage environments.
Creating an effective AI data pipeline also depends on having access to data that is accurate, consistent, complete, and relevant – a requirement that drives the need for highly effective quality management processes. In turn, the management process depends on adherence to comprehensive data governance policies and procedures to ensure that all data within an organization is appropriately documented, stored, and maintained.
Effective data governance policies and procedures include conducting regular data audits, assigning clear ownership and responsibility, and establishing guidelines for data creation and storage. Effective data governance is also essential for mitigating risks associated with unstructured data, such as security vulnerabilities, compliance issues, and operational inefficiencies. In this context, AI and data management strategies can co-exist in a way that balances our collective need to collect and retain data with building reliable and trustworthy models that deliver positive business impact.
