Data: Unused, unanalysed and undervalued?

The time has come for companies to adapt their information management strategies to become more efficient, differentiate themselves from the competition, mitigate risk and comply with the continued onslaught of regulations.

By Gaurav Verma, SAS

We think of our products/services, people and intellectual property as strategic and vital to our organisations. It’s now time to include data in that category. By 2010, digital data will grow at an unprecedented rate and for many businesses most of it will go unused, unanalysed and under-appreciated and undervalued.

You need to study the acquisition, creation, usage, maintenance, integration, storage and accessibility of data to understand the hidden wealth in your system. Data rarely improves with age. Left unmanaged, information decays, content loses context, meaning is lost, and unstructured data is lost due to a lack of taxonomies and indexing.

For companies to derive maximum benefit from their data assets, they need to think about it holistically. This is no small task as many of the people tasked with collecting and managing data are analytically strong, not necessarily empowered to be 'big picture' types. It also isn’t simple because a holistic view means looking at all your data assets, be they structured, unstructured or semi-structured, in its entirety, and razing the silos that keep information locked in separate systems.

Many executives complain that their companies are not getting maximum value from information. Fewer still have a systematic plan for evolving information capabilities to the next level. Organisations have typically invested in information-related technologies for years but in a siloed approach based on the type of data. Structured data goes to one silo; unstructured data to another. In most organisations 70% of the data is unstructured, 25% is structured and 5% is semi-structured. Once in their respective silos, the data is managed by different systems.

BI solutions are deployed to manage structured data. Enterprise content management (ECM) solutions manage unstructured and semi-structured data derived from web content, email and records management tools. Keeping the information separate deprives companies of the opportunity to see patterns that emerge when the data is housed together

Integration of these different data assets at a transactional level has been achieved successfully over the years, but the approaches to this fractured data paradigm have their drawbacks.

Integration at the data tier using transformation engines that attach an XML or an XML-derivative metatag to content has inherent limitations. The process is rule-driven and needs to be defined, governed and maintained. Failure to enforce and maintain the rules base as newer types of data are infused rapidly marginalises the usefulness of this integration approach.

Enterprise search has its genesis from content and document management that is now being applied to the structured data world of BI where it allows for high availability in a highly easy-to-use environment such that the indexing is driven not just by the header in a report but the actual content in the report as well as the metadata associated with the report. Enterprise search now gives the ability to search across these two silos but the ensuing result sets are extremely large given the inherent indexed based design paradigm of search.

A harmonised approach

A more precise search result is predicated by a precise search string, negating the ease-of-use and exploratory utility of search. Advanced techniques like learning algorithms, scoring, prompting, filters etc. are definitely improving the search results and newer Web 2.0 concepts of social tagging are beginning to work their way into the enterprise but again require governance frameworks.

Organisations that have invested in text mining/analytics tools have done so mostly in isolation and typically do analysis on structured and unstructured data separately and if they join results at all, it is at the report level. This potentially gives them a higher confidence in their analysis given that they have leveraged all their data assets. But because of the ambiguity and numerous ways to represent similar concepts, information implicit in text-based data is not easy to discern, quantify or analyse in correlation to associated structured data when done in isolation.

For BI to live up to its potential, organisations need to transform unstructured and semi-structured data into a structured format so the facts (address, complaints, parts) about key entities (customers, accounts, products) can be extracted and stored in a file or database. The real value in recognising these variables is relating them with similar facts derived from transactions systems, so organisations can apply analysis to predict something before it happens and take corrective action.

These applications require a harmonised analytical approach to structured, semi-structured and unstructured data so that entities, facts, relationships and sentiments are related and integrated at the source, and normalised and optimised for analytics and reporting. This also allows an automated, highly refined and integrated metadata and taxonomy definition process for all data assets within an organisation, potentially providing the underpinnings for enterprise ontology.

A data model that represents a set of concepts within a domain and the relationships between those concepts, enterprise ontology is used to reason about the objects within that domain and influences a highly effective search paradigm for the entire ecosystem of users within an organisation. This integration of unstructured and semi-structured data can be defined into the established discipline of BI and information extraction and analysis as information management.

Early adopters of information management include manufacturing companies with early warning systems for warranty analysis, government intelligence agencies and financial services companies. Gaining traction are customer experience intelligence and voice of the customer initiatives, as well as PR, reputation, brand and competitive intelligence applications where Web 2.0-influenced online communities are breaking the traditional channels of influence on an organisation’s brand and competitive mix.

While this confluence calls for a coherent information management strategy on the part of CIOs, information management is not a product but a strategy or approach for an organisation to leverage information to be its most compelling asset regardless of type or source. The time has come for companies to adapt their information management strategies to become more efficient, differentiate themselves from the competition, mitigate risk and comply with the continued onslaught of regulations.

With so much data, and so many needs to extract knowledge from it, we are now at a critical moment. For many companies, the time is right to make information management part of their corporate DNA – to incorporate it seamlessly in all things they do. Information management is important for all levels of an organisation, especially those that are looking for the autonomy to make their own business decisions based on enterprise information. Built on a comprehensive platform, an effective information management strategy lets business leaders take their gaze off the rear view mirror and focus on the road ahead, resulting in information that is accurate, timely and meaningful.

Gaurav Verma is director of BI marketing for SAS.

Other articles in this focus report:

• BI market consolidation: What does it mean for you?


• Business intelligence: Only the tip of the iceberg


• Business intelligence + enterprise resource planning = competitive advantage


• Business intelligence in action


• Business intelligence vendors