Navigating Unstructured Data

When it comes to structured data (data in databases), enterprises know what they have. But unstructured data is exactly what its name implies "unstructured" --which makes it very difficult to get a handle on. In an attempt to control it, users rely on contrived structure known as taxonomies. To create taxonomies, they use products that are not exact (are error prone) and do not account for the dynamic nature of unstructured content. To add to the challenge, unstructured content is constantly changing. Users download content from the Internet (both appropriate and inappropriate content) and save it on hard drives. These users create documents and emails by the thousands and send them to other users (literally) at near light speed. Every day the unstructured data component of the risk equation grows, morphing and taking the shape of whatever is happening at that moment in the business.

Unstructured data is fraught with risk and it is changing constantly--a data management nightmare. And, as I mentioned above, traditional tools use a pre-defined taxonomy that requires very specific expertise to create. Unfortunately for users, this approach isn't practical in a world where the content in the data changes hourly.

So, on one side of the coin is the risk associated with unstructured data. On the other side of the coin there is value. Until an enterprise understands what intellectual property and other valuable unstructured data assets they have on their servers, in their SharePoint environments, and in their storage infrastructure; they cannot leverage the expertise of their own people that is locked away, hidden somewhere within their own company. The need to realize the maximum value of data makes unstructured data management tools that can handle and present enormous volumes of data in the context of a user's interests a must have in today's data environment.

I also want to touch on traditional keyword search, because the current "state of the art" in keyword search exposes the inadequacy of data analysis as most people know it today. What is required is a more comprehensive and useful view into the unstructured data--one that can grow as the enterprise data pile grows and can provide a foundation for search that makes it more effective and accurate.

Next time, I'll take a look at some technology issues that make this problem a difficult problem to solve. I'll also touch on the Digital Reef approach, including a look at our similarity engine and why it is designed the way it is. I'll talk about why unstructured data should "speak for itself" if an enterprise really wants to gain control over it.

I'll be interested to hear your thoughts.

Labels: data storage, eDiscovery, keyword search, unstructured data management