Big Government Meets Big Data

Government agencies across North America are not only burdened by more requests for services, they are struggling to dig themselves out from under an avalanche of data. A new study has concluded that IT departments in American Federal agencies are grappling with same issues as their peers in the private sector: now that they have all this new data, how should they best make use of it, and where are they going to get the resources to do so?

Mark Weber, President of NetApp, the company that conducted the survey of 151 Federal CIOs and IT managers, said in an interview with Federal News Radio that the issue is largely one of resources and technology. “Big Data is just datasets that have gotten so large and complex that people don’t have the tools or the ability to capture it, store it, search it, retrieve it, analyze it. They just don’t have the proper equipment or technology.”

Some interesting numbers from the study:

• 64% of Federal IT professionals say the “amount of unstructured data they store has increased in the past two years.”
• 57% say their department works with at least one dataset that has “grown too big to work with using their current management tools and/or infrastructure.”
• Respondents are unclear about who owns the data that is being collected, with 42% saying IT departments own it, 28% reporting that “the data belongs to the department that generates it,” and 12% reporting the data is the property of the C-level.

Part of the problem is that much of this new data is unstructured and received in untraditional forms (video, images, etc.), making it harder for analysts to properly break it down into workable datasets. “How you search and mine that kind of information is really pretty complicated,” Weber said in the interview. “That doesn’t fit nicely into a relational database. That’s the piece of data that is exploding exponentially is the unstructured side.”

American government agencies certainly recognize the potential cost-savings and efficiency gains of this new data. The key to realizing those potentials, Weber believes, is innovation and proper resource allocation.