Government Agencies Adding a Petabyte of New Data in Next Two Years; Making Little Progress Yet in Big Data

May 7, 2012 No Comments »

Government data is growing and agencies are looking to leverage big data to support government mission outcomes.  However, most agencies lack the data storage/access, computational power, and personnel they need to take advantage of the big data opportunity, according to a new study by MeriTalk sponsored by NetApp.  The new report, “The Big Data Gap,” reveals that Federal IT professionals believe big data can improve government but that the promise of big data is locked away in unused or inaccessible data.

President Obama’s recently announced Big Data Research and Development Initiative highlights the big data promise – that improving our ability to extract knowledge and insights from large and complex collections of data will help government solve problems.  Federal IT professionals agree.  According to the Big Data Gap report, Federal IT professionals say improving overall agency efficiency is the top advantage of big data (59 percent) followed by improving speed/accuracy of decisions (51 percent) and the ability to forecast (30 percent).

While Federal IT professionals agree there are many benefits to big data, the technology and applications needed to successfully leverage big data are still emerging.  Sixty percent of civilian agencies and 42 percent of Department of Defense/intelligence agencies say they are just now learning about big data and how it can work for their agency.  While the promise of big data is strong, most agencies are still years away from using it.  Just 60 percent of IT professionals say their agency is analyzing the data it collects and less than half (40 percent) are using data to make strategic decisions.  On average, Federal IT professionals report that it will take their agencies three years to take full advantage of big data.

Federal IT professionals report that the amount of government data will continue to grow.  Eighty-seven percent of Federal IT professionals say their agency’s stored data has grown in the last two years.  The majority of Federal IT professionals – 96 percent – expect their agency’s stored data to grow in the next two years by an average of 64 percent.

“Government has a gold mine of data at its fingertips,” said Mark Weber, president of U.S. Public Sector for NetApp.  “The key is turning that data into high-quality information that can increase efficiencies and inform decisions.  Agencies need to look at big data solutions that can help them efficiently process, analyze, manage, and access data, enabling them to more effectively execute their missions.”

While agencies have a huge amount of data – that continues to grow – in many agencies the data is locked away.  Nearly a third of agency data is unstructured and therefore substantially less useful.  The amount of unstructured data is growing – 64 percent of Federal IT professionals report that the amount of unstructured data they store has increased in the past two years.  Data ownership further complicates agencies’ ability to use big data.  Agencies are unclear on who owns the data, with 42 percent reporting IT departments own the data, 28 percent reporting that the data belongs to the department that generates it, and 12 percent reporting the data belongs to the C-level.

Federal IT professionals also identify a gap between the big data possibility and reality with nine out of 10 reporting challenges on the path to harnessing big data.  Agencies estimate that they have just 49 percent of the data content storage/access, 46 percent of the bandwidth/computational power, and 44 percent of the personnel they need to leverage big data and drive mission results.  In addition, 57 percent say they have a least one dataset that has grown too big to work with using their current management tools and/or infrastructure.

Despite the challenges, agencies are working to harness big data.  Sixty-four percent of IT professionals say their agency’s data management system can be easily expanded/upgraded on demand.  However, they estimate that it would take an average of 10 months to double their short- to medium-term capacity.  In addition, some agencies are taking steps to improve their ability to manage and make decisions with big data.  Top tactics include investing in IT infrastructure to optimize data storage (39 percent), training IT professionals to manage/analyze big data (33 percent), and improving the security of stored data (31 percent).

“The Big Data Gap” is based on a survey of 151 Federal government CIOs and IT managers in March 2012.  The report has a margin of error of +/- 7.95 percent at a 95 percent confidence level.  To download the full study, please visit

Leave a Reply

Pin It on Pinterest

Share This