Infrastructure and skills restrictions hindering 'big data' use at US state and local authorities

Just 2% of 150 state and local government chief information officers (CIOs) and IT managers surveyed by MeriTalk said the organisation they worked for had a "complete big data strategy", whilst fewer than half (47%) said they were using the data they have collected to inform strategic decision making. Many identified practical problems that are currently preventing them from utilising big data. 

"Agencies estimate they have just 46% of the data storage/access; 42% of the computational power; and 35%of the personnel they need to leverage big data and drive mission results," MeriTalk's survey report  (registration required to view 17-page / 1.26MB PDF) said. "79% of agencies say it will be at least three years before they are able to take full advantage of big data." 

MeriTalk reported that the problems in exploiting big data exist despite there being a "steady incline" in the amount of data the state and local government agencies are collecting. 

"87% of state and local IT professionals say their stored data has grown in the last two years and one in three have a data set that has grown too big to analyze," the report said. "97% expect their data to grow in the next two years (by an average of 53%)." 

However, 44% of respondents said that big data was not even being discussed within their organisation, whilst 39% said that they are only now "learning about the topic and how it could work for their agency". 

The biggest advantage cited to successfully managing big data was that it could be used to improve overall efficiency within an organisation (57%). Improving the speed and accuracy of decisions (54%) and obtaining a greater understanding of citizen needs and how to meet them (37%) were the next most cited advantages, MeriTalk said. 

EU privacy body the Article 29 Working Party recently described the term 'big data' as referring to organisations' extensive analysis of "gigantic digital datasets" using computer algorithms with a view to better understanding and exploiting data for the purpose of making "better and more informed decisions".