Big data doesn’t require big iron anymore.
In an IT landscape that’s seeing cloud-based and managed analytics crop up at lower prices, there’s no longer a need for sophisticated and costly on-premise hardware, said Jane Griffin, executive advisor for Deloitte Canada, in an interview at Predictive Analytics World in Toronto recently.
“It used to be expensive to buy these applications,” she said. “I’ve got to buy a server, I’ve got to buy an expensive piece of software, I’ve got to hire someone to configure or install it. That’s not necessarily the case now. There are a lot of cloud-based services. Google Analytics, for example, is extremely cheap.”
The increased willingness of both large organizations and governments to reveal their data to the world is allowing software developers to create analytics applications, many of them for mobile devices, that can cost as little as a few dollars. She cited the example of the City of San Francisco, which opened up its traffic data to the public.
It only took one “wise young person” to create a handy routing application for taxi drivers to let them know where there were high or low concentrations of cabs, she said.
“The concept of open data not only helps big businesses,” Griffin said. “It helps small businesses.”
Canada also has cloud-based vendors that tap into the vast amount of data that government agencies are putting online, she said, including most of the information in Statistics Canada.
Many Canadian municipalities have opened up their data stores in the past, with some cities, including Vancouver and Regina, creating data catalogues and even open-source development environments that let ordinary citizens create applications for their neighbours, such as garbage collection reminders.
As such, open data creates opportunities for both individual developers as well as for larger vendors that can offer SMEs more sophisticated analytics, perhaps based on a yearly subscription, Griffin said.
Meanwhile, for organizations that need dedicated on-demand analytics, there’s a “huge opportunity” for the burgeoning community of data scientists to provide “insights-as-a-service,” she added. While data scientists are in high demand in large companies and don’t come cheap, small- and medium-sized businesses could contract them on a pay-as-you-go basis, said Griffin.
“I can timeshare the data scientist—I don’t have to hire one.”
And the space between on-premise big data appliances and cloud-based analytics is also starting to narrow. One conference attendee, Jerrard Gaertner, president of the Canadian Information Processing Society (CIPS) in Ontario, said that hybrid deployments of on-premise servers linked to a managed service provider can bring down the cost of analytics considerably. Instead of an appliance stacked with in-memory, solid-state and spinning disk doing all the data processing in-house, custom-built servers can simply call home to the provider to do the heavy lifting.
The co-founder and vice-president of the Markham, Ont.-based Managed Analytics Services, Gaertner’s company targets businesses that can’t afford to pay upwards of $100,000 for the types of on-premise appliances that large enterprises buy. Instead, they’ll pay less than a tenth of that cost for the leaner commodity servers, he says.
Small- and medium-sized enterprises in Canada certainly have plenty of options to tame their big data today, but the sticking point may be the IT skills shortage that organizations like the Canadian Coalition for Tomorrow’s ICT Skills (CCICT) have been warning about.
The greatest skills shortages tend to be in emerging fields, such as data science. The prospect of “timesharing” data scientists is an attractive idea for SMEs, but the payoff will have to be equally attractive for these IT professionals, who tend to get snapped up quickly by large organizations and guarded carefully against rival companies who will try to “steal” them.
One of the current buzzwords out there is the supposed “democratization” of big data, implying equal access to analytics for all. But with the requisite talent scarce, it’s not clear whether we’re there yet. Governments are beginning to share their data, but it remains to be seen whether the combined market demand from smaller enterprises will be enough to force big corporations to give up some of their human capital.