Self-Service Access, Cloud Drive Big Data Value- Valutrics

Data governance has emerged as a top concern, and business intelligence remains the top workload for big data technology Hadoop, according to the third edition of the Big Data Maturity Survey.

Business intelligence is still the biggest — and growing — workload planned for big data. And organizations are also adding more self-service access to big data and putting more big data in the cloud, all positive trends for getting value from investments in this technology.

That’s according to the most recent edition of the Big Data Maturity Survey, conducted by BI-on-Hadoop vendor AtScale, together with Tableau, Trifacta, Cognizant, and the three major Hadoop distributors Cloudera, Hortonworks, and MapR. The survey is based on 2,550 responses from big data pros at 1,400 companies in 77 countries.

But there’s a bit of a disconnect when it comes to the tangible value achieved in big data projects, with the likelihood of value depending on who is driving their adoption. While IT organizations drove 57% of big data projects, business users drove 33%, and an executive mandate drove 11%, projects that had the executive mandate as a driver were most likely to achieve tangible value — 56%.

When IT was the driver 47% achieved tangible value, according to the survey results. For business user-driven big data projects, 52% achieved tangible value.

BI Continues To Grow

The survey shows a full 75% of respondents saying they plan to use big data for BI, up from the 69% who said the same thing last year. The results mark the continued momentum of BI workloads on big data platforms such as Hadoop. Last year’s survey found that BI had overtaken ETL and data science as the top workload planned for Hadoop and related big data platforms.

This year’s survey shows growth in cloud deployments for big data, too, with a full 53% saying they are using cloud for their big data deployment now, and 72% saying they plan on doing big data in the cloud in the future.

What’s more, while 27% of respondents say they have more than half of their big data in the cloud today, 42% say they are planning to have more than half of their big data operations in the cloud in the future.

“There’s been a clear surge in use of big data in the cloud over the last year, and what’s perhaps as interesting is the fact that respondents are far more likely to achieve tangible value when their data is in the cloud,” said AtScale co-founder and CTO Matt Baird in a prepared statement.

Baird is citing another finding of the survey there. The survey found that respondents who have deployed all of their big data in the cloud are 34% more likely to achieve tangible value than those who have none of their big data in the cloud.

More companies are also providing users with self-service access to big data, according to this year’s survey. A full 47% of companies this year offered users self-service access, marking a 15% increase from last year. And those who offered self-service access this year were more likely to have achieved tangible value from their big data investments, 57%, compared to 43% at the companies that didn’t offer self-service access.

Hadoop And Spark

More organizations are using Hadoop now than before — 73% compared to last year’s 65%.

But Apache Spark, while it may have a very vocal group of enthusiasts among technologists, is largely still in the experimental phase today in organizations surveyed.

While the majority of organizations — 60% — are still only using Apache Spark for educational purposes, organizations that have deployed Spark in production are 85% more likely to achieve value, according to the report.

Big Data’s Role In Enterprise

The survey also asked about big data’s role in enterprise data warehouses, and only 18% of respondents viewed big data as a replacement for the enterprise data warehouse. The rest, 82%, are looking to big data to augment and optimize the existing data warehouse rather than replacing it. That future data warehouse will combine traditional data warehousing solutions such as Teradata, with newer data platforms such as Hadoop, and with big data services such as Google BigQuery, Amazon Redshift, and Microsoft Azure, according to AtScale’s report.

But organizations still expressed some concerns about big data going forward. The top three concerns cited by respondents were accessibility, security and governance, with worries related to governance growing the most at 21%, according to the report.