TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

How to Avoid the Hazards of Big Data Projects

A new TDWI report looks at six things your enterprise can do to avoid pitfalls and maximize the benefits of big data integration and analytics projects.

• By James E. Powell
• October 17, 2018

Big, bigger, biggest. Big data is getting bigger and can be one of the biggest projects an enterprise takes on. With so much at stake, how can your enterprise reap the biggest benefits from big data while avoiding common project pitfalls?

For Further Reading: 5 Steps to Securing Your Big Data Infrastructure Data Quality Evolution with Big Data and Machine Learning 5 Use Cases for Integrating Big Data Tools with a Data Warehouse

In the recent TDWI Checklist Report: Six Keys to Maximizing Big Data Benefits and Project Success, David Stodder, senior director of TDWI Research for business intelligence, explores how an enterprise can deliver business value and avoid common pitfalls in big data integration and analytics projects.

Big data is about more than just the increasing volume of data your enterprise collects and processes. "Big data projects focus on enabling analysis of and interaction with new types and combinations of data at a far greater scale than has been possible with traditional enterprise business intelligence (BI) and data warehousing systems," Stodder explains. Established and new organizations alike need to invest in advanced analytics and cutting-edge information-driven applications that use artificial intelligence techniques such as machine learning.

Some organizations are not simply augmenting existing systems to handle big data. Instead, they are replacing their current systems with big data lakes built with Hadoop and Spark ecosystem technologies and/or cloud-based data platforms, Stodder notes.

How do you avoid pitfalls and hazards so you can exploit big data to achieve greater customer insights, improve operations, and meet your other critical goals? The TDWI report discusses six issues to address so you can start big data projects off right and then manage them to achieve objectives faster and more easily.

Stating (and Meeting) Clear Goals

As with many projects, you can reduce time to value if you know where you're headed. Stodder recommends your team first clarify your objectives in order to deliver benefits sooner.

Stodder says that TDWI research points to two common obstacles to reducing time to value: poor project definition and scoping and a shortage of skilled personnel. According to his recommendations, your team should carefully define the business case and articulate how the project will contribute to a key objective, such as increasing sales and market share, reducing costs, or generating new products and services for partners or customers.

Once you know where you're going, it's important to deliver incremental benefits. Traditional BI and data warehousing projects often won't deliver results for months (or years), by which time the goals have probably shifted. By using agile methods that support strong teamwork between business leaders, analysts, data scientists, developers, and data engineers, organizations can take advantage of shorter, incremental development cycles. Users can work with and provide feedback based on intermediate deliverables and adjust the project accordingly.

Instead of starting with an empty canvas, consider using blueprints and templates when you develop a project plan, many of which are publicly available at no cost. These outlines prevent you from investing in technology before you have a solid plan and help you deal with the complexities and details of a big data project. "Templates and blueprints also support reuse of proven processes and routines, which can save organizations time and money by avoiding reinventing wheels unnecessarily," Stodder explains. Typically they can be customized to accommodate your particular needs and goals.

In the report, Stodder also explores how you can optimize technology platforms for data ingestion and transformation, unlock the potential value of your big data platform, and take advantage of centralizing diverse data to support innovative applications. Echoing a common theme from TDWI researchers, he recommends that your enterprise make data governance and stewardship priorities, not afterthoughts.

You can read the full report here. Visitors new to TDWI must complete a short, one-time registration for access.