From Warehouse To Lake, Then To Lakehouse – Data Architecture Has Evolved. Lakehouse Adopters Can Gain A Competitive Edge In The Wealth Management Space.
For many organizations, data warehouses represent a time-tested, traditional solution for reporting and analytics. So why would a wealth management firm want to leave its warehouse behind?
Although the strength of a data warehouse lies in its ability to perform efficient dimensional analysis and reporting, this same feature may also be the primary reason to consider moving on from them. Modern firms have needs for data that aren’t well served by warehouses. For example, they often have fixed dimensional schemas that limit the types of analyses that can be performed. This can be problematic if an organization wants to utilize more of its data, like when building a comprehensive profile of advisory clients.
Data warehouses are typically designed to serve business consumers, rather than providing the personalized advice that advisors and their clients may require. While warehouses may be an effective repository for data, they are not designed to serve as an integration hub for other lines of business within an organization. A data “lakehouse” architecture – which represents a hybrid, “best-of-both-worlds” approach to data management – is poised to address most of these shortcomings.
The Four Stages Of Data Architecture
Let’s examine the four stages of data architecture found in most firms.
Stage Zero. Sadly, many firms are stuck in stage zero. They rely on third parties to own their data and often have no direct way of accessing it other than via that third party. This approach can make it impossible for them to normalize and combine that data in meaningful ways, except by means of arduous spreadsheet work.
Data Warehouse. Most firms have a data warehouse for some, if not most, of their data. For traditional business and reporting needs, it’s not a bad solution, which is why it’s commonplace. As firms expand, they often face issues related to their data warehousing systems. Many warehouses lack the capability to provide intraday reporting, primarily due to inefficiencies in extracting data from transactional systems and the costs associated with normalizing and transforming the data within the warehouse.
Additionally, the dimensional models used by their warehouses may become inflexible and difficult to modify to meet evolving business needs. As data volumes increase, the cost of using a warehouse can become prohibitively high due to the coupling of storage and computing resources, as well as competition between analytics users and normalization/transformation jobs.
Data Lake. Mature firms can address some of these problems by building a data lake to feed their data warehouse. A data lake can solve many of the shortcomings of a vanilla data warehouse architecture. By removing the extraction, normalization and transformation processes from the warehouse, the system can focus on its core strength of dimensional reporting and analysis.
Processing in a data lake is often faster and more cost-effective than doing so within a warehouse, as there are numerous open-source, massive parallel processing tools available. Data lakes are not limited by traditional dimensional schemas, which allows for greater flexibility and more innovative approaches to generating insights.
One common pitfall of data lakes is that often data is inserted into the lake without taking care to organize it or provide an inventory of all of that data. This can often lead to the proverbial data swamp – large amounts of data stored in the data lake that become nearly useless to the firm.
Data Lakehouse. Why are savvy firms accelerating toward data lakehouses? While a data lake improves the ability to process data more quickly, efficiently and cheaply, a data lake is still limited by the structured processing of a warehouse. On the other hand, data lakehouses eliminate the movement of the data into the warehouse, providing numerous advantages over a data lake architecture.
Data lakehouses offer several advantages over both traditional data warehouses and simple data lake architectures. One key benefit is their ability to process and update business aggregates in near real-time, enabling intraday reporting and analytics. Since data is updated throughout the day, a lakehouse can serve as an effective integration hub for other lines of business within the firm, while offering flexibility when it comes to data modeling.
They are not constrained to traditional dimensional modeling, allowing data to be aggregated for specific use cases, making consumption even more efficient. Finally, they’re effective at facilitating the construction of complex data models such as graphs, which can provide advisors with valuable insights about their clients.
Why Are Firms Stuck In The Warehouse?
There can be many reasons that a firm is stuck with a warehouse or even a stage zero architecture. Smaller firms may lack the necessary resources to move beyond a basic data warehouse architecture, while other firms may lack the expertise or awareness of newer, more modern approaches.
Additionally, many firms may not fully understand the business value that these architectures can provide, such as improved advice, client satisfaction and brand differentiation. Therefore, senior management may not prioritize investing in newer data architecture approaches.
Ultimately, firms will need to realize the value of their data to their business to invest in a more modern data architecture.
The Demand For Lakehouses
The demand for lakehouses will continue to grow over time. Firms that have mature data teams and scale are increasingly implementing data lakehouse methods for their advisors and professionals, establishing a competitive advantage over other firms. Although smaller firms may not yet be at that level, these competitive pressures will likely push them in that direction in the near future.
Once firms view their data architecture as a differentiator, the spark will be ignited and they’ll invest in a data lakehouse. The call to action is clear – forward-thinking firms that better leverage their data will have a distinct competitive advantage over their peers.
Mark Eaton is an Executive in Residence at wealth management consulting services provider F2 Strategy.
1 comment
Comments are closed.