“We’re going to cease fixing information within the BI layer beginning in the present day!”
I’ve heard some model of this chorus numerous instances in my profession, and the final decade introduced a ton of improvements that at instances made it appear doable. Improvements together with the vast adoption of cloud computing, cloud warehouse and lakehouse architectures, and the shift from ETL to ELT. At each step of the method, well-meaning information groups swore off the apply of fixing information issues in BI as a result of apparent points it creates: Metrics locked in BI, inconsistent and low-quality information, and a excessive quantity of rework.
Did anybody succeed although?
Nearly universally, no. Enterprise actuality adjustments too quick for even essentially the most adept information staff. Information fashions begin shedding relevance the second they’re deployed. Finish customers run quicker than your governance program can sustain with. BI-layer information transformation, modeling, and calculation is an inevitable and inescapable a part of information, no matter our makes an attempt to forestall it.
The First Step
We took on this problem at GoodData by introducing one of many trade’s first and strongest open enterprise intelligence semantic layers. Not are these essential ‘final mile’ transformations or calculations locked right into a proprietary entrance finish. Due to our SDKs, APIs, and SQL entry level, any system can profit from the superb work of your information analysts and BI engineers. It was an essential first step, however our clients nonetheless struggled with complicated information construction, velocity, and price points that semantics alone merely couldn’t remedy.
The Subsequent Stage of our Imaginative and prescient
It’s for that motive, we’re excited to announce FlexQuery, the following stage of our imaginative and prescient to offer highly effective, open instruments to information builders fixing ‘final mile’ information issues that merely can’t be addressed in information warehouses. FlexQuery is a composable information service layer constructed on common open-source applied sciences like Apache Arrow, Apache Iceberg, and DuckDB. We’ve pre-built essentially the most helpful enterprise intelligence providers into it, resembling caching, information federation, transformation, and computation. Nonetheless, in line with our dedication to developer-first approaches, FlexQuery is open so that you can construct modules that meet your particular wants.
So what are you able to truly do with FlexQuery?
Lower your expenses with superior caching
FlexQuery robotically caches full and partial outcome units for re-use, saving you vital cash in your underlying cloud information structure. Based mostly on Apache Arrow, the ‘FlexCache’ helps super-fast in-memory computation on giant volumes of information. Transferring ahead, we plan to offer increasingly more granular cache controls that will help you stability information recency vs. price financial savings.
Remodel and federate information with agility
FlexQuery’s built-in transformation and federation capabilities make it simple to mix information from all your purposes through SQL or APIs. You possibly can materialize information in reminiscence or on disk question it dwell out of your databases, and even combine the 2 modes collectively. It’s simple to do information engineering straight inside FlexQuery to create the complicated pipelines essential to prep information for consumption.
Construct with one of the best open-source instruments
FlexQuery is constructed on one of the best open-source analytics instruments, together with Apache Arrow, Iceberg, and DuckDB, pre-configured for core BI use instances. Nonetheless, at GoodData we all the time provide the keys to the dominion: It’s simple to develop your personal FlexQuery modules to deal with information engineering or information science wants at low price straight in your BI platform.
Welcome to the Analytics Lake
FlexQuery is now on the computational coronary heart of what we name the ‘Analytics Lake.’ In contrast to an information lake, the Analytics Lake incorporates information, metadata, information fashions, metrics, and reviews/visualizations in a single platform, making it simple to create and govern any type of analytics software with out complicated software program integration challenges. Furthermore, due to our semantic layer, every person accesses the Analytics Lake by their favourite interface — Python for information scientists, React for software builders, SQL for information engineers and, no code/low code and chat interfaces for enterprise customers.
FlexQuery and the Analytics Lake idea characterize the following step ahead for enterprise intelligence platforms, shifting past easy point-and-click desktop interfaces to embrace the total complexity of information improvement within the trendy period. With a mixture of intuitive person interfaces, highly effective analytics-as-code instruments, and now the FlexQuery analytics engine, GoodData provides organizations giant and small the world’s strongest, versatile, and environment friendly enterprise intelligence platform.
Prepared To Study Extra?
To study extra about FlexQuery and the Analytics Lake head right here, or to discover it hands-on together with the remainder of the GoodData analytics platform, request a demo.
