Why semantics matter within the trendy knowledge stack

Most organizations at the moment are effectively into re-platforming their enterprise knowledge stacks to cloud-first architectures. The shift in knowledge gravity to centralized cloud knowledge platforms brings huge potential. Nevertheless, many organizations are nonetheless struggling to ship worth and show true enterprise outcomes from their knowledge and analytics investments.

The time period “trendy knowledge stack” is usually used to outline the ecosystem of applied sciences surrounding cloud knowledge platforms. So far, the idea of a semantic layer hasn’t been formalized inside this stack.

When utilized appropriately, a semantic layer kinds a brand new middle of information gravity that maintains the enterprise context and semantic which means crucial for customers to create worth from enterprise knowledge property. Additional, it turns into a hub for leveraging energetic and passive metadata to optimize the analytics expertise, enhance productiveness and handle cloud prices.

What’s the semantic layer?

Wikipedia describes the semantic layer as “a enterprise illustration of knowledge that lets customers work together with knowledge property utilizing enterprise phrases equivalent to product, buyer or income to supply a unified, consolidated view of knowledge throughout the group.”

The time period was coined in an age of on-premise knowledge shops — a time when enterprise analytics infrastructure was pricey and extremely restricted in performance in comparison with immediately’s choices. Whereas the semantic layer’s origins lie within the days of OLAP, the idea is much more related immediately. 

What’s the trendy knowledge stack?

Whereas the time period “trendy knowledge stack” is often used, there are a lot of representations of what it means. In my view, Matt Bornstein, Jennifer Li and Martin Casado from Andreessen Horowitz (A16Z) provide the cleanest view in Rising Architectures for Fashionable Information Infrastructure.

I’ll consult with this simplified diagram based mostly on their work under:

This illustration tracks the stream of knowledge from left to proper. Uncooked knowledge from varied sources transfer by means of ingestion and transport providers into core knowledge platforms that handle storage, question and processing and transformation previous to being consumed by customers in a wide range of evaluation and output modalities. Along with storage, knowledge platforms provide SQL question engines and entry to Synthetic Intelligence (AI) and machine studying (ML) utilities. A set of shared providers cuts throughout all the knowledge processing stream on the backside of the diagram. 

The place is the semantic layer?

A semantic layer is implicit any time people work together with knowledge: It arises organically except there’s an intentional technique carried out by knowledge groups. Traditionally, semantic layers had been carried out inside evaluation instruments (BI platforms) or inside an information warehouse. Each approaches have limitations.

BI-tool semantic layers are use case particular; a number of semantic layers are likely to come up throughout totally different use instances resulting in inconsistency and semantic confusion. Information warehouse-based approaches are typically overly inflexible and too complicated for enterprise customers to work with straight; work teams will find yourself extracting knowledge to native analytics environments — once more resulting in a number of disconnected semantic layers. 

I exploit the time period “common semantic layer” to explain a skinny, logical layer sitting between the information platform and evaluation and output providers that summary the complexity of uncooked knowledge property in order that customers can work with business-oriented metrics and evaluation frameworks inside their most popular analytics instruments.

The problem is methods to assemble the minimal viable set of capabilities that provides knowledge groups enough management and governance whereas delivering end-users extra advantages than they may get by extracting knowledge into localized instruments.

Implementing the semantic layer utilizing transformation providers

The set of transformation providers within the A16Z knowledge stack contains metrics layer, knowledge modeling, workflow administration and entitlements and safety providers. When carried out, coordinated and orchestrated correctly, these providers type a common semantic layer that delivers vital capabilities, together with:

• Making a single supply of fact for enterprise metrics and hierarchical dimensions, accessible from any analytics device.
• Offering the agility to simply replace or outline new metrics, design domain-specific views of knowledge and incorporate new uncooked knowledge property.
• Optimize analytics efficiency whereas monitoring and optimizing cloud useful resource consumption.
• Implement governance insurance policies round entry management, definitions, efficiency and useful resource consumption.

Let’s step by means of every transformation service with a watch towards how they need to work together to function an efficient semantic layer.

Information modeling

Information modeling is the creation of business-oriented, logical knowledge fashions which are straight mapped to the bodily knowledge buildings within the warehouse or lakehouse. Information modelers or analytics engineers give attention to three vital modeling actions:

Making knowledge analytics-ready: Simplifying uncooked, normalized knowledge into clear, principally de-normalized knowledge that’s simpler to work with.

Definition of study dimensions: Implementing standardized definitions of hierarchical dimensions which are utilized in enterprise evaluation — that’s, how a corporation maps months to fiscal quarters to fiscal years. 

Metrics design: Logical definition of key enterprise metrics utilized in analytics merchandise. Metrics will be easy definitions (how the enterprise defines income or ship amount). They are often calculations, like gross margin ([revenue-cost]/income). Or they are often time-relative (quarter-on-quarter change).

I wish to consult with the output of semantic layer-related knowledge modeling as a semantic mannequin. 

The metrics layer 

The metrics layer is the one supply of metrics fact for all analytics use instances. Its main operate is sustaining a metrics retailer that may be accessed from the total vary of analytics customers and analytics instruments (BI platforms, functions, reverse ETL, and knowledge science instruments).  

The time period “headless BI” describes a metrics layer service that helps consumer queries from a wide range of BI instruments. That is the basic functionality for semantic layer success — if customers are unable to work together with a semantic layer straight utilizing their most popular analytics instruments, they are going to find yourself extracting knowledge into their device utilizing SQL and recreating a localized semantic layer.

Moreover, metrics layers have to help 4 vital providers:

Metrics curation: Metrics stewards will transfer between knowledge modeling and the metrics layer to curate the set of metrics offered for various analytics use instances.

Metrics change administration: The metrics layer serves as an abstraction layer that shields the complexity of uncooked knowledge from knowledge customers. As a metrics definition adjustments, current experiences or dashboards are preserved. 

Metrics discoverability: Information product creators want to simply discover and implement the correct metrics for his or her goal. This turns into extra vital because the record of curated metrics grows to incorporate a broader set of calculated or time-relative metrics.  

Metrics serving: Metrics layers are queried straight from analytics and output instruments. As finish customers request metrics from a dashboard, the metrics layer must serve the request quick sufficient to offer a constructive analytics consumer expertise.

Workflow administration

Transformation of uncooked knowledge into an analytics-ready state will be based mostly on bodily materialized transforms, digital views based mostly on SQL or some mixture of these. Workflow administration is the orchestration and automation of bodily and logical transforms that help the semantic layer operate and straight affect the fee and efficiency of analytics. 

Efficiency:  Analytics customers have a really low tolerance for question latency. A semantic layer can not introduce a question efficiency penalty; in any other case, intelligent finish customers will once more go down the information extract route and create various semantic layers. Efficient efficiency administration workflows automate and orchestrate bodily materializations (creation of combination tables) in addition to resolve what and when to materialize. This performance must be dynamic and adaptive based mostly on consumer question conduct, question runtimes and different energetic metadata. 

Value: The first price tradeoff for efficiency is said to cloud useful resource consumption. Bodily transformations executed within the knowledge platform (ELT transforms) devour compute cycles and value cash. Finish consumer queries do the identical. The selections made on what to materialize and what to virtualize straight affect cloud prices for analytics packages. 

Analytics performance-cost tradeoff turns into an attention-grabbing optimization downside that must be managed for every knowledge product and use case. That is the job of workflow administration providers.

Entitlements and safety

Transformation-related entitlements and safety providers relate to the energetic utility of knowledge governance insurance policies to analytics. Past cataloging knowledge governance insurance policies, the fashionable knowledge stack should implement insurance policies at question time, as metrics are accessed by totally different customers. Many various kinds of entitlements could also be managed and enforced alongside (or embedded in) a semantic layer.

Entry management: Correct entry management providers guarantee all customers can get entry to all the knowledge they’re entitled to see.  

Mannequin and metrics consistency:  Sustaining semantic layer integrity requires some stage of centralized governance of how metrics are outlined, shared and used. 

Efficiency and useful resource consumption: As mentioned above, there are fixed tradeoffs being made on efficiency and useful resource consumption. Consumer entitlements and use case precedence might also issue into the optimization.

Actual time enforcement of governance insurance policies is vital for sustaining semantic layer integrity.

Integrating the semantic layer inside the trendy knowledge stack

Layers within the trendy knowledge stack should seamlessly combine with different surrounding layers. The semantic layer requires deep integration with its knowledge material neighbors — most significantly, the question and processing providers within the knowledge platform and evaluation and output instruments. 

Information platform integration

A common semantic layer mustn’t persist knowledge outdoors of the information platform. A coordinated set of semantic layer providers must combine with the information platform in just a few vital methods:

Question engine orchestration: The semantic layer dynamically interprets incoming queries from customers (utilizing the metrics layer logical constructs) to platform-specific SQL (rewritten to replicate the logical to bodily mapping outlined within the semantic mannequin). 

Remodel orchestration: Managing efficiency and value requires the aptitude to materialize sure views into bodily tables. This implies the semantic layer should be capable of orchestrate transformations within the knowledge platform. 

AI/ML integration: Whereas many knowledge science actions leverage specialised instruments and providers accessing uncooked knowledge property straight, a formalized semantic layer creates the chance to offer enterprise vetted options from the metrics layer to knowledge scientists and AI/ML pipelines. 

Tight knowledge platform integration ensures that the semantic layer stays skinny and might function with out persisting knowledge domestically or in a separate cluster.

Evaluation and output

A profitable semantic layer, together with a headless BI method to implementing the metrics layer, should be capable of help a wide range of inbound question protocols — together with SQL (Tableau), MDX (Microsoft Excel), DAX (Microsoft Energy BI), Python (knowledge science instruments), and RESTful interfaces (for utility builders) — utilizing customary protocols equivalent to ODBC, JDBC, HTTP(s) and XMLA.

Augmented analytics

Main organizations incorporate knowledge science and enterprise AI into on a regular basis decision-making within the type of augmented analytics. A semantic layer will be useful in efficiently implementing augmented analytics. For instance:

• Semantic layers can help pure language question initiatives. “Alexa, what was our gross sales income final quarter?” will solely return the proper outcomes if Alexa has a transparent understanding of what income and time imply. 
• Semantic layers can be utilized to publish AI/ML-generated insights (predictions and forecasts) to enterprise customers utilizing the identical analytics instruments they use to investigate historic knowledge. 
• Past simply prediction values, semantic layers could make broader inference knowledge obtainable to enterprise customers in a method that may improve explainability and belief in enterprise AI.

The middle of mass for information gravity within the trendy knowledge stack

The A16Z mannequin implies that organizations may assemble a cloth of home-grown or single-purpose vendor choices to construct a semantic layer. Whereas actually potential, success will probably be decided by how well-integrated particular person providers are. As famous, even when a single service or integration fails to ship on consumer wants, localized semantic layers are inevitable.

Moreover, it is very important contemplate how very important enterprise information will get sprinkled throughout knowledge materials within the type of metadata. The semantic layer has the benefit of seeing a big portion of energetic and passive metadata created for analytics use instances. This creates a possibility for forward-thinking organizations to raised handle this information gravity and higher leverage metadata for enhancing the analytics expertise and driving incremental enterprise worth.

Whereas the semantic layer remains to be rising as a expertise class, it is going to clearly play an vital position within the evolution of the fashionable knowledge stack.

This text is a abstract of my present analysis round semantic layers inside the trendy, cloud-first knowledge stack. I’ll be presenting my full findings on the upcoming digital Semantic Layer Summit on April 26, 2023. 

David P. Mariani is CTO and cofounder of AtScale, Inc.