Tuesday, April 5, 2022
HomeBig DataConstruct information lineage for information lakes utilizing AWS Glue, Amazon Neptune, and...

Construct information lineage for information lakes utilizing AWS Glue, Amazon Neptune, and Spline

Information lineage is without doubt one of the most important elements of a knowledge governance technique for information lakes. Information lineage helps be sure that correct, full and reliable information is getting used to drive enterprise choices. Whereas a knowledge catalog gives metadata administration options and search capabilities, information lineage exhibits the complete context of your information by capturing in higher element the true relationships between information sources, the place the information originated from and the way it will get remodeled and converged. Totally different personas within the information lake profit from information lineage:

  • For information scientists, the power to view and observe information circulation because it strikes from supply to vacation spot helps you simply perceive the standard and origin of a specific metric or dataset
  • Information platform engineers can get extra insights into the information pipelines and the interdependencies between datasets
  • Adjustments in information pipelines are simpler to use and validate as a result of engineers can determine a job’s upstream dependencies and downstream utilization to correctly consider service impacts

Because the complexity of knowledge panorama grows, clients are dealing with important manageability challenges in capturing lineage in an economical and constant method. On this put up, we stroll you thru three steps in constructing an end-to-end automated information lineage answer for information lakes: lineage capturing, modeling and storage and eventually visualization.

On this answer, we seize each coarse-grained and fine-grained information lineage. Coarse-grained information lineage, which regularly targets enterprise customers, focuses on capturing the high-level enterprise processes and general information workflows. Usually, it captures and visualizes the relationships between datasets and the way they’re propagated throughout storage tiers, together with extract, rework and cargo (ETL) jobs and operational data. High quality-grained information lineage offers entry to column-level lineage and the information transformation steps within the processing and analytical pipelines.

Answer overview

Apache Spark is without doubt one of the hottest engines for large-scale information processing in information lakes. Our answer makes use of the Spline agent to seize runtime lineage data from Spark jobs, powered by AWS Glue. We use Amazon Neptune, a purpose-built graph database optimized for storing and querying extremely linked datasets, to mannequin lineage information for evaluation and visualization.

The next diagram illustrates the answer structure. We use AWS Glue Spark ETL jobs to carry out information ingestion, transformation, and cargo. The Spline agent is configured in every AWS Glue job to seize lineage and run metrics, and sends such information to a lineage REST API. This backend consists of producer and shopper endpoints, powered by Amazon API Gateway and AWS Lambda features. The producer endpoints course of the incoming lineage objects earlier than storing them within the Neptune database. We use shopper endpoints to extract particular lineage graphs for various visualizations within the frontend software. We carry out advert hoc interactive evaluation on the graph by means of Neptune notebooks.

We offer pattern code and Terraform deployment scripts on GitHub to shortly deploy this answer to the AWS Cloud.

Information lineage capturing

The Spline agent is an open-source undertaking that may harvest information lineage robotically from Spark jobs at runtime, with out the necessity to modify the prevailing ETL code. It listens to Spark’s question run occasions, extracts lineage objects from the job run plans and sends them to a preconfigured backend (resembling HTTP endpoints). The agent additionally robotically collects job run metrics such because the variety of output rows. As of this writing, the Spline agent works solely with Spark SQL (DataSet/DataFrame APIs) and never with RDDs/DynamicFrames.

The next screenshot exhibits find out how to combine the Spline agent with AWS Glue Spark jobs. The Spline agent is an uber JAR that must be added to the Java classpath. The next configurations are required to arrange the Spline agent:

  • spark.sql.queryExecutionListeners configuration is used to register a Spline listener throughout its initialization.
  • spark.spline.producer.url specifies the tackle of the HTTP server that the Spline agent ought to ship lineage information to.

Spline Agent Configuration on AWS Glue

We construct a knowledge lineage API that’s suitable with the Spline agent. This API facilitates the insertion of lineage information to Neptune database and graph extraction for visualization. The Spline agent requires three HTTP endpoints:

  • /standing – For well being checks
  • /execution-plans – For sending the captured Spark execution plans after the roles are submitted to run
  • /execution-events – For sending the job’s run metrics when the job is full

We additionally create extra endpoints to handle numerous metadata of the information lake, such because the names of the storage layers and dataset classification.

When a Spark SQL assertion is run or a DataFrame motion is named, Spark’s optimization engine, specifically Catalyst, generates completely different question plans: a logical plan, optimized logical plan and bodily plan, which will be inspected utilizing the EXPLAIN assertion. In a job run, the Spline agent parses the analyzed logical plan to assemble a JSON lineage object. The article consists of the next:

  • A singular job run ID
  • A reference schema (attribute names and information sorts)
  • A listing of operations
  • Different system metadata resembling Spark model and Spline agent model

A run plan specifies the steps the Spark job performs, from studying information sources, making use of completely different transformations, to lastly persisting the job’s output right into a storage location.

To sum up, the Spline agent captures not solely the metadata of the job (resembling job title and run date and time), the enter and output tables (resembling information format, bodily location and schema) but in addition detailed details about the enterprise logic (SQL-like operations that the job performs, resembling be a part of, filter, undertaking and mixture).

Information modeling and storage

Information modeling begins with the enterprise necessities and use instances and maps these wants right into a construction for storing and organizing our information. In information lineage for information lakes, the relationships between information property (jobs, tables and columns) are as essential because the metadata of these. Because of this, graph databases are appropriate to mannequin such extremely linked entities, making it environment friendly to know the advanced and deep community of relationships inside the information.

Neptune is a quick, dependable, absolutely managed graph database service that makes it straightforward to construct and run functions with extremely linked datasets. You should utilize Neptune to create refined, interactive graph functions that may question billions of relationships in milliseconds. Neptune helps three fashionable graph question languages: Apache TinkerPop Gremlin and openCypher for property graphs and SPARQL for W3C’s RDF information mannequin. On this answer, we use the property graph’s primitives (together with vertices, edges, labels and properties) to mannequin the objects and use the gremlinpython library to work together with the graphs.

The target of our information mannequin is to offer an abstraction for information property and their relationships inside the information lake. Within the producer Lambda features, we first parse the JSON lineage objects to kind logical entities resembling jobs, tables and operations earlier than establishing the ultimate graph in Neptune.

Lineage Processing

The next diagram exhibits a pattern information mannequin used on this answer.

Lineage Data Model

This information mannequin permits us to simply traverse the graph to extract coarse-grained and fine-grained information lineage, as talked about earlier.

Information lineage visualization

You possibly can extract particular views of the lineage graph from Neptune utilizing the buyer endpoints backed by Lambda features. Hierarchical views of lineage at completely different ranges make it straightforward for the end-user to investigate the data.

The next screenshot exhibits a knowledge lineage view throughout all jobs and tables.

Lineage Visualization Overall

The next screenshot exhibits a view of a selected job plan.

Lineage Visualization at Job Level

The next screenshot exhibits an in depth look into the operations taken by the job.

Lineage Visualization at Execution Level

The graphs are visualized utilizing the vis.js community open-source undertaking. You possibly can work together with the graph parts to be taught extra in regards to the entity’s properties, resembling information schema.


On this put up, we confirmed you architectural design choices to robotically gather end-to-end information lineage for AWS Glue Spark ETL jobs throughout a knowledge lake in a multi-account AWS surroundings utilizing Neptune and the Spline agent. This method permits searchable metadata, helps to attract insights and obtain an improved organization-wide information lineage posture. The proposed answer makes use of AWS managed and serverless companies, that are scalable and configurable for prime availability and efficiency.

For extra details about this answer, see Github. You might modify the code to increase the information mannequin and APIs.

In regards to the Authors

Khoa Nguyen is a Senior Large Information Architect at Amazon Internet Providers. He works with massive enterprise clients and AWS companions to speed up clients’ enterprise outcomes by offering experience in Large Information and AWS companies.

Krithivasan Balasubramaniyan is a Principal Advisor at Amazon Internet Providers. He permits international enterprise clients of their digital transformation journey and helps architect cloud native options.

Rahul Shaurya is a Senior Large Information Architect at Amazon Internet Providers. He helps and works intently with clients constructing information platforms and analytical functions on AWS.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments