Using PipeRider with your CI process

What is Continuous Integration (CI)

Continuous integration (CI) is the practice of automating routine tasks on a software project. For instance, after pushing code changes to your project repository, CI can trigger building, testing, and linting functions to ensure the quality of your commit. This eases the process of contributing to a project by removing the need to run these stages manually.

CI for a data project

Similar to a software project, a CI process is also useful for data projects, especially now that data projects contain code responsible for building the data pipeline and model transformations.

When used in a data project, a CI process can perform the model transformations, make modifications to databases after merging a pull request, and deploy the pipeline, among other actions.

The benefit of using PipeRider with a CI process

When integrated with a CI process, PipeRider can add a data profiling step after new models are transformed. This data profile can then be compared with the data profile of the production environment, conveniently showing how the data has changed between environments.

PipeRider can:

  • Provide full data profiling reports in the form of artifacts

  • Generate a data profile comparison summary that compares the production and modified environments

The comparison summary is generated in Markdown and is designed to be added to the pull request comment for easy review. The summary contains useful metrics about the table schema and data, such as:

  • Row count

  • Column count

  • Column type

  • New or deleted columns

  • Percentage of valid rows

  • percentage of distinct rows

The following image show an example of a PipeRider comparison summary taken from our article on detecting schema change:

In the following section we demonstrate some suggested guidelines for integrating PipeRider into your data project's CI process.

Last updated