Example

An example of adding a PipeRider report to your CI workflow

The concept of using PipeRider with your CI process is based on PR (pull request) review.

The following CI examples use dbt's Jaffle Shop project to demonstrate how PipeRider can integrate with your CI process.

GitHub - InfuseAI/jaffle_shop: A self-contained dbt project for testing purposesGitHub

Example project of how PipeRider can integrate with your CI procress

Compare the Pull Requests by PipeRider

PipeRider CI procress

To better understand the changes to the data pipeline when a pull request is created, you can follow these steps to automatically transform your model changes, profile the data source, and compare the changes between the pull request and main branches.

1. Setup PipeRider Compare Recipe file

The pipeRider compare feature provides a way to compare two runs based on a pre-defined recipe file. The default recipe file should be located at .piperider/compare/default.yml

.piperider/compare/default.yml

base:
  branch: main
  dbt:
    commands:
    - dbt deps
    - dbt build --target jaffle_shop
  piperider:
    command: piperider run --data-source jaffle_shop
target:
  dbt:
    commands:
    - dbt deps
    - dbt build --target jaffle_shop
  piperider:
    command: piperider run --data-source jaffle_shop

2. Run PipeRider compare command on the PR branch

The PipeRider compare command will be based on the pre-defined recipe file to compare your PR and main branches.

piperider compare -o /tmp/piperider/comparison --summary-file /tmp/piperider/comparison/summary.md

3. Post the comparison report to your PR comment

The compare feature outputs a Markdown-formatted comparison summary that you can attach to your pull request comment or upload to your own storage.

Option 1 - Use PipeRider Compare Action

The PipeRider Compare Action is a GitHub Action that executes compare command appends PR comments, and uploads the result to the artifacts.

# GitHub workflow snippet
name: PR with PipeRider

on: [pull_request]

jobs:
  piperider-compare:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    steps:
    - uses: actions/checkout@v3

    - name: PipeRider Compare
      uses: InfuseAI/piperider-compare-action@v1
        

Option 2 - Use your own script

Upload the Markdown file and the comparison report to your own storage using your own script, if required.

post-comparison-summary.sh /tmp/piperider/comparison/summary.md

upload-comparison-report.sh /tmp/piperider/comparison

View the example PR job on GitHub

jaffle_shop/pr-compare.yml at main · InfuseAI/jaffle_shopGitHub

Example PR Job on GitHub