Initial Configuration

dbt and PipeRider project configuration for CI

The concept of using PipeRider with your CI process is based on there being a PR (pull request) environment (schema and data source), in addition to the usual 'dev' and 'prod' environments.
The following CI examples use dbt (Jaffle Shop project) and Snowflake, and the configuration files for dbt and PipeRider are configured as follows.
dbt profile
The dbt profile contains three environments, dev, pr, and prod.
~/.dbt/profiles.yml
# dbt profile
jaffle_shop:
  target: dev
  outputs:
    dev:
      type: snowflake
      schema: DEV
      account: <your snowflake setup>
      user: <your snowflake setup>
      password: <your snowflake setup>
      role: <your snowflake setup>
      database: <your snowflake setup>
      warehouse: <your snowflake setup>
      threads: <your snowflake setup>
    pr:
      type: snowflake
      schema: PR
      account: <your snowflake setup>
      user: <your snowflake setup>
      password: <your snowflake setup>
      role: <your snowflake setup>
      database: <your snowflake setup>
      warehouse: <your snowflake setup>
      threads: <your snowflake setup>
    prod:
      type: snowflake
      schema: PUBLIC
      account: <your snowflake setup>
      user: <your snowflake setup>
      password: <your snowflake setup>
      role: <your snowflake setup>
      database: <your snowflake setup>
      warehouse: <your snowflake setup>
      threads: <your snowflake setup>
View this example file on GitHub:
jaffle_shop/profiles.yml at main · InfuseAI/jaffle_shop
GitHub
example dbt profiles.yml for multiple data sources/schema
As mentioned above, in addition to the production schema, there is another 'PR' schema that will be used as a staging schema to run the new model transformation in the pull request.
PipeRider project config
Corresponding to the dbt profile setup, PipeRider also needs additional data sources to link to the separate dbt profiles, dev, pr, and prod.
For initial dbt project configuration please refer to the dbt integration page. Additional data sources must be manually added to the PipeRider config:
.piperider/config.yml
# piperider config
dataSources:
- name: jaffle_shop_dev
  type: snowflake
  dbt:
    profile: jaffle_shop
    target: dev
    projectDir: .
- name: jaffle_shop_pr
  type: snowflake
  dbt:
    profile: jaffle_shop
    target: pr
    projectDir: .
- name: jaffle_shop
  type: snowflake
  dbt:
    profile: jaffle_shop
    target: prod
    projectDir: .
View this example file on GitHub:
jaffle_shop/config.yml at main · InfuseAI/jaffle_shop
GitHub
Example PipeRider config.yml for multiple data sources

Continuous Integration - Previous

Introduction

Next - Continuous Integration

Basic Example

Last modified 1mo ago