Quick Start
Integrate PipeRider with your dbt project in 5 mins
Navigate to your dbt folder, and install PipeRider.
pip install 'piperider[<connector>]'
PipeRider supports the following data connectors
connectors | install |
---|---|
snowflake | pip install 'piperider[snowflake]' |
postgres | pip install 'piperider[postgres]' |
bigquery | pip install 'piperider[bigquery]' |
redshift | pip install 'piperider[redshift]' |
parquet | pip install 'piperider[parquet]' |
csv | pip install 'piperider[csv]' |
duckdb | pip install 'piperider[duckdb]' |
PipeRider requires python 3.7+
Go to your dbt project, and initalize PipeRider.
piperider init
The
init
command creates a .piperider
directory inside the current directory. This is where all of the piperider project files will be stored, including data source configuration, data quality assertions, data profiling information, and generated report files.After initialization, you can verify the configuration by running
piperider diagnose
. It will use the dbt profile file profiles.yml
to connect to the data warehouse.Collect profiling statistics by using
piperider run
The
run
command will generate profiling statistics for your table models, such as row_count
, non_nulls
, min
, max
, distinct
, quantiles, topk
and more.PipeRider is designed for code review. You can initiate the comparison in your local environment.
- 1.Run in the base branch. Usually, it's the main branch.git switch maindbt buildpiperider run
- 2.Run in the target branch. Usually, it's the PR branch for code review.git switch features/my-awesome-featuredbt buildpiperider run
- 3.Generate the comparison report. You then can compare the branch of your new Pull Request against the main branch and explore the impact of your changes by opening the generated HTML comparison report\piperider compare-reports --lastThe
--last
option automatically selects the last two data profiles for comparison. Omit this option to manually select the profiles you would like to compare. - 4.Post the markdown summary to the PR comment. Aside from an HTML report, PipeRider generate a Markdown summary. You can add this summary of the data changes to your Pull Request comment so that your reviewer can review with impact information and merge with confidence🎉
- Specify which models to profile
Last modified 15d ago