Sources

​

Setting up a Resource

A resource serves as a connection to data sources. Resources can be a data warehouse (BigQuery, Snowflake), transactional databases (Postgres), object storage (S3), data lakes (Delta) or local files (parquet, csv, json). Vinyl comes with standard resource connectors:

• TableFileConnector (csv, json, parquet)
• BigQueryConnector
• PostgresConnector
• DatabaseFileConnector (duckdb)

Resources are defined in the resources.py file of our project. Resources are used to fetch metadata and source data queries. Here’s an example of a postgres connection:

from vinyl import resource
from vinyl.connectors import PostgresConnector

@resource
def saas_product_data():
    return PostgresConnector(
        host="db.mypostgres.com",
        port="5432",
        user="postgres",
        password="password",
        tables=["postgres.*.*"],
    )

​

Generate Sources

Once we’ve defined a resource, we can pull in source data schemas automatically using the following vinyl CLI command

vinyl generate sources

This command populates the sources/ directory with typed source data schemas. These schemas are used to understand the structure of the data and to generate source data queries. Here’s an example of a source schema for a postgres table:

class User:
    user_id: t.String(nullable=False)
    created_at: t.Timestamp(timezone=None, scale=None, nullable=True)
    email: t.String(nullable=True)
    karma: t.Int32(nullable=True)
    last_login: t.Timestamp(timezone=None, scale=None, nullable=True)
    payment_method_id: t.String(nullable=True)

​

Using Sources

Sources are used just like any python class. You can import a source in a notebook, script or vinyl project:

from my_project.sources import User

user_table = User()
print(user_table.schema)

Generated sources serve as a data contract. With sources, you can validate data transforms without hitting the database and detect upstream schema drift to prevent breaking changes.

​

List Sources

As you’re adding resources and generating sources, you can list all the sources in your project using the following vinyl CLI command:

vinyl sources list