dlt test pipeline for azure to snowflake

[britt-allen]

This is a minimal dlt pipeline that loads two Azure SQL database tables into Snowflake. The pipeline ran successfully and the data can be found in raw_dev.dlt. This closes this Asana task.

[ian-r-rose]

The code for running the pipeline looks good to me (I like how short it is!).

The thing that's missing here is the orchestration piece. You ran this locally, and it worked. I can't see the way you connected to the database (that config is probably in a dlt secrets file that's uncommitted?). The question then becomes, what's next? Running something once is different from running an incremental sync several times per day.

It's a little undecided what we might want to do for a tool like dlt, though in the linked Asana task we have:

running dlt inside of a Snowflake stored procedure could work and be pretty low infrastructure. We could also use Airflow for that.

I think we could merge this and create a follow-on task for orchestration, or could do it here. But doing the orchestration piece would force you to answer questions like:

1. What is the service account we should use here?
2. What's the authentication mechanism to the database?
3. Do we need to allow-list any additional IP addresses?
4. What frequency should we sync?

[ram-kishore-odi]

Are you also planning to check in the .gitignore file? Just checking.

[britt-allen]

Are you also planning to check in the .gitignore file? Just checking.

Check for what @ram-kishore-odi ? I am not sure I understand.