Data Engineer Path

About this Path

A path for data engineers who are new to D.Hub.

You register a public weather API as a data connection, ingest its hourly readings into a dataset, aggregate them into a daily summary, put the whole thing on a daily schedule, and finish with the dashboard chart an analyst sees. After six lessons, your training collection holds one working pipeline and one dashboard built on its output.

The point of this path is that everything from attaching an external system to handing the result to someone else forms a single line.

flowchart LR
    accTitle: Data Engineer path end-to-end flow
    accDescr: A public REST API is registered as a data connection, ingested by a code node into an hourly dataset, cleaned and aggregated with Polars into a daily mart, put on a schedule with run monitoring, and handed off as a dashboard chart.
    api[(Public weather API)] --> connector[REST data connection]
    connector --> code[Code node ingest]
    code --> src[(Hourly readings)]
    src --> prepare[Preparation code]
    prepare --> stg[(Cleaned hourly)]
    prepare --> mart[(Daily summary mart)]
    schedule[Daily schedule] --> code
    mart --> dashboard[Dashboard chart]

Terms to know before you start

• Data connection (connector): A reusable bundle of address and authentication settings for an external system. This path registers a public weather API here.
• Pipeline: An execution flow that reads data, transforms it, and writes it somewhere. This path grows one pipeline lesson by lesson.
• Code node: A pipeline step written in Python for work no standard transform can express. Calling an external API is the classic case.
• Transform node: A no-code step for renaming, casting, filtering, joining, and aggregating.
• Mart: The result dataset an analyst can use directly, once aggregation and naming are settled.

Prerequisites

• Access to the D.Hub portal. Registering a data connection requires the admin or manager role.
• One collection for the exercise. Every connector, dataset, code asset, pipeline, and dashboard in this path lands there.
• Either the Analyst Path or a working understanding of collections and datasets.
• A D.Hub environment with outbound internet access. The connection test runs from the manager process, not your browser, so a blocked outbound path stops you in lesson 02.

Nothing here requires a signup or an API key. The API this path uses is a public service you can call without authentication.

Prior experience with dbt, Airflow, or Snowflake will help you map D.Hub's concepts faster.

What you'll be able to do

• Register a managed REST data connection and inspect its response with the connection test and Query Console.
• Bind a data connection to a code node and load an external API's response into a dataset.
• Clean a raw dataset and build an analysis-ready daily mart with a Polars code node.
• Choose write and read modes so repeated runs do not corrupt the result.
• Put a batch pipeline on a recurring schedule and read its run history.
• Trace a failed run to its cause in history, fix it, and rerun the pipeline.
• Document a result dataset, set permissions, and hand it to an analyst as a dashboard chart.

What comes after this Path

• Retail Inventory Intelligence — Practice connectors, pipelines, code nodes, and schedules together in one domain scenario. A 90-minute course covering the full analyst and engineer workflow.
• Quick scenario import (advanced, 10 min) — Pull a dhub2-examples scenario into your own environment with a single command.

Selecting the checkbox next to each lesson saves your progress automatically.