How Kadoa works

The building blocks of Kadoa and how they fit together

You describe the data you need. Kadoa’s agent writes deterministic pipeline code that extracts it. Each version of the code is reviewed and tested before it runs.

Kadoa watches every workflow and checks every run. When something goes wrong, such as a site change, a failed run, or a data quality check that doesn’t pass, the agent finds the cause, fixes the code, tests the fix, and redeploys it. If the agent can’t fix it, Kadoa’s operations team takes over.

The rest of this page defines each building block and links to its full page.

Building a workflow

Workflows

A workflow extracts one kind of data from one website. It runs once, on a schedule, or continuously. Everything below is part of a workflow or works with one. See Create workflows.

Sources

A workflow’s source is the set of URLs it extracts from. The agent finds it from your prompt, or you paste the URLs. One workflow covers one website.

Prompts

A prompt tells the agent what to extract and how to reach it, such as which filters to set or how many pages to go through. Your description of the data becomes the prompt.

Schemas

A schema lists the fields of each record, with a name, a data type, and a description. The agent proposes the fields, or uses the ones you name.

Key fields

Key fields identify a record across runs, such as a posting ID or a product URL. They let Kadoa report an edited record as one update instead of a removal and an addition. The agent picks them.

Variables

Variables are named values you use in prompts, such as search terms or ZIP codes. They belong to the team, so every workflow in it can use them. Secret variables hold passwords and API keys, and no one can read their values back.

Running a workflow

Agent

Kadoa’s agent builds each workflow. It finds the source, proposes the schema, and writes and tests the pipeline code. It repairs the code whenever a run or a check fails. You work with it in the dashboard, from code, or from your AI tool.

Previews

Before a workflow goes live, the agent runs it on a small sample, the preview. You check the data and approve it. Previews are free.

Runs

A run is one execution of a workflow’s pipeline code. A workflow runs once or on a schedule. Workflow lifecycle lists every state a workflow and a run can be in.

Monitors

A monitor is a workflow that watches a website continuously instead of on a schedule. It alerts you when the data changes. Use one when minutes matter.

Changes

After each run, Kadoa compares the data with the previous run and marks each record as added, removed, or updated. Change detection explains how records are matched.

Scaling up

Templates

A template is one configuration that many workflows share, such as the setup prompt, schema, and schedule. You change the template once, then roll the change out to its linked workflows.

Datasets

A dataset is the same kind of record collected from many websites. Each website gets its own workflow, and one template keeps them all in the same structure.

Delivery and alerts

Destinations

Destinations receive the data of each completed run: Amazon S3, Google Cloud Storage, Snowflake, Databricks, or SFTP. You can also export data on demand as a download, through the API, or from your AI tool.

Notifications

Notifications alert you when a workflow finishes, fails, recovers, or finds changes. Each one goes to a channel: an email address, a Slack channel, a webhook, or a WebSocket.

Teams

A team holds your workflows, data, variables, templates, and channels. Every member sees the team’s workflows, and roles decide who can change what.