> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kadoa.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Cloud Storage

> Automatic data delivery to S3, GCS, and Azure Blob Storage

Kadoa automatically pushes workflow results to your cloud storage after each run. This is ideal for data pipelines, warehousing, and archival.

## Supported Providers

1. Amazon S3
2. Google Cloud Storage
3. Azure Blob Storage

## Data Formats

Choose which formats to export:

| Format      | Description                     | Use Case                    |
| ----------- | ------------------------------- | --------------------------- |
| **Parquet** | Columnar, compressed            | Analytics, Snowflake, Spark |
| **JSONL**   | JSON Lines, one record per line | Streaming, log processing   |
| **JSON**    | JSON with metadata envelope     | APIs, integrations          |
| **CSV**     | Comma-separated values          | Excel, legacy systems       |

<Note>
  Parquet is recommended for analytics workloads. It's compressed and optimized for columnar queries.
</Note>

## File Organization

Data is organized by team, workflow, and run:

```
s3://your-bucket/kadoa/{teamId}/{workflowId}/{runDatetimeSafe}-{runId}/
├── data.parquet
├── data.jsonl
├── data.json
└── data.csv
```

### Path Variables

| Variable            | Description            | Example                                |
| ------------------- | ---------------------- | -------------------------------------- |
| `{teamId}`          | Your team UUID         | `a1b2c3d4-5e6f-7a8b-9c0d-e1f2a3b4c5d6` |
| `{workflowId}`      | Workflow identifier    | `b2c3d4e5-6f7a-8b9c-0d1e-f2a3b4c5d6e7` |
| `{runId}`           | Run identifier         | `c3d4e5f6-7a8b-9c0d-1e2f-a3b4c5d6e7f8` |
| `{runDatetimeSafe}` | Filename-safe datetime | `2025-01-15_10-30-00Z`                 |
| `{runDatetime}`     | ISO 8601 datetime      | `2025-01-15T10:30:00.000Z`             |
| `{runDate}`         | Run date               | `2025-01-15`                           |

<Note>
  `{runDatetimeSafe}` replaces colons with hyphens and the `T` separator with an underscore, making it safe for systems that treat colons as illegal path characters (e.g., Windows filesystems, some S3 sync tools). Use `{runDatetime}` when you need a standard ISO 8601 timestamp and your tooling handles colons correctly.
</Note>

## S3 Setup

Submit a request through the [Support Center](/docs/support-center) to configure the data connector with:

* Bucket name
* Region
* Access method (bucket policy or IAM credentials)
* Desired export formats

## Export Behavior

### Automatic Push

After each workflow run completes successfully:

1. Data is converted to requested formats
2. Files are uploaded to your bucket
3. Export is logged for auditing

### Retry Logic

Failed uploads are automatically retried:

* 3 attempts with exponential backoff (1s, 2s, 4s)
* Failures are logged and can trigger alerts

### Metadata

Each uploaded file includes S3 metadata:

* `x-kadoa-workflow-id`: Workflow identifier
* `x-kadoa-job-id`: Job identifier
* `x-kadoa-format`: File format

## Additional Fields

You can enrich exported rows with extra metadata columns. This is available for CSV, JSONL, and JSON formats (not Parquet).

Each additional field has a custom **name** (the column header) and a **value** that can be:

* A **static string**, e.g. `Kadoa` or `production`
* A **dynamic variable**, e.g. `{workflowId}` or `{runDate}`, resolved at export time

Added fields appear as extra columns after the existing data columns. You can configure different additional fields per export format (CSV, JSONL, JSON).

### Examples

| Field Name     | Value            | Result                                                   |
| -------------- | ---------------- | -------------------------------------------------------- |
| `source`       | `Kadoa`          | Every row gets `source = "Kadoa"`                        |
| `workflow`     | `{workflowId}`   | Every row gets `workflow = "b2c3d4e5-..."`               |
| `workflowName` | `{workflowName}` | Every row gets `workflowName = "Product Monitor"`        |
| `exportedAt`   | `{runDatetime}`  | Every row gets `exportedAt = "2025-01-15T10:30:00.000Z"` |

### Available Variables

| Variable         | Description         | Example                                |
| ---------------- | ------------------- | -------------------------------------- |
| `{teamId}`       | Your team UUID      | `a1b2c3d4-5e6f-7a8b-9c0d-e1f2a3b4c5d6` |
| `{workflowId}`   | Workflow identifier | `b2c3d4e5-6f7a-8b9c-0d1e-f2a3b4c5d6e7` |
| `{workflowName}` | Workflow name       | `Product Monitor`                      |
| `{runId}`        | Run identifier      | `c3d4e5f6-7a8b-9c0d-1e2f-a3b4c5d6e7f8` |
| `{runDate}`      | Run date            | `2025-01-15`                           |
| `{runDatetime}`  | ISO 8601 datetime   | `2025-01-15T10:30:00.000Z`             |

## Use with Snowflake via Snowpipe

Cloud storage can feed a customer-managed
[Snowflake via Snowpipe](/docs/integrations/snowflake-snowpipe) pipeline.
Kadoa also offers a native [Snowflake Integration](/docs/integrations/snowflake)
that does not require customer-managed S3 or Snowpipe.

```
Kadoa → S3 → Snowpipe → Snowflake Table
```

## Use with Other Data Warehouses

The same S3 data can feed other warehouses:

| Warehouse      | Integration Method               |
| -------------- | -------------------------------- |
| **Snowflake**  | Snowpipe auto-ingestion          |
| **BigQuery**   | External Tables or Data Transfer |
| **Redshift**   | COPY command or Spectrum         |
| **Databricks** | Direct S3 access                 |

Kadoa also offers a native [Databricks Integration](/docs/integrations/databricks)
through Delta Sharing. Use Cloud Storage when you want to own the storage path
or ingestion pipeline; use Databricks when you want Kadoa to publish and share
Delta tables directly.

## Next Steps

* [Set up Snowflake with Snowpipe](/docs/integrations/snowflake-snowpipe)
* [Use native Databricks Delta Sharing](/docs/integrations/databricks)
* [Contact us](https://www.kadoa.com/contact?type=sales) for setup assistance