Cloud Storage

Automatic data delivery to S3, GCS, and Azure Blob Storage

Kadoa automatically pushes workflow results to your cloud storage after each run. This is ideal for data pipelines, warehousing, and archival.

Supported Providers

1. Amazon S3
2. Google Cloud Storage
3. Azure Blob Storage

Data Formats

Choose which formats to export:

Format	Description	Use Case
Parquet	Columnar, compressed	Analytics, Snowflake, Spark
JSONL	JSON Lines, one record per line	Streaming, log processing
JSON	JSON with metadata envelope	APIs, integrations
CSV	Comma-separated values	Excel, legacy systems

Parquet is recommended for analytics workloads. It’s compressed and optimized for columnar queries.

File Organization

Data is organized by team, workflow, and run:

s3://your-bucket/kadoa/{teamId}/{workflowId}/{runDatetimeSafe}-{runId}/
├── data.parquet
├── data.jsonl
├── data.json
└── data.csv

Path Variables

Variable	Description	Example
{teamId}	Your team UUID	a1b2c3d4-5e6f-7a8b-9c0d-e1f2a3b4c5d6
{workflowId}	Workflow identifier	b2c3d4e5-6f7a-8b9c-0d1e-f2a3b4c5d6e7
{runId}	Run identifier	c3d4e5f6-7a8b-9c0d-1e2f-a3b4c5d6e7f8
{runDatetimeSafe}	Filename-safe datetime	2025-01-15_10-30-00Z
{runDatetime}	ISO 8601 datetime	2025-01-15T10:30:00.000Z
{runDate}	Run date	2025-01-15

{runDatetimeSafe} replaces colons with hyphens and the T separator with an underscore, making it safe for systems that treat colons as illegal path characters (e.g., Windows filesystems, some S3 sync tools). Use {runDatetime} when you need a standard ISO 8601 timestamp and your tooling handles colons correctly.

S3 Setup

Submit a request through the Support Center to configure the data connector with:

• Bucket name
• Region
• Access method (bucket policy or IAM credentials)
• Desired export formats

Export Behavior

Automatic Push

After each workflow run completes successfully:

1. Data is converted to requested formats
2. Files are uploaded to your bucket
3. Export is logged for auditing

Retry Logic

Failed uploads are automatically retried:

• 3 attempts with exponential backoff (1s, 2s, 4s)
• Failures are logged and can trigger alerts

Metadata

Each uploaded file includes S3 metadata:

• x-kadoa-workflow-id: Workflow identifier
• x-kadoa-job-id: Job identifier
• x-kadoa-format: File format

Additional Fields

You can enrich exported rows with extra metadata columns. This is available for CSV, JSONL, and JSON formats (not Parquet).

Each additional field has a custom name (the column header) and a value that can be:

• A static string, e.g. Kadoa or production
• A dynamic variable, e.g. {workflowId} or {runDate}, resolved at export time

Added fields appear as extra columns after the existing data columns. You can configure different additional fields per export format (CSV, JSONL, JSON).

Examples

Field Name	Value	Result
source	Kadoa	Every row gets source = "Kadoa"
workflow	{workflowId}	Every row gets workflow = "b2c3d4e5-..."
workflowName	{workflowName}	Every row gets workflowName = "Product Monitor"
exportedAt	{runDatetime}	Every row gets exportedAt = "2025-01-15T10:30:00.000Z"

Available Variables

Variable	Description	Example
{teamId}	Your team UUID	a1b2c3d4-5e6f-7a8b-9c0d-e1f2a3b4c5d6
{workflowId}	Workflow identifier	b2c3d4e5-6f7a-8b9c-0d1e-f2a3b4c5d6e7
{workflowName}	Workflow name	Product Monitor
{runId}	Run identifier	c3d4e5f6-7a8b-9c0d-1e2f-a3b4c5d6e7f8
{runDate}	Run date	2025-01-15
{runDatetime}	ISO 8601 datetime	2025-01-15T10:30:00.000Z

Use with Snowflake via Snowpipe

Cloud storage can feed a customer-managed Snowflake via Snowpipe pipeline. Kadoa also offers a native Snowflake Integration that does not require customer-managed S3 or Snowpipe.

Kadoa → S3 → Snowpipe → Snowflake Table

Use with Other Data Warehouses

The same S3 data can feed other warehouses:

Warehouse	Integration Method
Snowflake	Snowpipe auto-ingestion
BigQuery	External Tables or Data Transfer
Redshift	COPY command or Spectrum
Databricks	Direct S3 access

Kadoa also offers a native Databricks Integration through Delta Sharing. Use Cloud Storage when you want to own the storage path or ingestion pipeline; use Databricks when you want Kadoa to publish and share Delta tables directly.

Next Steps

• Set up Snowflake with Snowpipe
• Use native Databricks Delta Sharing
• Contact us for setup assistance