Data Quality

Read and edit the per-field data quality rules of a workflow using the SDK

Data quality rules are per-field checks that run at the end of every workflow run. Kadoa generates a default set for each workflow. With the SDK you read the current rules, replace the rules of specific fields, and remove the rules of a field.

For the available rule kinds and how each one is evaluated, see Data Quality.

Prerequisites

• Kadoa account with API key
• SDK installed: npm install @kadoa/node-sdk
• A workflow with a schema. Rules are keyed by schema field name.

Data quality rules are currently available in the Node SDK. Python SDK support is coming soon.

Set rules for fields

Fields in the request replace their existing rules. Fields you leave out keep theirs. Every rule carries editedBy and editedAt, so the dashboard can show who changed it. API clients send editedBy: "user".

Node SDK

import { KadoaClient } from "@kadoa/node-sdk";

const client = new KadoaClient({ apiKey: "YOUR_API_KEY" });
const workflowId = "WORKFLOW_ID";

// Every rule records who set it and when
const edited = {
  editedBy: "user",
  editedAt: new Date().toISOString(),
} as const;

const rules = await client.dataQuality.upsertRules(workflowId, {
  title: {
    kind: "STRING",
    presence: { target: 100, ...edited },
    maxLength: { value: 120, ...edited },
  },
  link: {
    kind: "STRING",
    format: {
      kind: "FORMAT",
      source: { kind: "PRESET", preset: "url" },
      ...edited,
    },
  },
});

console.log(Object.keys(rules)); // ["title", "link"]

The response is the merged ruleset of the whole workflow. Rule edits apply on the next run.

Rule shapes

The kind of a field selects its rule shape and casts the field’s values to that type before evaluation.

Kind	Rules
STRING	presence, uniqueness, minLength, maxLength, minHtmlElements, maxHtmlElements, format
NUMBER	presence, uniqueness, minimum, maximum, maxDecimalPlaces
DATE	presence, uniqueness, minimum, maximum (a fixed YYYY-MM-DD date or a preset relative to the run date)
OBJECT	presence, uniqueness, additionalProperties, and properties with the rules of each sub-field
ARRAY	presence, uniqueness, minItems, maxItems, and items with the rules applied to every element
OTHER	presence, uniqueness

presence and uniqueness take a target of 0, 20, 40, 60, 80, or 100 percent. A string format is one of three kinds: FREE_TEXT with a charset preset, FORMAT with a pattern preset or a custom regular expression, and LIST with a list preset or custom values. The Node SDK exports these shapes as types, starting from DataQualityRules.

Read the rules of a workflow

Node SDK

import { KadoaClient } from "@kadoa/node-sdk";

const client = new KadoaClient({ apiKey: "YOUR_API_KEY" });
const workflowId = "WORKFLOW_ID";

// null when the workflow has no rules
const rules = await client.dataQuality.getRules(workflowId);

for (const [field, fieldRules] of Object.entries(rules ?? {})) {
  console.log(field, fieldRules.kind, fieldRules.presence?.target);
}

Remove the rules of one field

Node SDK

import { KadoaClient } from "@kadoa/node-sdk";

const client = new KadoaClient({ apiKey: "YOUR_API_KEY" });
const workflowId = "WORKFLOW_ID";

// Other fields keep their rules
const remaining = await client.dataQuality.deleteFieldRules(
  workflowId,
  "link",
);

console.log(Object.keys(remaining)); // ["title"]

Limits

• A rule tree nests at most 4 levels deep and holds at most 200 rules.
• Workflows linked to a template inherit their rules from the template. Editing them returns 409 with code TEMPLATE_CONTROLLED_FIELD. Unlink the workflow from its template first.
• Edits do not change the results of the current run. They apply from the next run on.

Learn more

• Data quality concepts and rule kinds
• Edit rules in the UI
• API reference