Understand baselines and drift detection.
Learn how Knotbind captures response baselines, detects structural drift, and helps you distinguish breaking changes from safe API evolution.
What is a baseline?
A Baseline is the shape of a Check's answer, recorded
the first time that Check passes. Knotbind keeps the fields, their types
and how they nest, and not the values. A field such as
"id": "cust_12345" is kept as "id": string.
On every later Run, the Check's answer is compared with its Baseline. If the shape is different, the Check fails, even when the call itself returned what the Check expected.
Why Baselines matter
An assistant that relies on a field breaks when that field disappears or changes type. A Baseline notices that on the next Run, instead of an assistant finding out in production.
How a Baseline is captured
-
The first passing Run
The first time a Check passes, Knotbind records the shape of its answer as that Check's Baseline. Each Check has its own.
-
Shape, not values
Only the structure is kept: which fields exist, their types (
string,number,array) and nesting. A different id or timestamp on its own does not fail the Check.
What counts as a change
A Baseline is strict. Any difference in shape fails the Check:
- A field is missing that was in the Baseline
- A field is new, one that was not in the Baseline. This fails the Check too, even though adding a field is usually safe for an assistant.
- A field has a different type, for example a string that is now a number
A Baseline does not tell a safe change from a breaking one. It tells you that the answer changed, and you decide.
What happens when the shape changes
-
The Check fails.
The Run is marked failed, and the result says which field differs.
-
After two failing Runs in a row, a Finding opens.
One failure is not yet a Finding, so a single flaky call does not wake you. You get an alert when the Finding opens.
-
You decide what the change is.
It was either intended or it is a bug. See below.
When you meant to change the answer
If the change was planned, accept it as the new Baseline. Open Dashboard, find the failed Run in the Runs table, open the failed Check, and press Accept as new Baseline.
- The Baseline becomes the answer the Check just returned, and the Audit Record notes who accepted it.
- The button appears only when the call returned what the Check expected and the shape is what changed. If the call failed outright (an error status, no response), it is not offered, because accepting that would hide a real outage.
- It appears on the latest Run only, and only for the Owner.
Baselines and declared schemas
A Check can also carry a declared schema, for example a
tool's published outputSchema. The two are different:
| Baseline | Declared schema | |
|---|---|---|
| Who defines it | Knotbind, from the first passing Run | You, or the tool's own output schema |
| What it checks | The answer has exactly the shape it had | The answer fits the schema, extra fields allowed |
| A new field | Fails the Check | Passes, unless the schema forbids it |
| How you resolve it | Accept as new Baseline, or fix the API | Fix the API, or change the schema |
Common scenarios
You added a new field
The Check fails, because the Baseline did not have that field. If you meant it, accept the new Baseline. Assistants that ignore the new field are not affected.
You renamed a field
The old name is missing and a new one appeared, so the Check fails. This one can break assistants that used the old name. Accept the new Baseline only when the assistants have been told.
A field changed from string to number
Type changes break clients that expect a type. Either revert the change, or accept it and let assistant owners know.
A field is sometimes missing
The Check fails on the runs where the field is absent. If the field should always be there, fix the API. If the answer legitimately varies in shape, a Baseline is the wrong tool for that Check, because it compares the whole shape every time.
The Check fails on every Run
A different id, timestamp or other value does not fail a Baseline, since only the shape is compared. If it fails every time, the shape itself is changing between calls. Read the result to see which field differs.
Best practices
- Accept early. If the change is intended, accept it at the first failure so no Finding opens.
- Write breaking changes down. When you accept a new Baseline after a rename or a type change, add it to your API changelog so assistant developers know.
- Use a declared schema for what must never change. Required fields and types you promise belong in a schema, which keeps failing when they break.
- Test before you deploy. Run your Checks against a staging copy before an API change reaches production.