The Work You Know Best Is Often the Hardest to Document
After a few years in software or data engineering, many decisions begin to feel automatic.
An on-call alert fires. You know which dashboard to check, which log group to open, and what to search for. When you inherit a data pipeline, you look for null checks and schema validation before reading its main logic. A new requirement arrives, and you can quickly tell whether it looks like one day of work or a week.
These judgments happen almost without thinking. But ask yourself: "Could you write this all down so a new hire could follow it step by step?" You'd probably pause.
Knowledge management calls this “knowing how to do something but struggling to explain it” tacit knowledge. In Nonaka's SECI model, externalization is the move from tacit knowledge to knowledge that can be expressed and shared.
As AI-assisted development tools became routine, Skills offered another place to encode that knowledge.
A Skill is a workflow specification that AI can discover, load, and execute step by step. It is still documentation, but it goes beyond a README or prompt template by defining triggers, branches, constraints, and acceptance criteria. For an engineer, it is an interface through which AI can apply engineering judgment.
The rest of this article covers how to move from experience in your head to a Skill you can rely on.
How a Skill Loads: Relevance First, Details Later
Before you start writing, understand how AI actually uses your Skill — otherwise you'll invest effort in the wrong places.
AI usually does not load every Skill file into context at once. It uses progressive disclosure: read enough to judge relevance, then pull in details when the task requires them.
Layer one: The description in frontmatter. AI sees this in every conversation. Token cost is tiny, but these few sentences are how it decides "Is this Skill relevant to the current task?" This is the make-or-break filter.
Layer two: The SKILL.md body. Only loaded into context after AI determines the Skill is relevant. Your full instructions, decision logic, and constraints live here.
Layer three: Resources in the references/ directory. API docs, templates, example code. AI only reaches for these when it has a concrete need.
You can think of the description as the first paragraph of a README, the body as API documentation, and references as the examples/ directory.
If the frontmatter is unclear, the rest of the Skill may never load.
Step 1: Mine Your Experience from Real Scenarios
A common mistake is to skip extraction and jump straight into SKILL.md.
The result is either too abstract ("analyze the problem and propose a solution") or too granular (listing fifteen CLI commands without explaining when to use which one).
A more reliable approach is to revisit at least three to five real cases and ask, “What would AI need to know to take this over?”
Open your work notes, Slack conversations, or PR review history from the past month. Find tasks you've done repeatedly. If you can identify the same judgment pattern across three different cases, that's a strong candidate for a Skill.
Then use AI as a structured interviewer. Do not ask it to write the Skill yet; ask it to extract rules from your cases:
I recently handled these three data pipeline incidents:
[Case A: Upstream schema change caused downstream null pointer]
[Case B: Kafka lag spike caused processing delays]
[Case C: S3 permission update caused silent ETL job failure]
From these three cases, please identify:
1. What criteria I used to classify the problem type
2. My fixed investigation sequence or priority order
3. Under what conditions I skip certain steps
4. Things I absolutely never do (e.g., never manually modify production tables)
AI can find patterns in concrete examples, but it cannot infer your preferences from nothing. You supply the cases; it helps organize the repeated judgments.
After a few rounds of back-and-forth, you'll have a rough but authentic process transcript. This isn't the final Skill, but it's your most genuine raw material.
Step 2: Compress Raw Material into Engineering Structure
With a transcript in hand, the next step is structuring it. This step determines whether AI can follow your process as a spec rather than freestyling.
From a data engineering perspective, I think of it as schema design — except you're defining not data fields but the "judgment fields" and "transformation rules" for AI when processing a task.
A good Skill body typically consists of the following building blocks (not all required every time — choose based on complexity):
Decision Branches: Let AI Take Different Paths Based on Input
Your workflow has decision points. The same type of problem — different source, different severity — may require a completely different approach.
A Skill without decision branches forces AI onto the default path every time — usually whichever it considers "safest," which isn't necessarily what you need.
When designing decision branches, keep the node count to four or fewer. Each additional judgment layer degrades AI's execution accuracy. Same principle as designing branching logic in a data pipeline — too many branches equals no branches.
Step-by-Step Procedures: Write It Like a Runbook, Not a Manual
Execution steps should be written as directives AI can act on directly, not concept descriptions.
Bad: "Check data quality." Good: "Run null count on the latest batch of the target table; if it exceeds 5%, halt subsequent steps and report."
Think of each step as a runbook entry — the kind of clarity that lets you follow it at 3 AM when your brain is half asleep. AI needs that same level of clarity.
The NEVER List: Your Hard-Won Lessons
The NEVER list is often one of the most valuable sections in a Skill.
Without explicit constraints, AI fills gaps with generic defaults. In engineering work, those defaults are often where the risk sits. A useful Skill therefore defines both the required procedure and the boundaries it must not cross.
Every NEVER rule should map to a real incident you've lived through. For example:
- NEVER modify production resources without a dry-run
- NEVER assume upstream schema won't change — always do defensive parsing
- NEVER write credentials into code or logs
Rules tied to actual incidents or near misses are more useful than generic safety slogans. AI does not know what has gone wrong inside your organization; you have to provide that context.
Done Criteria: What Does "Finished" Actually Mean?
A Skill without done criteria leaves AI unsure where to stop. It might finish too early or keep going down rabbit holes it doesn't need to explore.
Good done criteria should be mechanically verifiable, for example:
- All relevant tests pass
- Output files conform to the specified schema
- git diff is within expected scope
- No lint errors or type errors
Same thinking as a data pipeline quality gate — you wouldn't let a pipeline go live without passing data validation, and you shouldn't accept a Skill execution result that hasn't passed acceptance criteria.
Step 3: Draw the Trigger Boundary in Frontmatter
Back to the first layer of progressive disclosure. The description in frontmatter is the sole basis for AI's decision to load or skip your Skill.
Many people treat description as a field to fill in casually. In reality, it's closer to an API endpoint routing rule — it defines which requests should be routed here.
A well-designed description addresses three dimensions simultaneously:
Positive trigger conditions (when to apply this Skill):
USE FOR: data pipeline failure diagnosis, silent ETL job troubleshooting,
upstream-downstream schema mismatch investigation
Negative exclusion conditions (when not to use it):
DO NOT USE FOR: local dev environment debugging, SQL query optimization,
new pipeline architecture design (use pipeline-design skill instead)
Semantic trigger words (how users might phrase it):
Trigger words: pipeline broken, ETL failed, data missing,
schema mismatch, job stuck
Negative exclusions are easy to miss. A Skill without boundaries resembles an API endpoint with no content-type check: unrelated requests get routed there, and false triggers multiply.
Step 4: Let AI Generate the First Draft, Then Correct It with Your Experience
With structured material ready, you can now ask AI to assemble the first version of SKILL.md.
Your prompt should include full context — not just "write me a Skill":
I need to create an Agent Skill for [one-sentence description].
Here's the material I extracted from real cases:
## Trigger Conditions
[trigger conditions from the first step]
## Out-of-Scope Scenarios
[scenarios where this Skill should NOT apply]
## Decision Branches
[your decision nodes]
## Execution Steps
[your step-by-step workflow]
## Red Lines
[your NEVER list]
## Done Criteria
[verifiable completion conditions]
Please output in SKILL.md format, with frontmatter including name and description.
The description should include both positive trigger words and DO NOT USE FOR exclusions.
AI-generated first drafts commonly have two problems:
- Overly generic step descriptions. AI abstracts your specific logic — turning "check Kafka consumer group lag" into "check message queue delay" — which loses precision.
- Including things AI already does by default. "Ensure the code compiles" or "use correct Markdown formatting" are pure noise — AI doesn't need you to remind it of basics, and writing these wastes context space.
After receiving the draft, your correction tasks are:
- Revert generalized steps back to your specific scenario language
- Remove all statements about things AI already does by default
- Verify every NEVER rule maps to a real, memorable incident
For each line in the Skill, ask: “Would removing this make the output worse?” If not, delete it. Keep the judgment AI cannot derive on its own. Extra prose consumes tokens and makes important rules harder to notice.
Step 5: Test the Skill from Three Directions
You wouldn't deploy a data pipeline straight to production the moment it compiles. Same goes for Skills.
Testing a Skill involves three distinct verification dimensions, each catching a different category of problems:
Trigger Boundary Testing
Is your description accurate? Prepare three sets of conversations:
- Positive match: Clearly belongs to this Skill's scenario. Example: "Help me figure out why pipeline X didn't run this morning"
- Synonym rewrite: Same meaning, different phrasing. Example: "ETL job X's output table is empty today — what happened?"
- Unrelated: Completely different intent. Example: "Write me a Python function to read a CSV"
If the first two don't trigger, your description is too narrow. If the third also triggers, it's too broad.
Process Compliance Testing
Once triggered, does AI actually follow the steps you wrote?
The point isn't whether AI completed the task — it's whether it skipped steps. When AI encounters vague step descriptions, it tends to self-assess importance and skip what it considers unimportant.
Skipped steps almost always point to the same root cause: the instruction for that step isn't specific enough. Changing "check relevant settings" to "read the source_table and target_table fields in config.yaml and verify both sides have matching schema versions" usually fixes the problem.
With-Skill vs Without-Skill Comparison Testing
Run the same task twice — once with the Skill enabled and once without. Compare output quality.
If the difference isn't significant, your Skill isn't actually guiding AI — likely because most of the body's content is stuff AI would do anyway. Time to go back and re-distill, adding more content that genuinely comes from your unique experience.
Feed Every Failure Back into the Spec
If you've built data pipelines, you know this: the first version is never the final version.
Schemas evolve, upstream sources change formats, business logic shifts. Your pipeline needs ongoing maintenance. Skills are exactly the same.
After each time AI executes your Skill, observe where the output diverges from your expectations. Don't just accept or reject — trace back to which rule wasn't clear enough and fix it.
Common iteration scenarios:
AI doesn't trigger the Skill: Usually the description lacks sufficient keyword coverage. An effective debugging method — ask AI directly "When would you use this Skill?" It will quote the description back to you, and you'll immediately see what's missing.
AI triggers too often: Add negative exclusion conditions. Explicitly tell it "this isn't your job — for scenario X, use a different Skill."
AI completes but the result is wrong: Step descriptions aren't specific enough, or a decision branch is missing a scenario.
AI does something it shouldn't: The NEVER list needs expansion — add the new edge case you just discovered.
Each correction moves another piece of unwritten judgment into the specification.
Classification Thinking: Not Every Skill Produces Code
Engineering experience isn't limited to writing code. Your judgment takes different forms at different stages, and the Skill design should reflect that.
Companion/Thinking type: Guides you through clarifying your thoughts without directly producing output. For example, requirements review — AI shouldn't jump into implementation but should ask you a series of questions first. Characteristics: open-ended input, output is a decision direction rather than code files, minimal tool requirements.
Planning type: Converts fuzzy requirements into actionable plans. Output is typically structured documents (task lists, architecture decision records). These Skills draw on your schema design experience — which fields are required, which have defaults, which formats upstream and downstream systems accept.
Execution type: Given a clear spec, it gets to work. Writing tests, doing code review, running deployment workflows. These Skills have the most detailed steps, the strictest NEVER lists, and the most rigorous done criteria — because execution-type errors are the most expensive.
Before you start writing, determine which type your Skill belongs to. Different types have different structural centers of gravity — writing a planning Skill with execution-type rigidity makes it too rigid, writing an execution Skill with thinking-type open-ended guidance makes it uncontrollable.
A Finished Skill Still Needs Verification
The file-level barrier is low: one folder and one Markdown file. The hard part is stating when the workflow applies, how decisions are made, what must never happen, and what counts as done.
Anthropic's writing on agent systems emphasizes that tool interfaces matter as much as the tools themselves. SKILL.md is the interface between you and AI, so it deserves the same version control and testing discipline as code.
A useful Skill can be versioned, tested, and revised. Start with three to five real cases, turn the judgments you make automatically into explicit conditions, and then verify that the Skill actually changes how AI handles the task.