Before You Build a Cortex Agent: Turn Business Questions into Semantic View Acceptance Criteria

Summary

Before building a Cortex Agent, turn the business logic in a Datamart into testable questions and Ground Truth: the evidence you will use for acceptance. Using a Datamart pipeline I maintain as an example, this article explains how I use a company-approved AI development assistant to identify candidate questions, design a Semantic View from them, and evaluate Cortex Analyst and Cortex Agent separately. If the requirement is limited to question answering over structured data, the workflow can stop at Cortex Analyst. A Cortex Agent becomes useful when the task spans documents, tools, or multiple coordinated steps. Only after the answers are stable do I expose the capability to external applications through MCP.

When I started implementing a Snowflake Semantic View and studying Cortex Agent and MCP, I assumed that connecting tables to an agent would be enough for users to start asking questions.

In practice, configuration was not the first obstacle. The question list was: What, exactly, should the system answer, and what answers will we use to accept or reject it?

Two different roles appear in this article. The “AI development assistant” helps organize requirements and draft configuration. “Cortex Agent” is the production agent deployed in Snowflake to answer users. They serve different purposes.

Why Can an Agent Still Be Wrong After You Connect the Tables?

I am responsible for a Datamart pipeline at my company. Because I help develop and maintain it, I know where the data comes from and how it is transformed. I also have more context on how metrics are calculated, which records must be excluded, how statuses should be interpreted, which date field represents the business event, and which tables can be joined safely.

Those rules rarely appear in full inside the tables. They are scattered across SQL, application code, tickets, documentation, and the developers' own understanding. If a model receives only column names, data types, and sample values, it still does not know why a number must be calculated in a particular way.

A Semantic View centralizes business concepts, data entities, and relationships in a native, schema-level Snowflake object. Cortex Analyst reads those semantic definitions and generates SQL against the physical tables. When a Cortex Agent needs to query structured data, it uses the Semantic View through a Cortex Analyst tool. That is why I would not let a production agent guess directly from raw data. Snowflake's Semantic View overview

But an earlier question comes before Semantic View design: Who will use this data, for which decision, and what action will they take after receiving the answer?

An AI Assistant Produces Question Hypotheses, Not Requirements

My first step is to give the documented pipeline logic, column definitions, and intended data uses to a company-approved AI development assistant and ask it to propose candidate questions. I defer the decision to build a Cortex Agent until the semantic layer has passed evaluation. I do not provide credentials, personal data, actual data rows, or business information that should not leave the controlled environment.

The assistant can turn rules hidden in the pipeline into questions, but it does not know whether users genuinely care about them. Its output is a set of requirements hypotheses that still need validation through user interviews, existing reports, tickets, and query logs. Snowflake's Semantic View Autopilot also considers Query History instead of relying only on a model to infer use cases. Semantic View Autopilot

I organize the validated questions into question cards. The examples below illustrate the format; they are not based on my company's actual data or schema.

This table forces me to answer four questions: Is the available data sufficient? Can the question be interpreted in more than one way? How costly would a wrong answer be? Should the system answer, ask a follow-up question, or refuse?

Homepage examples are only one use for the question list. Its more important job is to drive design and acceptance together.

How Do Questions Become Semantic View Acceptance Criteria?

I design the Semantic View only after the questions are validated. I give the Datamart structure, the business logic embedded in the pipeline, and the question cards to the AI development assistant, then ask it to help draft logical tables, relationships, dimensions, facts, metrics, synonyms, and descriptions.

It is important to distinguish a fact from a metric. A fact is a row-level property in a logical table, often describing “how much” or “how many.” A metric is a business measure aggregated across rows. A numeric field still belongs at the level appropriate to its grain and calculation. Semantic View concepts

AI-generated definitions cannot be treated as correct by default, especially metrics, relationships, and filters. A model can read SQL and infer a column's likely purpose without understanding the business rule behind an ordinary-looking condition. The following example is deliberately simplified; neither its field names nor its conditions come from my company's schema:

WHERE status = 'COMPLETED'
  AND is_test_data = FALSE
  AND effective_date <= CURRENT_DATE

These conditions might mean, respectively, that only completed cases count in official reporting, test data must be excluded, and records that are not yet effective cannot be counted early. A Semantic View should preserve the business meaning of those conditions, not only their SQL.

I treat a Semantic View as a semantic contract between the data team and the business. It centralizes the metrics, dimensions, and relationships used to understand structured data, reducing the room for a model to guess. It cannot guarantee a correct answer by itself. Data quality, freshness, access controls, and testing still affect the result.

One product behavior also requires attention. Cortex Analyst's current Routing Mode prioritizes semantic SQL, but if the Semantic View cannot cover a question, it can fall back to standard SQL over the physical tables. Because this feature is currently in Preview, evaluation should monitor fallback queries rather than assume that every generated statement uses only definitions from the Semantic View. Routing Mode

How Are Verified Queries Different from an Evaluation Dataset?

The same business questions feed two acceptance mechanisms, but those mechanisms test different things.

A Verified Query stores a natural-language question and validated SQL. It can be added to a Semantic View to help Cortex Analyst handle similar questions. Snowflake can also run a Cortex Analyst Evaluation directly against a Semantic View, using verified queries as acceptance evidence for SQL correctness, regression, and latency. Creating verified queries for a Semantic View, Cortex Analyst Evaluations

A Cortex Agent Evaluation Dataset tests the complete agent: whether it selects the right tool, executes that tool correctly, and produces an acceptable final answer. The official dataset format has two columns, an input query and Ground Truth. Ground Truth is a VARIANT that can contain ground_truth_output and ground_truth_invocations.

The JSON below illustrates the format only. The actual tool_name must match the tool exposed by the agent; a generic label such as “Cortex Analyst” is not sufficient.

{
  "ground_truth_output": "The answer must use the specified period, completion status, and exclusions, then return the case count from the acceptance snapshot.",
  "ground_truth_invocations": [
    {
      "tool_name": "case_metrics_analyst",
      "tool_input": "How many cases were completed last month?",
      "tool_output": "Use completion date, exclude test data, and deduplicate by case ID."
    }
  ]
}

Ground Truth is closer to acceptance evidence than to a single “correct answer.” It can preserve the expected output, tool, input, output conditions, and tolerance. For operational data that changes over time, I pin the query date or data snapshot and record the applicable period. I do not treat one number as permanently correct.

Every core Verified Query and Ground Truth record also needs an owner, confirmation date, and scope. Important metrics should be approved by the relevant data owner or business owner. A verified query in a Semantic View supports VERIFIED_BY and VERIFIED_AT, allowing responsibility and timing to be recorded with the query.

The test set cannot contain only exact copies of the original questions. I keep the original business intent and acceptance evidence, then add paraphrases, alternate date expressions, ambiguous conditions, out-of-scope requests, and previous failures. The original wording protects against regressions; the variations test generalization and clarification behavior. Cortex Agent Evaluations

When Do You Actually Need a Cortex Agent?

After the Semantic View and Cortex Analyst pass evaluation, I ask a basic question: Does this requirement actually need a Cortex Agent?

If users only need natural-language queries over structured data, Cortex Analyst may already be enough. Cortex Agent becomes materially useful when a task spans structured and unstructured data, breaks into multiple steps, requires a choice among tools, or must preserve conversational context. Snowflake's recommended use cases for Cortex Agents

A Cortex Agent is suited to deciding whether a request belongs with Cortex Analyst, Cortex Search, a custom tool, or an MCP connector, and then presenting the results in language users can understand. The Datamart, Semantic View, and data governance process should still determine how revenue is calculated, which statuses are excluded, and which date represents the business event.

More precisely, the Semantic View constrains how structured data is understood; Cortex Agent selects and coordinates tools. Data quality and evaluation are still required to establish whether the final answer is correct.

The response will not improve automatically through the Semantic View alone. In the agent's response instructions, I require the answer to state the period, major exclusions, and metric definition without dumping the full SQL on the user. Cortex Agent supports separate orchestration instructions and response instructions: the former influence tool selection, while the latter control how the answer is presented. Configuring agent instructions

How Do You Evaluate SQL, Tool Selection, and the Final Answer?

During evaluation, I first determine whether a failure belongs to the semantic layer or the agent layer.

1. Question interpretation: Confirm which date field “last month” refers to, which status means “completed,” and whether “case count” requires deduplication. Executable SQL is irrelevant if the meaning is already wrong.
2. SQL and results: Inspect joins, filters, aggregation grain, double counting, dates, and statuses. Then reconcile values, trends, grouped results, and detail-level totals against manually written SQL or an existing report. This primarily evaluates the Semantic View and Cortex Analyst.
3. Tool selection and execution: Confirm that Cortex Agent calls the correct tool and that the tool's input and output match expectations. This is the acceptance scope for the agent's orchestration behavior.
4. Answer usability: A correct number is not enough. The response must state the period, essential definitions, and exclusions. When the data is insufficient, it should ask for more information or say explicitly that it cannot answer.

Cortex Agent Evaluations provide Answer Correctness and Logical Consistency. Tool Selection Accuracy and Tool Execution Accuracy are currently in Public Preview and can be used to check whether tool choice and execution match expectations. These metrics can gradually turn manual review into repeatable regression tests, but they cannot replace a data owner's confirmation of business definitions. Evaluation metrics

After MCP Goes Live, How Do You Keep Monitoring and Correcting It?

In my workflow, MCP comes only after the answers are stable. A Snowflake-managed MCP Server can package Cortex Analyst, Cortex Search, Cortex Agent, SQL, and UDFs or stored procedures as MCP tools for external MCP clients. If an external application only needs question answering over structured data, it can use Cortex Analyst directly through MCP without introducing a Cortex Agent first.

An MCP connector runs in the opposite direction: it brings tools from Jira, Salesforce, or another remote MCP server into Cortex Agent.

Snowflake-managed MCP Server
Snowflake capabilities → external AI client

MCP connector
External system tools → Snowflake Cortex Agent

MCP does not fix an incorrect metric or invent a missing business rule. Before exposing capabilities externally, I still need to verify OAuth, RBAC, and least-privilege access. Permission to use an MCP server does not automatically grant access to every tool. Third-party MCP servers and their tool descriptions also need review. Snowflake-managed MCP Server, MCP connectors

The build sequence I ultimately use is:

1. Document the Datamart's business logic, including metrics, dates, statuses, exclusions, and relationships.
2. Use an AI assistant to generate candidate questions, then validate demand through user interviews, reports, and query logs.
3. Create Ground Truth for core questions, recording the owner, confirmation date, applicable scope, and cost of a wrong answer.
4. Design the Semantic View from those questions, add verified queries, and evaluate Cortex Analyst first.
5. Decide whether a Cortex Agent is genuinely required. If it is, configure its tools, orchestration instructions, and response instructions.
6. Use an Evaluation Dataset to test original questions, paraphrases, boundary cases, and previous failures.
7. Put the artifacts under version control and deployment management, verify permissions, and only then integrate externally through MCP.

Git stores the Semantic View SQL or YAML, agent specification, source-table schema used to create the Evaluation Dataset, de-identified test seed, DDL, evaluation YAML, test code, and MCP configuration. The actual Evaluation Dataset is an object in Snowflake. Its version and environment must also appear in deployment records; committing configuration files to Git is not enough.

The workflow does not end after launch. New questions, failures, and user feedback from production should flow back into the question cards, verified queries, Semantic View, and Evaluation Dataset:

Business logic → questions and acceptance evidence → Semantic View
      ↑                                             ↓
Monitoring and user feedback ← Evaluation ← Cortex Agent / MCP

An agent can accelerate implementation, but it cannot decide metric definitions for the data team. The enduring work is to maintain what users need to ask, how a correct answer is defined, and who confirms that the answer remains valid after data or business rules change.

References

• Overview of Semantic Views
• Semantic View Autopilot
• Using SQL commands to create and manage Semantic Views
• Cortex Analyst Evaluations
• Routing Mode for Cortex Analyst
• Cortex Agents
• Create and manage Cortex Agents
• Cortex Agent Evaluations
• Snowflake-managed MCP Server
• MCP Connectors