Prompting for formatted LLM output: instructions, few-shot examples and structured outputs

1.Start with the prompt: state the format you want

There are two ways to get a formatted response from an LLM. You can ask for the format in the prompt, or you can enforce it on the request. This lesson covers the prompt first, because every other technique builds on it. Even when you enforce a JSON schema later on, the instructions in the prompt still tell the model what goes into each field.

The MLflow Prompt Registry docs show the difference with two versions of the same summarization prompt. Version 1 is a single line: Summarize this text: {{content}}. It sets no length, structure or tone, so the model picks its own. Version 2 adds a role, a hard length requirement that it states twice, and a bulleted list of guidelines. It also ends with a Summary: cue, so the response begins right where the format begins.

prompt_v2 = mlflow.genai.register_prompt(
    name=PROMPT_NAME,
    template="""You are an expert summarizer. Create a summary of the following content in *exactly* 2 sentences (no more, no less - be very careful about the number of sentences).

Guidelines:
- Include ALL core facts and key findings
- Use clear, concise language
- Maintain factual accuracy
- Cover all main points mentioned
- Write for a general audience
- Use exactly 2 sentences

Content: {{content}}

Summary:""",
    commit_message="v2: Added comprehensive fact coverage with 2-sentence requirement"
)

The format can also be a template variable. Another registry example asks the model to condense content "into exactly {{ num_sentences }} clear and informative sentences", so one registered template can serve callers who need different lengths. Databricks' guidance for Genie Code follows the same rule: say how much detail you want and what structure the answer should have, for example "Provide instructions in numbered steps".

A chat request also has a system message, which is the natural place for the role and the conversion task. The structured-outputs examples put the extraction role in the system message and the raw text in the user message:

messages = [{
        "role": "system",
        "content": "You are an expert at structured data extraction. You will be given unstructured text from a research paper and should convert it into the given structure."
      },
      {
        "role": "user",
        "content": "..."
      }]

Describing a format in words has limits. Few-shot examples show the format instead. Databricks' RAG quality guidance recommends putting well-formed example queries and their ideal responses inside the prompt template, because the examples teach the model the format, style and content you expect. Choose examples from the query types your application struggles with, and make them representative of real traffic.

Sources123

2.Enforce JSON with response_format

Prompt instructions make the format likely. They do not make it certain. When a program parses the output, as in batch extraction, a data pipeline or any downstream code, you need machine-readable JSON every time. That is the job of structured outputs. You set a response_format field on the chat request, and the request looks the same whichever provider serves the model. Databricks translates it into each provider's own mechanism, so you never write a provider-specific format. Structured outputs work on Foundation Model APIs pay-per-token and provisioned throughput endpoints.

Databricks recommends structured outputs for three kinds of work: extracting data from large numbers of documents (for example, labelling product reviews as negative, positive or neutral), batch inference that needs outputs in a specified format, and turning unstructured data into structured data.

You can choose between three output modes:

In the json_object example from the docs, the prompt and the request setting do separate jobs. The user message names the fields and asks for "a JSON object", and wraps the input in <description> tags so the model can tell instructions from data. The response_format then guarantees the result is JSON.

response_format = {
      "type": "json_object",
    }

messages = [
      {
        "role": "user",
        "content": "Extract the name, size, price, and color from this product description as a JSON object:\n<description>\nThe SmartHome Mini is a compact smart home assistant available in black or white for only $49.99. It's 5 inches wide.\n</description>"
      }]

When you know the fields in advance, use json_schema. You give the schema a name (the docs use research_paper_extraction), describe it as an object with typed properties, and set "strict": True. To run the same request on a Claude model, the docs change only the model value, from databricks-gpt-oss-20b to databricks-claude-sonnet-4-5.

Sources4

3.Design a schema the model can actually follow

Foundation Model APIs broadly accept the structured outputs format that OpenAI accepts. They deliberately support only part of JSON Schema, though, because simpler schemas produce better JSON. Plan around these limits when you design the schema:

- Not supported: regular expressions using pattern; composition and validation using anyOf, oneOf, allOf, prefixItems or $ref; and lists of types, except for the special case [type, "null"]. - Not enforced: length and size keywords such as maxProperties, minProperties and maxLength. - Capped: a schema can specify at most 64 keys. - Quality: deeply nested schemas produce worse output, so flatten the schema where you can.

Structured outputs also have a cost. Databricks uses prompt injection and other techniques to improve them, which changes how many input and output tokens you are billed for.

Claude models have further limits. They support only json_schema, not json_object; for unconstrained output you leave response_format out. Structured outputs do not work with streaming, so stream must be false. And on Claude, response_format cannot be combined with tools or tool_choice.

Sources4

4.The same enforcement from SQL, and when not to use function calling

In batch work the prompt often runs from SQL. Databricks recommends starting with a task-specific AI Function when one fits your goal. Use ai_query when you need more precise control of the prompt, the model parameters or the output format. ai_query takes a responseFormat argument that enforces a schema so downstream processing gets predictable output. You can write that argument as a JSON schema like the one above, or more compactly as a DDL-style type string:

SELECT ai_query(
    "system.ai.gpt-oss-20b",
    "Extract research paper details from the following abstract: " || abstract,
     ?  => 'STRUCT<research_paper_extraction:STRUCT<title:STRING, authors:ARRAY<STRING>, abstract:STRING, keywords:ARRAY<STRING>>>'
)
FROM research_papers;

With either form, the docs show output like { "title": "Understanding AI Functions in Databricks", "authors": ["Alice Smith", "Bob Jones"], ... }. That is one JSON object per row, ready to parse.

Function calling also produces structured JSON, and this confuses some candidates. With function calling, you describe functions in the tools parameter, and each function's arguments are a JSON schema. The model does not run those functions. It returns a JSON object of arguments, and your code calls the function. Function calling is for a model that has to decide whether to use an API, such as get_customers(min_revenue: int, created_before: string, limit: int). For batch inference or for turning unstructured data into structured data, Databricks recommends structured outputs instead.

Sources567

5.Measure format compliance across prompt versions

A schema can be enforced by the API. A rule like "exactly 2 sentences" or "answer in Markdown" cannot, so you measure it. The prompt-evaluation tutorial registers both summary prompts, runs each version against an evaluation dataset, and scores the outputs with two scorers: the built-in Correctness scorer and a custom judge that checks only the format rule.

sentence_count_judge = make_judge(
    name="sentence_count_compliance",
    instructions="""Evaluate if this summary follows the 2-sentence requirement.

Summary: {{ outputs }}

Count the sentences carefully. Return true if the summary has exactly 2 sentences, and false otherwise.""",
    feedback_value_type=bool,
)

MLflow averages boolean feedback, so each prompt version ends up with a pass rate (sentence_count_compliance/mean) that you can compare with its correctness score. A related notebook takes this a step further. It starts with the prompt Answer this question: {{question}}, which does not produce Markdown, and uses a judge that rates Markdown quality as high, medium or low. Those ratings become scores of 1.0, 0.5 and 0.0, and a prompt optimizer uses them to rewrite the prompt until the output follows the format.

Sources89

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Putting maxLength or minProperties in the JSON schema makes the model's output respect those limits.Why is that wrong?

   Foundation Model APIs do not enforce length or size keywords. Put such limits in the prompt and check them afterwards.

   Covered in Design a schema the model can actually follow

2. 2.Because Databricks translates response_format for every provider, a Claude model can stream structured output and use tools in the same request.Why is that wrong?

   On Claude, structured outputs need stream set to false, and response_format cannot be combined with tools or tool_choice.

   Covered in Design a schema the model can actually follow

3. 3.Function calling is the best way to turn a large batch of unstructured documents into structured JSON.Why is that wrong?

   Function calling is for a model that decides whether to call your APIs. For batch inference and converting unstructured data into structured data, Databricks recommends structured outputs.

   Covered in The same enforcement from SQL, and when not to use function calling

4. 4.A richer schema, with anyOf branches, $ref reuse and deep nesting, gives the model more guidance and produces better JSON.Why is that wrong?

   anyOf, oneOf, allOf, prefixItems and $ref are not supported, and heavily nested schemas lower output quality. Flatten the schema where you can.

   Covered in Design a schema the model can actually follow

Practise it for real

Run ai_query from SQL and enforce an output schema on the result, first as a DDL type string and then as a JSON schema

1. 1.On a SQL warehouse or cluster (not a Classic SQL warehouse) running Databricks Runtime 15.4 LTS or above, run the documented ai_query against system.ai.gpt-oss-20b with responseFormat => 'STRUCT<research_paper_extraction:STRUCT<title:STRING, authors:ARRAY<STRING>, abstract:STRING, keywords:ARRAY<STRING>>>' over a table that has an abstract column.

   Why: The DDL form is the most compact way to declare an output schema from SQL.

   You should see: One JSON string per row containing title, authors, abstract and keywords, like the example output in the ai_query docs.

2. 2.Replace the DDL string with the documented JSON schema responseFormat: type json_schema, name research_paper_extraction, the same four properties, and strict true.

   Why: This is the same schema in the json_schema form that the chat API uses, so you can see the two forms are interchangeable.

   You should see: The output has the same shape as in step 1.

3. 3.Add failOnError => false to the call.

   Why: On large workloads, one failed row should not stop the whole query.

   You should see: The query finishes and returns error messages for failed rows next to the successful results.