CertSafari

    Free Databricks Certified Generative AI Engineer Associate Sample Questions

    35 free sample questions from our bank of 373+, covering every exam domain, with answers and detailed explanations. Updated August 2026.

    Practice all 373 questions in quiz mode

    Domain 1: Design Applications

    1.2 Select model tasks to accomplish a given business requirement

    1.Which model task is primarily designed to convert textual data into a dense, numerical vector representation, which is useful for tasks like semantic search and clustering?

    1. A.Embedding
    2. B.Text Generation
    3. C.Tokenization
    4. D.Classification
    Show answer & explanation

    Correct answer: A — Embedding

    • A. Correct. Embedding is the process of converting textual data into dense, numerical vector representations. These vectors capture the semantic meaning of the text, making them ideal for downstream tasks like semantic search, clustering, and retrieval-augmented generation (RAG) where understanding context and similarity is crucial.
    • B. Incorrect. Text generation is the task of producing new, human-like text based on an input prompt. While it uses internal vector representations, its primary output is text, not the vector itself for use in other applications.
    • C. Incorrect. Tokenization is a crucial preprocessing step where text is broken down into smaller units called tokens (e.g., words or subwords). It is a necessary precursor to embedding but is not the process of creating the final dense numerical vector.
    • D. Incorrect. Classification is a supervised learning task that assigns a predefined label or category to input text. While classification models may use embeddings internally, their primary function is to predict a category, not to output a vector representation for general use.

    1.1 Design a prompt that elicits a specifically formatted response

    2.When a developer provides the beginning of the desired output in the prompt (e.g., `{"name": `) to encourage the model to complete it in the same format, what is this technique commonly called?

    1. A.Output parsing
    2. B.Priming the response
    3. C.Temperature scaling
    4. D.Prompt templating
    Show answer & explanation

    Correct answer: B — Priming the response

    • A. Incorrect. Output parsing is the process of extracting and structuring data from the model's generated response after it has been produced. It is a post-processing step, not a technique used within the prompt to guide the format of the generation itself.
    • B. Correct. This technique is known as priming the response. By providing the beginning of the desired output, the developer gives the model a strong cue to continue the pattern and complete the response in the specified format, such as completing a JSON object.
    • C. Incorrect. Temperature scaling is a model parameter that controls the randomness and creativity of the output. A lower temperature makes the output more deterministic, while a higher temperature increases diversity. It does not relate to guiding the format via prompt structure.
    • D. Incorrect. Prompt templating involves creating a reusable prompt structure, often with placeholders that are dynamically filled with different inputs. While it helps standardize prompts, it does not specifically describe the technique of starting the model's output for it.

    1.4 Translate business use case goals into a description of the desired inputs and outputs for the AI pipeline

    3.Which of the following is the most critical first step when translating a business requirement into the necessary inputs and outputs for an AI pipeline?

    1. A.Selecting the specific Large Language Model (LLM) based on its architecture and pre-training data to align with project scope.
    2. B.Establishing a clear, unambiguous definition of the business problem and the desired, measurable outcome.
    3. C.Provisioning the required cloud infrastructure and GPU clusters to support the anticipated data volume and model complexity.
    4. D.Collecting all available company data and cleaning it within a central data lake to create a single, unified source of truth.
    Show answer & explanation

    Correct answer: B — Establishing a clear, unambiguous definition of the business problem and the desired, measurable outcome.

    • A. Incorrect. Selecting a specific LLM based on architecture and pre-training data is a technical implementation detail that should occur later in the project lifecycle. Premature model selection without a clear business problem definition risks misalignment with actual needs.
    • B. Correct. Establishing a clear, unambiguous definition of the business problem and the desired, measurable outcome is the most critical first step. This foundation guides all subsequent decisions, including data selection, model choice, evaluation metrics, and infrastructure, ensuring the AI pipeline aligns with business goals.
    • C. Incorrect. Provisioning cloud infrastructure and GPU clusters is a necessary step for development and deployment, but it is not the first step. Infrastructure decisions depend on the defined problem, selected model, and expected workload, which are determined after the initial problem definition.
    • D. Incorrect. Collecting and cleaning all available company data in a central data lake is not the first step. Data collection must be guided by the well-defined business problem; gathering all data without a clear purpose is inefficient and may miss critical data points.

    1.3 Select chain components for a desired model input and output

    4.An application needs to process unstructured text from customer feedback emails and extract specific pieces of information: the customer's name, the product they are referencing, and a sentiment score from 1 to 5. The extracted data must be returned as a JSON object to be stored in a structured database. Which set of components is best suited for this structured data extraction task?

    1. A.A PromptTemplate that asks for a list of the name, product, and score, an LLM, and a CommaSeparatedListOutputParser.
    2. B.A VectorStoreRetriever to find similar emails for context, an LLM to extract the required fields, and a StrOutputParser.
    3. C.A PromptTemplate with instructions and formatting examples for JSON output, an LLM, and a JsonOutputParser.
    4. D.A ChatPromptTemplate instructing a ChatModel to extract the name, product, and score, combined with a simple StrOutputParser.
    Show answer & explanation

    Correct answer: C — A PromptTemplate with instructions and formatting examples for JSON output, an LLM, and a JsonOutputParser.

    • A. Incorrect. A `CommaSeparatedListOutputParser` is designed to parse a simple list of strings, not a structured JSON object with key-value pairs. This output format does not match the requirement to store the data with distinct fields for name, product, and sentiment score.
    • B. Incorrect. A `VectorStoreRetriever` is used for retrieval-augmented generation (RAG) to find relevant documents from a larger corpus, which is unnecessary for extracting information from a single, provided text. Furthermore, a `StrOutputParser` simply returns the model's raw string output, which does not guarantee a well-formed JSON structure.
    • C. Correct. This is the ideal approach for structured data extraction. A `PromptTemplate` that includes clear instructions and formatting examples for JSON output guides the LLM to generate output in the desired JSON format. The `JsonOutputParser` then validates and parses the model's string output, ensuring it is a valid JSON object ready for ingestion into a structured database.
    • D. Incorrect. Although a `ChatModel` can be prompted to produce JSON, relying on a simple `StrOutputParser` is not reliable. This parser does not validate or enforce the JSON structure, meaning the application could fail if the model's output deviates even slightly from the expected format.

    Domain 2: Data Preparation

    Subdomain 2.6: Use tools and metrics to evaluate retrieval performance

    5.A data scientist is debugging a RAG system's retriever. The evaluation shows a very high `context_precision` score but a very low `context_recall` score. The goal of the system is to find all documents pertaining to a specific project code. What issue do these metrics suggest?

    1. A.The retriever is returning many irrelevant documents in addition to the correct ones.
    2. B.The retriever is too restrictive, finding only a few highly relevant documents while missing many other relevant ones.
    3. C.The LLM is generating answers that are not relevant to the user's initial question.
    4. D.The document chunking strategy is creating chunks that are too small to be useful.
    Show answer & explanation

    Correct answer: B — The retriever is too restrictive, finding only a few highly relevant documents while missing many other relevant ones.

    • A. Incorrect. A high `context_precision` score indicates that the documents that are retrieved are highly relevant to the query. Returning many irrelevant documents would result in a low `context_precision` score.
    • B. Correct. This scenario perfectly describes the combination of high `context_precision` and low `context_recall`. High precision means the documents that were retrieved are indeed relevant. Low recall means that many other relevant documents that exist in the knowledge base were not retrieved. This indicates the retriever is too restrictive, likely due to a high similarity threshold or a query that is too narrow, causing it to find a small set of correct documents while missing the larger set of all relevant documents.
    • C. Incorrect. The metrics `context_precision` and `context_recall` are used to evaluate the performance of the retriever component of a RAG system, not the generator (LLM) component. These metrics measure the quality of the context provided to the LLM, not the quality of the LLM's final generated answer.
    • D. Incorrect. While the document chunking strategy can significantly impact retrieval performance, this specific pattern of high precision and low recall points more directly to the retrieval algorithm's behavior. A chunking issue might cause a variety of problems, including low scores on both metrics, but it doesn't uniquely explain why the retriever is accurate but not comprehensive.

    2.5 Identify needed source documents that provide necessary knowledge and quality for a given RAG application

    6.Why is a collection of structured documents, such as JSON files or well-formatted Markdown, generally preferred over a set of scanned, image-based PDFs for a RAG application's knowledge base?

    1. A.Scanned PDFs are universally smaller in file size, which speeds up processing.
    2. B.Structured formats enable more reliable text extraction and preservation of metadata, leading to better chunking and retrieval.
    3. C.Image-based PDFs can be directly indexed by vector databases without an Optical Character Recognition (OCR) step.
    4. D.JSON and Markdown files are inherently more secure than PDF files.
    Show answer & explanation

    Correct answer: B — Structured formats enable more reliable text extraction and preservation of metadata, leading to better chunking and retrieval.

    • A. Incorrect. Scanned, image-based PDFs are typically much larger in file size than their text-based counterparts (like JSON or Markdown) because they store pixel data for the entire page image. This larger size generally slows down, rather than speeds up, processing and ingestion.
    • B. Correct. Structured formats like JSON and Markdown allow for near-perfect, reliable text extraction. They also preserve inherent structure and metadata (e.g., keys in JSON, headings in Markdown), which can be leveraged to create more contextually-aware and effective data chunks. This leads to higher quality embeddings and significantly better retrieval performance in a RAG system.
    • C. Incorrect. This is the opposite of how the process works. Image-based PDFs contain images of text, not machine-readable text. They absolutely require an Optical Character Recognition (OCR) step to convert the images into text before they can be processed, chunked, and indexed by a vector database. This OCR process adds complexity and is a common source of errors.
    • D. Incorrect. While certain file formats can have different security considerations, security is primarily determined by system-level controls, access policies, and encryption, not the file format itself. The preference for structured text in a RAG context is overwhelmingly based on data quality, reliability, and processing efficiency, not an inherent security advantage.

    2.4 Define operations and sequence to write given chunked text into Delta Lake tables in Unity Catalog

    7.When writing a DataFrame of chunked text to a Delta Lake table using a three-level namespace (e.g., `catalog.schema.table`), what is the primary role of Unity Catalog in this operation?

    1. A.To chunk the raw text files before they are converted into a DataFrame.
    2. B.To provide a centralized governance, metastore, and access control layer for the table being created.
    3. C.To automatically generate vector embeddings for the text chunks during the write operation.
    4. D.To optimize the physical storage layout of the Parquet files on cloud storage.
    Show answer & explanation

    Correct answer: B — To provide a centralized governance, metastore, and access control layer for the table being created.

    • A. Incorrect. Unity Catalog does not perform data transformation or processing tasks like chunking. Chunking is a pre-processing step handled by application code (e.g., using libraries like LangChain or spaCy) before the data is written to a DataFrame.
    • B. Correct. This is the primary function of Unity Catalog. When a DataFrame is written using the three-level namespace, Unity Catalog acts as the central metastore to manage the table's schema and location. It also enforces governance policies, provides fine-grained access control (via `GRANT`/`REVOKE`), and captures data lineage for the operation.
    • C. Incorrect. Generating vector embeddings is a computational, ML-related task performed using embedding models. This operation is part of the data preparation or feature engineering pipeline and is entirely outside the scope of Unity Catalog's metadata and governance responsibilities.
    • D. Incorrect. The optimization of the physical storage layout (e.g., compaction of Parquet files, Z-Ordering) is a core feature of the Delta Lake storage format and the Databricks runtime, not Unity Catalog. Unity Catalog manages the metadata about the table, not the physical arrangement of its underlying data files.

    Domain 3: Application Development

    3.11: Utilize MLflow and Agent Framework for developing agentic systems

    8.An engineer needs to build an application that first summarizes a long article, then extracts key entities from that summary, and finally generates a social media post based on the extracted entities. The output of each step must be the input for the next. Which agent framework concept is best suited for creating this specific, ordered workflow?

    1. A.A ReAct Agent, which can decide on its own which tool to use next.
    2. B.A Sequential Chain, which orchestrates a series of calls in a predefined order.
    3. C.A custom Tool that contains the logic for all three steps in a single function.
    4. D.A Vector Store Retriever that fetches relevant documents for each step.
    Show answer & explanation

    Correct answer: B — A Sequential Chain, which orchestrates a series of calls in a predefined order.

    • A. Incorrect. A ReAct (Reasoning and Acting) Agent is designed for dynamic decision-making. It uses a loop of reasoning and acting to decide which tool to use next based on the current context and goal. This is ideal for complex, unpredictable tasks, but not for a simple, strictly defined, linear workflow as described in the question.
    • B. Correct. A Sequential Chain is specifically designed to execute a series of components (like LLM calls or functions) in a fixed, predetermined order. Crucially, it automatically passes the output of one step as the input to the subsequent step, which perfectly matches the requirements of summarizing, then extracting, then generating.
    • C. Incorrect. While technically possible, creating a single monolithic tool for all three steps goes against the principles of agent framework design, which favor modularity, reusability, and clarity. This approach would be less flexible and harder to maintain compared to orchestrating discrete components with a chain.
    • D. Incorrect. A Vector Store Retriever's purpose is to fetch relevant documents or data chunks from a vector database based on a query. It is a component used for information retrieval (like in RAG), not for orchestrating a sequence of processing tasks.

    3.17 Select Langchain/similar tools for use in a Generative AI application

    9.A developer is building a customer service chatbot that needs to recall the user's name and previous questions within the same conversation to provide a more personalized and coherent experience. Which LangChain component should be integrated into the application to achieve this stateful behavior?

    1. A.Output Parsers
    2. B.Memory
    3. C.Document Loaders
    4. D.Prompt Templates
    Show answer & explanation

    Correct answer: B — Memory

    • A. Incorrect. Output Parsers are responsible for structuring the raw text output from a language model into a more usable format (e.g., JSON, a list, or a custom object). They do not manage or store the history or state of a conversation.
    • B. Correct. Memory components in LangChain are specifically designed to maintain the state of a conversation. They store and recall previous interactions, such as user questions and assistant responses, enabling the application to have context-aware, personalized, and coherent conversations over multiple turns.
    • C. Incorrect. Document Loaders are used for ingesting data from various sources (like text files, PDFs, or web pages) into a format that can be processed by other LangChain components, typically for Retrieval-Augmented Generation (RAG). They do not manage the dynamic state of a conversation.
    • D. Incorrect. Prompt Templates are used to create reusable, structured prompts for language models. While conversation history from a Memory component is often inserted into a prompt template, the template itself is a stateless component and is not responsible for storing or recalling that history.

    3.3: Select chunking strategy based on model & retrieval evaluation

    10.A developer is building a chatbot to answer questions from a company's internal knowledge base, which consists of well-structured markdown documents with clear headings, subheadings, and code blocks. The goal is to retrieve sections that are self-contained and contextually complete. Which chunking strategy is best suited for this type of document structure?

    1. A.A fixed-size chunking strategy that ignores all markdown formatting.
    2. B.A recursive character splitting strategy that uses markdown headers and paragraph breaks as primary separators.
    3. C.A semantic chunking strategy that relies solely on embedding similarity, ignoring the document's explicit structure.
    4. D.Chunking each document into a single large chunk to preserve all information at once.
    Show answer & explanation

    Correct answer: B — A recursive character splitting strategy that uses markdown headers and paragraph breaks as primary separators.

    • A. Incorrect. A fixed-size chunking strategy completely ignores the logical structure provided by markdown formatting. This will arbitrarily split content across important semantic boundaries like headings, paragraphs, and code blocks, resulting in chunks that are not contextually complete and reducing retrieval effectiveness.
    • B. Correct. This strategy is ideal for well-structured documents like markdown. It leverages the inherent logical structure (headers, subheadings, paragraphs) as separators to create chunks that are naturally self-contained and contextually complete. This alignment with the document's structure significantly improves the relevance and accuracy of the retrieval system.
    • C. Incorrect. While semantic chunking can be powerful, relying solely on embedding similarity and ignoring the document's explicit structure is suboptimal in this case. The markdown formatting provides clear, human-defined contextual boundaries that this approach would overlook, potentially leading to less coherent chunks.
    • D. Incorrect. Treating an entire document as a single chunk is highly inefficient for a RAG system. This approach lacks the necessary granularity to pinpoint specific information for the chatbot. Furthermore, large chunks can exceed the model's context window or overwhelm it with irrelevant information, leading to poor-quality answers.

    3.7: Select the best LLM based on the attributes of the application to be developed

    11.A startup is creating a mobile application that provides real-time language translation. The application must perform translations directly on the user's device without a constant internet connection. Due to the hardware limitations of mobile devices, which model characteristics should be prioritized during selection?

    1. A.Large parameter count and maximum reasoning ability.
    2. B.Support for multi-turn conversations and long context windows.
    3. C.The model's reputation and popularity on leaderboards.
    4. D.Small memory footprint and fast inference speed.
    Show answer & explanation

    Correct answer: D — Small memory footprint and fast inference speed.

    • A. Incorrect. Models with a large parameter count and high reasoning ability require significant computational resources and memory. These characteristics are unsuitable for on-device applications on mobile phones, which have limited hardware capabilities.
    • B. Incorrect. Multi-turn conversation support and long context windows are features designed for chatbot-like applications. For real-time, single-instance translation, these features are not a priority and would consume unnecessary memory and processing power, hindering performance.
    • C. Incorrect. A model's popularity or high ranking on general leaderboards does not guarantee its suitability for a specific, resource-constrained use case. Practical characteristics related to on-device performance are far more important than general benchmarks.
    • D. Correct. For an application running on a mobile device without an internet connection, efficiency is paramount. A small memory footprint ensures the model can run within the device's limited RAM, while fast inference speed is crucial for providing the 'real-time' translation experience required by the user.

    3.10: Select the best model for a given task based on common metrics generated in experiments

    12.A development team is building a model to translate technical user manuals from English to Spanish. Which of the following metrics is the standard and most widely used for automatically evaluating the quality of this machine translation task by comparing the generated translation to one or more reference translations?

    1. A.ROUGE
    2. B.BLEU
    3. C.F1-Score
    4. D.Toxicity
    Show answer & explanation

    Correct answer: B — BLEU

    • A. Incorrect. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is primarily used for evaluating automatic summarization tasks. While it also compares n-gram overlap, its recall-oriented nature makes it better suited for tasks where capturing all key information from a source is important, which is different from the goal of machine translation.
    • B. Correct. BLEU (Bilingual Evaluation Understudy) is the standard and most widely used metric for evaluating the quality of machine translation. It works by measuring the precision of n-grams (contiguous sequences of n items) in the machine-generated translation against one or more high-quality reference translations.
    • C. Incorrect. The F1-Score is a common metric for binary and multi-class classification tasks. It calculates the harmonic mean of precision and recall, providing a single score that balances both. It is not designed to evaluate the quality or fluency of generated text like a translation.
    • D. Incorrect. Toxicity is a metric used in content moderation to evaluate the presence of harmful, offensive, or otherwise inappropriate content in a generated text. It does not measure the accuracy, fluency, or quality of a translation from one language to another.

    Subdomain 3.12: Compare the evaluation and monitoring phases of the Gen AI application life cycle

    13.A GenAI engineer is setting up the monitoring phase for a newly deployed summarization tool. They need to select metrics and signals that are exclusively or primarily collected during the monitoring phase, rather than the pre-deployment evaluation phase. Which two signals are primarily collected during the monitoring phase of the Gen AI application life cycle?(Select 2)

    1. A.Exact match accuracy against a static ground-truth dataset.
    2. B.Explicit user feedback (e.g., thumbs up / thumbs down clicks).
    3. C.LLM-as-a-judge grading on a curated golden dataset.
    4. D.Production endpoint latency and request volume.
    5. E.ROUGE-L scores calculated on a holdout validation set.
    Show answer & explanation

    Correct answers: B, D — Explicit user feedback (e.g., thumbs up / thumbs down clicks).; Production endpoint latency and request volume.

    • A. Incorrect. Exact match accuracy against a static ground-truth dataset is a classic pre-deployment evaluation metric. It is computed offline on labeled test or validation data to assess a model's performance baseline before it is deployed.
    • B. Correct. Explicit user feedback, such as thumbs up or thumbs down clicks, is primarily collected during the monitoring phase. This signal relies on real users interacting with the deployed model in a live environment to understand real-world satisfaction and model performance.
    • C. Incorrect. LLM-as-a-judge grading on a curated golden dataset is typically a pre-deployment evaluation technique. It is used to evaluate the performance of a candidate model against high-quality reference examples in a controlled, offline setting.
    • D. Correct. Production endpoint latency and request volume are operational telemetry signals exclusively gathered in the monitoring phase. These metrics provide insights into the health, scalability, and technical performance of the system once it is live.
    • E. Incorrect. ROUGE-L scores calculated on holdout validation sets are standard offline evaluation metrics for summarization. They are used to compare model-generated summaries against reference summaries during the model development and selection phases.

    3.15 Create a prompt that adjusts an LLM's response from a baseline to a desired output

    14.A developer needs to extract structured information from unstructured customer reviews. The initial prompt `Summarize this review:` returns a free-text paragraph. The developer needs the output in a machine-readable format to load into a database. Which prompt is best designed to adjust the LLM's response to the desired structured format?

    1. A.Please organize the summary of this review into clear sections, such as product details, customer sentiment, and key issues, and then output the result as a structured JSON object with keys like 'product_name', 'sentiment'.
    2. B.What is the overall sentiment of this review? Analyze the text to determine whether the customer's tone is positive, negative, or neutral, and then output the result as a JSON object with the key 'sentiment' containing the single label.
    3. C.From the following review, extract the product mentioned, the main issue, and the customer's location. Provide the output as a JSON object with the keys 'product_name', 'issue_description', and 'location'.
    4. D.Rewrite the summary of this review to be more concise and factual, then structure the output as a JSON object with keys like 'summary' and 'key_points', where 'key_points' is an array of the most important facts.
    Show answer & explanation

    Correct answer: C — From the following review, extract the product mentioned, the main issue, and the customer's location. Provide the output as a JSON object with the keys 'product_name', 'issue_description', and 'location'.

    • A. Incorrect. Although this prompt requests a structured JSON output, it still focuses on organizing a summary into sections rather than extracting specific, discrete fields. The keys like 'product_name' and 'sentiment' are mentioned, but the instruction to 'organize the summary' may lead to nested or free-text values instead of clean, machine-readable data suitable for direct database ingestion.
    • B. Incorrect. This prompt only asks for a single sentiment label, which is too narrow for extracting multiple structured fields from a review. While it does specify a JSON output, it fails to capture other important information like product name or issues, making it insufficient for a comprehensive structured extraction.
    • C. Correct. This prompt explicitly instructs the LLM to extract specific fields (product, issue, location) and output them as a JSON object with defined keys. This ensures the response is structured, predictable, and machine-readable, ideal for loading directly into a database.
    • D. Incorrect. This prompt focuses on rewriting the summary to be more concise and factual, but it does not shift the output from a free-text paragraph to a truly structured format. Even though it requests a JSON object, the values are still narrative text ('summary' and 'key_points'), which may not be as directly machine-parseable as discrete field extractions.

    3.6: Implement LLM guardrails to prevent negative outcomes

    15.What is the primary purpose of an input guardrail in a large language model (LLM) application?

    1. A.To ensure the LLM's response is formatted correctly by validating its structure, such as for valid JSON or XML, after generation.
    2. B.To check user prompts for harmful content, prompt injections, or policy violations before they are sent to the LLM.
    3. C.To log the LLM's output for later analysis by storing each generated response and its corresponding user prompt in a data warehouse.
    4. D.To limit the number of tokens in the LLM's final response to control costs by setting a `max_tokens` parameter in the API call.
    Show answer & explanation

    Correct answer: B — To check user prompts for harmful content, prompt injections, or policy violations before they are sent to the LLM.

    • A. Incorrect. This describes an output guardrail or response validation step that occurs after the LLM generates text. Input guardrails focus on the user's prompt before it reaches the model, not on the structure of the final response.
    • B. Correct. The primary purpose of an input guardrail is to act as a preventative filter, scanning user prompts for harmful content, prompt injections, or policy violations before they are processed by the LLM.
    • C. Incorrect. Logging responses and prompts is a monitoring or auditing function, not an input guardrail. Input guardrails are active, real-time checks on incoming prompts, whereas logging is a passive data collection mechanism for later analysis.
    • D. Incorrect. Limiting response tokens via `max_tokens` is an output control for cost management, not an input guardrail. Input guardrails inspect the content of the user's prompt, not the length of the model's reply.

    3.9: Select a model from a model hub or marketplace for a task based on model metadata/model cards

    16.In a well-documented model card, what is the primary purpose of the 'Biases and Limitations' section?

    1. A.To provide the command-line interface for running model inference, including required arguments and output formats.
    2. B.To describe the model's performance metrics on standard benchmarks, detailing accuracy and F1 scores for key tasks.
    3. C.To inform users about potential ethical risks, unintended behaviors, and performance weaknesses.
    4. D.To list the hardware specifications required for training, such as the necessary GPU type and total VRAM capacity.
    Show answer & explanation

    Correct answer: C — To inform users about potential ethical risks, unintended behaviors, and performance weaknesses.

    • A. Incorrect. The 'Biases and Limitations' section is not intended for technical usage instructions. Details like command-line interfaces and output formats belong in sections such as 'Usage' or 'Quick Start'.
    • B. Incorrect. While performance metrics like accuracy and F1 scores are important, they are typically reported in an 'Evaluation' or 'Performance' section. The 'Biases and Limitations' section focuses on qualitative risks and weaknesses rather than benchmark results.
    • C. Correct. This section is a key part of responsible AI documentation. Its primary purpose is to transparently communicate ethical risks, unintended behaviors, and performance weaknesses so users can make informed decisions and apply appropriate safeguards.
    • D. Incorrect. Hardware specifications for training, such as GPU type and VRAM, are usually documented in a 'Training Details' or 'Technical Specifications' section. They are separate from the discussion of biases and limitations.

    Domain 4: Assembling and Deploying Applications

    4.4: Choose the basic elements needed to create a RAG application: model flavor, embedding model, retriever, dependencies, input examples, model signature

    17.An engineer is logging a RAG application to the MLflow Model Registry. They want to ensure that the model's page in the Databricks UI includes a sample query in the correct format, making it easier for colleagues to test the endpoint. Which argument should they include in the `mlflow.pyfunc.log_model()` call?

    1. A.signature
    2. B.input_example
    3. C.dependencies
    4. D.retriever
    Show answer & explanation

    Correct answer: B — input_example

    • A. Incorrect. The `signature` argument defines the schema (data types, names, and shapes) of the model's inputs and outputs. It is used for validation and inference enforcement, but it does not provide a concrete, testable example query in the UI.
    • B. Correct. The `input_example` argument is specifically designed to log a sample input (e.g., a Pandas DataFrame or a dictionary) along with the model. This example is then displayed in the Databricks Model Registry UI, allowing users to easily understand the expected input format and test the model endpoint directly.
    • C. Incorrect. The `dependencies` argument is used to specify the external libraries and packages required for the model's environment to run correctly, such as those listed in a `conda.yaml` or `requirements.txt` file. It does not relate to sample input data.
    • D. Incorrect. A `retriever` is a fundamental component of a RAG application responsible for fetching relevant documents. However, it is part of the model's internal logic, not a standard argument for the generic `mlflow.pyfunc.log_model()` function.

    4.2 Control access to resources from model serving endpoints

    18.A financial services company is deploying a fraud detection model on a Databricks model serving endpoint. This model needs to query an external risk assessment API that requires a highly sensitive API key. What is the most secure and recommended method for providing this API key to the model?

    1. A.Hardcode the API key directly into the model's prediction script before logging it to MLflow.
    2. B.Store the API key in a Databricks secret scope and configure the endpoint to expose it as an environment variable.
    3. C.Pass the API key as a plain-text environment variable in the model serving endpoint configuration.
    4. D.Store the API key in a Unity Catalog table and grant the endpoint's service principal SELECT access to that table.
    Show answer & explanation

    Correct answer: B — Store the API key in a Databricks secret scope and configure the endpoint to expose it as an environment variable.

    • A. Incorrect. Hardcoding sensitive credentials like an API key directly into source code is a major security anti-pattern. It exposes the key in plain text within the code artifact, making it vulnerable to exposure in code repositories, logs, or to anyone with access to the code.
    • B. Correct. This is the most secure and recommended method on Databricks. Databricks secret scopes are specifically designed to securely store and manage sensitive information like API keys, providing encryption and fine-grained access control. Configuring the model serving endpoint to securely fetch the secret and expose it as an environment variable to the model container keeps the credential out of the model's code and logs, minimizing the risk of exposure.
    • C. Incorrect. While this approach decouples the key from the code, passing it as a plain-text environment variable in the endpoint configuration is still insecure. The plain-text value can be exposed in the UI, configuration files, or logs, and it lacks the encryption and robust access control features provided by a dedicated secrets management system like Databricks secret scopes.
    • D. Incorrect. While Unity Catalog provides strong access control for data assets, it is not designed or recommended for managing secrets like API keys. Using a table for this purpose is an anti-pattern. A dedicated secrets management tool, like Databricks secret scopes, is the appropriate and more secure choice for handling sensitive credentials.

    4.6: Create and query a Vector Search index

    19.An engineering team is designing a Vector Search index for a table containing product descriptions. The source Delta table has the columns `product_id`, `description`, `embedding_vector`, and `category`. When creating the index, which column must be designated as the primary key?

    1. A.`description`, because it contains the source text.
    2. B.`embedding_vector`, because it is the column being indexed.
    3. C.`category`, because it will be used for filtering.
    4. D.`product_id`, because it uniquely identifies each row.
    Show answer & explanation

    Correct answer: D — `product_id`, because it uniquely identifies each row.

    • A. Incorrect. While the `description` column contains the source text for the embeddings, it is not suitable as a primary key. A primary key must contain a unique value for each record, and text descriptions are highly unlikely to be unique across all products.
    • B. Incorrect. The `embedding_vector` column contains the vector data that is indexed for similarity search. However, the primary key's function is to uniquely identify the row, not to be the data that is searched. The system uses the primary key to link a found vector back to its original record.
    • C. Incorrect. The `category` column is a classification attribute and will contain duplicate values for products within the same category. A primary key requires unique values for every row, making `category` an unsuitable choice.
    • D. Correct. When creating a Databricks Vector Search index, a primary key column must be specified. This column must uniquely identify each row in the source Delta table. The `product_id` column is explicitly designed for this purpose, serving as the unique identifier for each product and its corresponding row.

    4.8: Explain the key concepts and components of Mosaic AI Vector Search

    20.A company is building a chatbot to answer questions based on its internal knowledge base, which is updated several times a day. The engineering team has a Delta Table that is continuously updated with new document chunks and their embeddings. To ensure the chatbot always provides answers based on the latest information with minimal operational overhead, which Vector Search Index configuration should they choose?

    1. A.A Direct Vector Access Index with a nightly batch job to upsert new vectors.
    2. B.A Delta Sync Index pointing to the source Delta Table.
    3. C.A new Vector Search Endpoint for each version of the knowledge base.
    4. D.An in-memory vector store managed within a Databricks Notebook.
    Show answer & explanation

    Correct answer: B — A Delta Sync Index pointing to the source Delta Table.

    • A. Incorrect. A Direct Vector Access Index requires manual updates via API calls. Relying on a nightly batch job introduces significant latency (up to 24 hours), which fails the requirement to provide answers based on the latest information from a source that is updated several times a day.
    • B. Correct. A Delta Sync Index is specifically designed for this use case. It automatically and incrementally synchronizes with a source Delta Table, ensuring that any changes (inserts, updates, deletes) in the table are reflected in the index with low latency. This provides the most up-to-date information with minimal operational overhead.
    • C. Incorrect. Creating a new Vector Search Endpoint for each update is operationally complex, inefficient, and resource-intensive. This approach would create a significant management burden and is contrary to the requirement for minimal operational overhead.
    • D. Incorrect. An in-memory vector store within a notebook is suitable for prototyping or small-scale experiments but is not a robust solution for a production environment. It lacks the persistence, scalability, and availability required to serve a production chatbot with continuously updated data.

    4.12 Create and query a Vector Search index

    21.A financial services company ingests and processes thousands of articles daily, generating embeddings and storing them in a Delta table. They need to build a RAG application that provides the most up-to-date information to their analysts with minimal latency. The Vector Search index must reflect new articles within minutes of their arrival in the Delta table. Which index type and sync mode should be used to meet this requirement?

    1. A.Direct Vector Access index with a TRIGGERED sync mode.
    2. B.Delta Sync index with a CONTINUOUS sync mode.
    3. C.Direct Vector Access index with a CONTINUOUS sync mode.
    4. D.Delta Sync index with a TRIGGERED sync mode.
    Show answer & explanation

    Correct answer: B — Delta Sync index with a CONTINUOUS sync mode.

    • A. Incorrect. A Direct Vector Access index is not designed for automatic synchronization with a source Delta table. Furthermore, the TRIGGERED sync mode requires manual intervention or a fixed schedule to update, which would introduce significant latency and fail to meet the requirement of updating within minutes.
    • B. Correct. This combination is specifically designed for this use case. A Delta Sync index automatically synchronizes with a source Delta table. The CONTINUOUS sync mode ensures that the index is incrementally updated as new data arrives in the table, typically reflecting changes within a few minutes. This provides the minimal latency and up-to-date data required for the RAG application.
    • C. Incorrect. This is not a valid configuration. Direct Vector Access indexes do not support the CONTINUOUS sync mode as they are intended for direct querying of embeddings, not for automated, continuous synchronization from a source Delta table.
    • D. Incorrect. While the Delta Sync index type is appropriate for syncing from a Delta table, the TRIGGERED sync mode is not suitable for near real-time requirements. It updates the index only when a sync job is explicitly triggered, either manually or on a schedule, which would cause delays that violate the 'within minutes' latency requirement.

    Subdomain 4.11: Configure a persistent datastore to store and retrieve intermediate memory or structured information

    22.In the context of Databricks Generative AI applications, what is the primary function of Databricks Vector Search when configured as a persistent datastore?

    1. A.To store and retrieve vector embeddings and metadata for semantic similarity searches.
    2. B.To provide a relational database for transactional processing of user chat logs.
    3. C.To cache intermediate LLM API responses to reduce token costs.
    4. D.To store the weights and biases of fine-tuned Large Language Models.
    Show answer & explanation

    Correct answer: A — To store and retrieve vector embeddings and metadata for semantic similarity searches.

    • A. Correct. Databricks Vector Search is a serverless similarity search engine designed to store and retrieve vector embeddings along with associated metadata. This enables semantic similarity search and retrieval-augmented generation (RAG) use cases by facilitating efficient nearest-neighbor lookups.
    • B. Incorrect. Relational database workloads and transactional processing of logs are better suited for Delta Lake or traditional SQL databases. Vector Search is optimized for similarity metrics rather than relational operations.
    • C. Incorrect. Caching LLM responses to reduce costs is typically handled by a specific application cache or a model gateway caching layer, not a vector search engine, which is used for retrieving context based on embeddings.
    • D. Incorrect. Fine-tuned model parameters such as weights and biases are stored in a model registry (like MLflow) or dedicated model storage, not in a vector search index.

    Subdomain 4.13: Integrate managed, external, and custom MCP servers based on a given application requirements

    23.What is the primary purpose of integrating a Model Context Protocol (MCP) server into a Generative AI application architecture?

    1. A.To fine-tune Large Language Models on proprietary datasets using distributed computing clusters.
    2. B.To provide a standardized, secure way for AI models to connect to external data sources, tools, and prompts.
    3. C.To compress the context window of an LLM to reduce token costs during inference.
    4. D.To automatically translate natural language SQL queries into Python code for data analysis.
    Show answer & explanation

    Correct answer: B — To provide a standardized, secure way for AI models to connect to external data sources, tools, and prompts.

    • A. Incorrect. Fine-tuning Large Language Models on proprietary datasets is a model training activity typically handled by specialized frameworks (like PyTorch or Spark) and distributed computing clusters. MCP is used at application runtime for integration, not for model training.
    • B. Correct. The Model Context Protocol (MCP) is an open standard designed to provide a standardized, secure way for AI models to connect to external data sources, tools, and prompts. It helps decouple the model from specific integrations while enabling controlled and consistent access to context and actions across different platforms.
    • C. Incorrect. Compressing the context window or reducing token costs is an inference optimization task (such as KV cache compression or prompt summarization). MCP structures how external context is accessed but does not modify the model's underlying token budget or compression algorithms.
    • D. Incorrect. While MCP might facilitate access to tools that perform data analysis, the protocol itself is not an automatic code translation mechanism for SQL or Python. That functionality is a capability of the LLM or a specialized agent.

    4.3 Code a simple chain according to requirements

    24.What is the primary role of the pipe operator (`|`) in the LangChain Expression Language (LCEL)?

    1. A.It performs a bitwise OR operation to combine the numeric outputs of two components into a single integer value for subsequent processing in the chain.
    2. B.It executes two components in parallel, waits for both to finish, and then merges their dictionary outputs into a single object for the next component.
    3. C.It defines a sequential chain where the output of the component on the left is passed as the input to the component on the right.
    4. D.It is used to specify conditional logic for routing between components, evaluating a predicate on the input to select which downstream branch to execute.
    Show answer & explanation

    Correct answer: C — It defines a sequential chain where the output of the component on the left is passed as the input to the component on the right.

    • A. Incorrect. The pipe operator (`|`) in LCEL is overloaded for chaining components and does not perform a bitwise OR operation. Its function is specific to constructing `RunnableSequence` objects, not numeric operations.
    • B. Incorrect. The pipe operator creates a sequential execution flow, not a parallel one. To run components in parallel and merge their outputs, constructs like `RunnableParallel` or a dictionary of runnables are used.
    • C. Correct. This is the fundamental purpose of the pipe operator (`|`) in LCEL. It creates a `RunnableSequence` by linking components together, where the output of the preceding component (on the left) is automatically passed as the input to the succeeding component (on the right). This syntax allows for building intuitive and readable chains.
    • D. Incorrect. The pipe operator is used for creating linear, sequential chains. Conditional logic and routing require different constructs within LCEL, such as `RunnableBranch`, which can direct the flow based on the output of a previous step.

    4.5: Register the model to Unity Catalog using MLflow

    25.A data scientist executes the following code block in a Databricks notebook to train and register a sentiment analysis model. What is the primary outcome of the `mlflow.sklearn.log_model` call within this block? ```python import mlflow from sklearn.ensemble import RandomForestClassifier mlflow.set_registry_uri('databricks-uc') with mlflow.start_run() as run: rfc = RandomForestClassifier() # Assume model is trained on data here mlflow.sklearn.log_model( sk_model=rfc, artifact_path='sentiment-model', registered_model_name='prod.nlp.sentiment_classifier' ) ```

    1. A.The model is logged to the run's artifacts under `sentiment-model`, but the registration to `prod.nlp.sentiment_classifier` fails because it must occur after the run completes.
    2. B.The model is registered to the Workspace Model Registry because the `mlflow.set_registry_uri` call is ignored and a UC-specific function, like `mlflow.uc.log_model`, is required.
    3. C.The code creates or updates the `sentiment_classifier` model in Unity Catalog under the `prod.nlp` schema and logs the model artifacts to the MLflow run.
    4. D.The code will fail because `mlflow.sklearn.log_model` does not accept both `artifact_path` and `registered_model_name` when the registry URI is set to `databricks-uc`.
    Show answer & explanation

    Correct answer: C — The code creates or updates the `sentiment_classifier` model in Unity Catalog under the `prod.nlp` schema and logs the model artifacts to the MLflow run.

    • A. Incorrect. The `mlflow.sklearn.log_model` function can both log the model artifacts and register the model in the same call, even within an active run. Registration does not require the run to complete first; it happens immediately when `registered_model_name` is provided.
    • B. Incorrect. The `mlflow.set_registry_uri('databricks-uc')` call explicitly sets the registry to Unity Catalog, and it is not ignored. Standard MLflow functions like `mlflow.sklearn.log_model` respect this setting, so no UC-specific function is required.
    • C. Correct. With the registry URI set to `databricks-uc`, the `mlflow.sklearn.log_model` call logs the model artifacts under the `sentiment-model` path in the run and registers the model as `sentiment_classifier` in the `prod.nlp` schema of Unity Catalog, creating a new version.
    • D. Incorrect. `artifact_path` and `registered_model_name` serve different purposes and can be used together without conflict. The registry URI being `databricks-uc` does not prevent using both parameters; the function handles logging and registration seamlessly.

    4.7: Identify how to serve an LLM application that leverages Foundation Model APIs

    26.A development team wants to serve a third-party LLM, such as Anthropic's Claude 3, through Databricks Model Serving. They need to provide the API key securely to the endpoint configuration. What is the recommended and most secure method for managing the API key?

    1. A.Store the API key in a text file within a project directory on DBFS and have the model's serving logic read the file's contents.
    2. B.Hardcode the API key as a string value within the `model_config` dictionary when defining the external model serving endpoint.
    3. C.Store the API key in a Databricks secret scope and reference the secret in the external model configuration.
    4. D.Set the API key as a cluster-level environment variable through the cluster UI's Spark configuration for the serving endpoint.
    Show answer & explanation

    Correct answer: C — Store the API key in a Databricks secret scope and reference the secret in the external model configuration.

    • A. Incorrect. Storing the API key in a text file on DBFS is insecure because DBFS is not designed for secrets management and may lack strict access controls. Anyone with read access to the file can obtain the key, increasing the risk of exposure.
    • B. Incorrect. Hardcoding the API key in the `model_config` dictionary exposes it in plaintext within the configuration code. This practice can lead to leakage through version control, logs, or direct inspection, making it a significant security risk.
    • C. Correct. Databricks secret scopes are purpose-built for securely storing sensitive information like API keys, with encryption and fine-grained access controls. Referencing the secret in the external model configuration (e.g., `{{secrets/scope_name/key_name}}`) injects the key at runtime without exposing it in plaintext.
    • D. Incorrect. Setting the API key as a cluster-level environment variable is less secure than using secret scopes because the variable can be exposed to users or processes with access to the cluster configuration or Spark UI. It also lacks the granular access control and automatic redaction features of Databricks secrets.

    Subdomain 4.10: Configure vector search for a particular solution based on number of embeddings, update frequency, latency, and cost requirements

    27.A financial institution is deploying a RAG application for internal policy search. The policy documents are updated in a Delta table only once at the end of each month. The engineering team needs to configure the vector search index to minimize compute costs while ensuring the index is reliably updated after the monthly changes. How should they configure the index synchronization?

    1. A.Use a Delta Sync Index with `CONTINUOUS` sync mode to automatically detect and apply the monthly Delta table updates as soon as they are committed.
    2. B.Use a Delta Sync Index with `TRIGGERED` sync mode and schedule a time-based Databricks Workflow to call the sync API at a fixed monthly interval.
    3. C.Use a Direct Vector Access Index and schedule a monthly job to drop and rebuild the index from the updated Delta table.
    4. D.Use a Delta Sync Index with `TRIGGERED` sync mode and configure a task dependency in a Databricks Workflow to invoke the sync API upon successful completion of the monthly data refresh.
    Show answer & explanation

    Correct answer: D — Use a Delta Sync Index with `TRIGGERED` sync mode and configure a task dependency in a Databricks Workflow to invoke the sync API upon successful completion of the monthly data refresh.

    • A. `CONTINUOUS` sync mode keeps a compute cluster always running, which is cost-prohibitive and unnecessary for monthly updates. According to documentation, `TRIGGERED` mode is recommended for batch updates and cost-sensitive workloads because the sync pipeline only runs when manually triggered.
    • B. Relying solely on a time-based schedule risks triggering the sync before the monthly data pipeline has fully completed, potentially leading to partial or stale index content. To ensure reliability, the sync should be chained to the data refresh completion.
    • C. This approach is inefficient and unnecessarily costly. Dropping and rebuilding the index each month does not leverage incremental updates via Change Data Feed (CDF). A Delta Sync Index with `TRIGGERED` mode processes only the changed rows, optimizing compute costs.
    • D. This is the recommended approach. `TRIGGERED` mode minimizes compute costs because the index pipeline runs on demand. Configuring the sync as a downstream task in a Workflow ensures it executes only after the monthly pipeline succeeds, guaranteeing reliability. The documentation confirms that scheduling a Workflow to call the sync API after data pipeline completion is a standard, effective pattern for batch updates.

    Domain 5: Governance

    5.1 Use masking techniques as guard rails to meet a performance objective

    28.A healthcare organization is deploying a Retrieval-Augmented Generation (RAG) system to allow doctors to query a knowledge base of patient records. The performance objective is to provide helpful summaries without leaking any patient names or medical record numbers (MRNs). Which approach most effectively uses masking to achieve this goal?

    1. A.Masking the doctor's query to remove any specific patient names before the retrieval step.
    2. B.Masking the PII within the patient records before they are ingested and stored in the vector database.
    3. C.Allowing the LLM to process the full, unmasked retrieved text and then masking the final generated summary.
    4. D.Anonymizing the final chat logs after the entire interaction is complete.
    Show answer & explanation

    Correct answer: B — Masking the PII within the patient records before they are ingested and stored in the vector database.

    • A. Incorrect. While masking the query can be a useful practice, it does not address the primary risk. The main source of potential PII leakage is the patient records in the knowledge base, not the doctor's query. Even with a masked query, the retrieval step could still pull documents containing unmasked PII, which could then be exposed in the final summary.
    • B. Correct. This is the most proactive and secure approach. By masking Personally Identifiable Information (PII) like names and MRNs before the data is ever ingested into the vector database, it ensures that sensitive information is never available during the retrieval or generation steps. This method acts as a strong guardrail, aligns with the principle of data minimization, and fundamentally prevents the RAG system from leaking the specified PII.
    • C. Incorrect. This approach is highly risky because it allows the LLM to process unmasked sensitive data. The LLM could inadvertently leak PII in intermediate steps, logs, or if the final masking filter fails. Relying on masking the output is a reactive measure, whereas the goal should be to prevent the model from accessing the sensitive data in the first place.
    • D. Incorrect. Anonymizing chat logs is a post-interaction cleanup activity. It does nothing to prevent PII from being exposed to the doctor during the live RAG session. This measure is far too late in the process to serve as an effective guardrail for the primary performance objective.

    5.6 Use masking techniques as guard rails to meet a performance objective

    29.When implementing data masking as a guardrail for GenAI applications, what is the primary purpose of using row filters in Unity Catalog?

    1. A.To modify or redact the content within specific columns, such as hiding a social security number.
    2. B.To restrict the visibility of entire rows in a table based on the user's identity or group membership.
    3. C.To improve query performance by creating a pre-computed subset of the most frequently accessed rows.
    4. D.To validate that data being inserted into a table conforms to a specific set of business rules.
    Show answer & explanation

    Correct answer: B — To restrict the visibility of entire rows in a table based on the user's identity or group membership.

    • A. Incorrect. This describes the function of column-level masking policies, not row filters. Column masks modify or redact content within specific columns, whereas row filters control access to entire rows.
    • B. Correct. Row filters in Unity Catalog are the mechanism for implementing Row-Level Security (RLS). Their primary purpose is to restrict which rows a user can see in a table based on that user's identity or their membership in specific groups, thereby enforcing data access policies.
    • C. Incorrect. While applying filters can impact query performance, the primary purpose of row filters is security and access control, not query optimization. Performance optimization is typically achieved through other means like materialized views or indexing.
    • D. Incorrect. This describes data validation, which is typically handled by table constraints or data quality checks. Row filters are applied at query time to restrict access to existing data, not to validate data during ingestion.

    5.5 Recommend an alternative for problematic text mitigation in a data source feeding a GenAI application

    30.An e-commerce company wants to fine-tune an LLM to generate product descriptions based on user reviews. However, some user reviews contain competitor names and pricing information, which should not appear in the final product descriptions. The sentence structure of these reviews is valuable. What is a suitable mitigation technique?

    1. A.Use regular expressions and entity detection to identify and mask competitor names and prices in the review data before fine-tuning.
    2. B.Remove all user reviews that mention any competitor, discarding the rest of the review even when the sentence structure is valuable for training.
    3. C.Instruct the model during inference with a system prompt like 'Do not mention competitors' to suppress competitor names and pricing in generated descriptions.
    4. D.Summarize the reviews first, relying on the summarization process to naturally omit competitor names and pricing details from the condensed text.
    Show answer & explanation

    Correct answer: A — Use regular expressions and entity detection to identify and mask competitor names and prices in the review data before fine-tuning.

    • A. Correct. Using regular expressions and entity detection to identify and mask competitor names and prices in the review data before fine-tuning is a precise and proactive data sanitization method. It removes sensitive information while preserving the valuable sentence structure, ensuring the model never learns to generate unwanted content.
    • B. Incorrect. Removing all user reviews that mention any competitor is overly aggressive and discards valuable training data. This approach wastes the sentence structure and linguistic diversity present in those reviews, potentially degrading the fine-tuned model's performance.
    • C. Incorrect. Instructing the model during inference with a system prompt is an inference-time mitigation, not a data sanitization technique for training. The fine-tuning data has a stronger influence on model behavior, and the model may still generate competitor names and prices despite the prompt.
    • D. Incorrect. Summarizing the reviews first is unreliable because the summarization process may not consistently omit competitor names and pricing details. Additionally, summarization inherently alters the original text, destroying the valuable sentence structure needed for fine-tuning.

    Domain 6: Evaluation and Monitoring

    6.5: Use Databricks features to control LLM costs

    31.A development team is building a multi-purpose RAG application. For simple queries, they want to use a fast, inexpensive model, but for complex, analytical queries, they need a state-of-the-art, more expensive model. Which Databricks feature allows them to manage both models behind a single API and apply logic to route requests to the appropriate model, thereby optimizing the cost-performance balance?

    1. A.Databricks Jobs with multi-task capabilities
    2. B.The AI Gateway with multiple routes
    3. C.A Delta Live Tables pipeline with quality expectations
    4. D.Unity Catalog with row-level security
    Show answer & explanation

    Correct answer: B — The AI Gateway with multiple routes

    • A. Incorrect. Databricks Jobs are designed for orchestrating and scheduling batch or streaming data processing workflows. They are not suited for serving as a real-time API interface for model inference or implementing dynamic routing logic between multiple models based on request content.
    • B. Correct. The Databricks AI Gateway is specifically designed to provide a unified, secure API endpoint for various LLMs. It supports configuring multiple 'routes', each pointing to a different model. This allows developers to implement routing logic that directs requests to the appropriate model (e.g., a fast, cheap model for simple queries and a powerful, expensive one for complex queries) based on the request's attributes, thereby optimizing the cost-performance ratio.
    • C. Incorrect. Delta Live Tables (DLT) is a framework for building reliable and maintainable data processing pipelines (ETL). Its focus is on data quality and reliability within a data pipeline, not on managing or routing real-time API requests for model inference.
    • D. Incorrect. Unity Catalog is the centralized data governance solution for Databricks, providing features like data discovery, access control, and auditing. While it governs data assets, it does not manage or route API requests between different LLM models.

    6.7 Identify evaluation judges that require ground truth

    32.Which of the following evaluation metrics fundamentally requires a ground truth or reference answer to calculate a score for a text summarization task?

    1. A.Toxicity
    2. B.ROUGE-L
    3. C.Fluency
    4. D.Readability
    Show answer & explanation

    Correct answer: B — ROUGE-L

    • A. Incorrect. Toxicity is a metric that assesses the presence of harmful, offensive, or otherwise problematic language within the generated text itself. It is a content-based evaluation that does not require comparison against a reference or ground truth answer.
    • B. Correct. ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation - Longest Common Subsequence) is a standard metric for text summarization quality. It functions by directly comparing the generated summary against one or more human-written reference summaries (ground truth). The score is calculated based on the length of the longest common subsequence between the two texts, making the ground truth reference essential for its computation.
    • C. Incorrect. Fluency measures the linguistic quality of the generated text, focusing on aspects like grammatical correctness and natural language flow. This can be evaluated without a ground truth summary, often using another language model as a judge or through human assessment.
    • D. Incorrect. Readability evaluates how easy a piece of text is to understand. Metrics like the Flesch-Kincaid readability test calculate a score based on intrinsic properties of the text, such as average sentence length and syllables per word. This analysis is performed solely on the generated text and does not require a reference summary.

    6.1 Select an LLM choice (size and architecture) based on a set of quantitative evaluation metrics

    33.In the context of selecting an LLM, what is the most common trade-off when choosing a smaller model architecture (e.g., 7 billion parameters) over a much larger one (e.g., 70 billion parameters)?

    1. A.Smaller models are invariably better at few-shot learning due to a focused architecture that adapts well to in-context examples, whereas larger models excel at zero-shot tasks.
    2. B.Smaller models produce more factually correct information because their limited size constrains hallucination, but this same architectural limit also reduces their creative capacity.
    3. C.Smaller models generally have lower operational costs and faster inference speeds but may exhibit weaker performance on highly complex or nuanced tasks.
    4. D.Smaller models require significantly more VRAM for inference due to the higher overhead associated with aggressive quantization techniques, but they are faster to fine-tune.
    Show answer & explanation

    Correct answer: C — Smaller models generally have lower operational costs and faster inference speeds but may exhibit weaker performance on highly complex or nuanced tasks.

    • A. Incorrect. Larger models generally outperform smaller ones in both few-shot and zero-shot learning due to their greater capacity and broader training data. Smaller models are not invariably better at few-shot learning; their focused architecture does not typically adapt better to in-context examples than larger models.
    • B. Incorrect. Model size alone does not determine factual correctness; larger models often have a broader knowledge base that can improve accuracy. Hallucination is not simply constrained by smaller size, and factual correctness depends on training data and fine-tuning rather than just parameter count.
    • C. Correct. Smaller models require less computational power and memory, leading to lower operational costs and faster inference. However, this efficiency often comes at the expense of performance on highly complex or nuanced tasks where larger models excel.
    • D. Incorrect. Smaller models have fewer parameters and thus require significantly less VRAM for inference, not more. Aggressive quantization techniques are used to reduce memory usage further, and the statement about higher overhead is factually wrong.

    Subdomain 6.9: Use Databricks custom Scorers for evaluating agents and LLMs

    34.A data engineer wants to evaluate a RAG application's tone to ensure it aligns with a specific set of corporate communication guidelines. They decide to use an LLM-as-a-judge to score the tone of the generated responses. Which approach should the engineer use to create this custom scorer in Databricks?

    1. A.Use `mlflow.metrics.genai.make_genai_metric`, providing a metric definition, a grading prompt, and the Databricks model serving endpoint to be used as the judge.
    2. B.Define a standard Python function that calls the judge LLM with the corporate guidelines, then pass this function to the `custom_llm` parameter within the `mlflow.evaluate()` call.
    3. C.Modify the default `mlflow.metrics.toxicity` metric by overriding its internal prompt template, replacing the default instructions with the specific corporate guidelines for tone.
    4. D.Create a Spark UDF that queries the judge LLM with the corporate guidelines, then apply this function to the output dataframe after the `mlflow.evaluate()` call has completed.
    Show answer & explanation

    Correct answer: A — Use `mlflow.metrics.genai.make_genai_metric`, providing a metric definition, a grading prompt, and the Databricks model serving endpoint to be used as the judge.

    • A. Correct. `mlflow.metrics.genai.make_genai_metric` is the standard Databricks and MLflow-supported utility for building custom GenAI metrics using an LLM-as-a-judge. This function allows you to specify a metric name, a definition, a grading prompt (with criteria), and the model serving endpoint that will serve as the evaluator.
    • B. Incorrect. While you can define custom metrics as Python functions, passing them to a `custom_llm` parameter is not the correct mechanism for defining an LLM-as-a-judge scorer. In `mlflow.evaluate()`, custom metrics are typically passed through the `extra_metrics` argument, and they must follow the specific MLflow metric interface.
    • C. Incorrect. The built-in `toxicity` metric is a predefined metric with a specific objective. It is not intended to be repurposed by overriding internal templates. For a tone evaluation aligned with corporate guidelines, the recommended path is to create a new, distinct GenAI metric.
    • D. Incorrect. Using a Spark UDF after the evaluation has completed would be a post-processing step. This approach fails to integrate with the Databricks evaluation UI and MLflow's automated tracking, which are the primary benefits of using custom scorers during the evaluation phase.

    6.4 Use inference logging to assess deployed RAG application performance

    35.An operations team is monitoring a RAG application and notices a sudden increase in user complaints about responses being too slow. They have an inference table enabled for the serving endpoint. Which SQL query against the inference table would help them quantify this latency issue?(Select 2)

    1. A.SELECT date_trunc('hour', timestamp) AS hour, COUNT(*) AS null_context_count FROM my_rag_inference_table WHERE response.predictions.retrieved_context IS NULL GROUP BY hour ORDER BY hour;
    2. B.SELECT date_trunc('hour', timestamp) AS hour, AVG(response.predictions.output_token_count) AS avg_tokens FROM my_rag_inference_table WHERE response.predictions.output_token_count > 0 GROUP BY hour ORDER BY hour;
    3. C.SELECT date_trunc('hour', timestamp) AS hour, AVG(unix_timestamp(response_timestamp) - unix_timestamp(timestamp)) AS avg_latency_seconds FROM my_rag_inference_table GROUP BY hour;
    4. D.SELECT date_trunc('hour', timestamp) AS hour, COUNT(*) AS sample_count, AVG(unix_timestamp(response_timestamp) - unix_timestamp(timestamp)) AS avg_latency_seconds FROM my_rag_inference_table GROUP BY hour;
    Show answer & explanation

    Correct answers: C, D — SELECT date_trunc('hour', timestamp) AS hour, AVG(unix_timestamp(response_timestamp) - unix_timestamp(timestamp)) AS avg_latency_seconds FROM my_rag_inference_table GROUP BY hour;; SELECT date_trunc('hour', timestamp) AS hour, COUNT(*) AS sample_count, AVG(unix_timestamp(response_timestamp) - unix_timestamp(timestamp)) AS avg_latency_seconds FROM my_rag_inference_table GROUP BY hour;

    • A. This query counts requests missing retrieved context per hour. It measures a potential quality issue (null context), not the latency (response time) that is causing user complaints. It does not help quantify how slow the responses are.
    • B. This query calculates the average output token count per hour. While the number of generated tokens can influence latency, it does not directly measure time. Latency is measured in seconds or milliseconds, not token counts, so it fails to quantify the actual response delay.
    • C. This query directly computes the average latency in seconds by subtracting the request timestamp from the response timestamp. It is a valid method to identify hourly latency trends, as confirmed by Databricks documentation on inference tables. (For production monitoring, Databricks recommends using the built-in `total_latency_ms` column where available.)
    • D. This query also calculates average hourly latency, and it includes the number of requests (`sample_count`) per hour. The count provides important context, such as the volume of traffic during each period, which can help determine if the average is skewed by a small number of extreme values. It is a more complete query for latency analysis.

    Want the full experience?

    These are just samples. Practice the full Databricks Certified Generative AI Engineer Associate question bank in quiz mode — free, no signup, with domain practice and exam simulation.

    Practice all 373 questions in quiz mode