Subdomain 1.8: Select and use models to create text embeddings.
1.Which of the following is NOT a typical use case for text embeddings?
- A.Apply dimensionality reduction like t-SNE
- B.Apply a clustering algorithm like k-means
- C.Train a classifier on the generated embeddings
- D.Convert the embeddings back to text for review
Show answer & explanation
Correct answer: D — Convert the embeddings back to text for review
- A. Dimensionality reduction like t-SNE is commonly applied to embeddings for visualization in 2D or 3D space, making it a valid use case.
- B. Clustering algorithms like k-means are frequently used on embeddings to group similar texts or assess semantic similarity, making it a typical use case.
- C. Training a classifier on embeddings is a standard practice for supervised learning tasks, as embeddings capture semantic meaning.
- D. Converting embeddings back to text is not a typical use case because embeddings are numerical vectors that encode compressed semantic information and cannot be losslessly reversed to the original text.
Subdomain 1.8: Select and use models to create text embeddings.
2.In a retrieval-augmented generation (RAG) workflow, what is the purpose of using text embeddings?(Select 2)
- A.To generate the final response text from the documents
- B.To encode both the user query and the document chunks
- C.To reduce the total size of the underlying language model
- D.To compute similarity scores between query and documents
- E.To fine-tune the language model during inference
Show answer & explanation
Correct answers: B, D — To encode both the user query and the document chunks; To compute similarity scores between query and documents
- A. Incorrect. Generating the final response text is the role of the language model after relevant context has been retrieved. Text embeddings are used for retrieval and semantic representation, not for generation.
- B. Correct. Text embeddings encode both the user query and document chunks into numerical vectors, enabling efficient semantic comparison and retrieval of relevant documents.
- C. Incorrect. Text embeddings are vector representations for semantic search and retrieval; they do not compress or reduce the size of the underlying language model.
- D. Correct. Text embeddings enable the computation of similarity scores (e.g., cosine similarity) between query and document embeddings to identify the most relevant documents for retrieval.
- E. Incorrect. Text embeddings are not used for fine-tuning; fine-tuning is a separate training process that occurs before deployment, not during inference.
Subdomain 1.6: Familiarity with the capabilities of Python natural language packages (spaCy, NumPy, vector databases, etc.).
3.Which spaCy pipeline component is responsible for assigning part-of-speech tags to tokens?
- A.Tokenizer
- B.Tagger
- C.Parser
- D.NER
Show answer & explanation
Correct answer: B — Tagger
- A. Incorrect. The tokenizer splits text into tokens such as words or punctuation, but does not assign part-of-speech tags.
- B. Correct. The tagger assigns part-of-speech tags (e.g., noun, verb, adjective) to tokens. It is a standard spaCy pipeline component.
- C. Incorrect. The parser analyzes syntactic structure (dependency parsing). It is a different component from the tagger.
- D. Incorrect. NER recognizes named entities (e.g., persons, organizations). This task is distinct from part-of-speech tagging.
Subdomain 1.4: Curate and embed content datasets for RAGs.
4.Which of the following is NOT a typical step in a Retrieval-Augmented Generation (RAG) pipeline?
- A.Generating a final answer by conditioning the LLM on the retrieved information.
- B.Transforming the user's natural language query into a dense vector for similarity matching.
- C.Retrieving top-k relevant document chunks from the vector store given the query.
- D.Continuously training the language model on domain-specific data to improve performance.
Show answer & explanation
Correct answer: D — Continuously training the language model on domain-specific data to improve performance.
- A. Generating a final answer by conditioning the LLM on the retrieved information is a core step in a RAG pipeline. It is the generation phase that combines the query with external knowledge to produce a response.
- B. Transforming the user's natural language query into a dense vector is part of the embedding or retrieval step in RAG. It enables semantic similarity search against stored document embeddings.
- C. Retrieving top-k relevant document chunks from the vector store is the retrieval step in a RAG workflow. It provides the context for the LLM to generate an answer.
- D. Continuously training the language model on domain-specific data describes fine-tuning or continued pretraining, which is not a typical step in RAG. RAG relies on retrieving external context at inference time rather than retraining the model.
Subdomain 1.4: Curate and embed content datasets for RAGs.
5.Which metric is most appropriate for evaluating the retrieval component of a RAG system?
- A.BLEU score
- B.ROUGE-L
- C.nDCG (normalized Discounted Cumulative Gain)
- D.F1 score
Show answer & explanation
Correct answer: C — nDCG (normalized Discounted Cumulative Gain)
- A. Incorrect. BLEU score is primarily used for evaluating machine translation by measuring n-gram overlap with reference translations. It is not suitable for evaluating retrieval ranking in a RAG system.
- B. Incorrect. ROUGE-L measures longest common subsequence overlap and is often used for summarization or text generation quality. It is not designed for assessing retrieval performance; it is more applicable to evaluating the generation component of RAG.
- C. Correct. nDCG is a ranking metric commonly used to evaluate information retrieval systems, including the retriever in a RAG pipeline. It rewards relevant documents appearing higher in the ranked list, making it well-suited for assessing retriever performance.
- D. Incorrect. F1 score is used for classification and token-level overlap tasks, such as QA extraction. It does not directly measure ranking quality of retrieved documents in a RAG pipeline.
Subdomain 1.10: Use Python packages (spaCy, NumPy, Keras, etc.) to implement specific traditional machine learning analyses.
6.In spaCy, which attribute of a Token object is used to obtain the part-of-speech tag (e.g., 'NN', 'VB')?
- A.token.pos_
- B.token.tag_
- C.token.dep_
- D.token.lemma_
Show answer & explanation
Correct answer: B — token.tag_
- A. Incorrect. `token.pos_` returns the coarse-grained universal part-of-speech label (e.g., 'NOUN', 'VERB'), not the fine-grained tag like 'NN' or 'VB'.
- B. Correct. `token.tag_` returns the fine-grained part-of-speech tag from the tagger, such as 'NN' for noun or 'VB' for verb, used for Penn Treebank-style tags.
- C. Incorrect. `token.dep_` returns the syntactic dependency relation of the token (e.g., 'nsubj', 'dobj'), describing sentence structure, not the POS tag.
- D. Incorrect. `token.lemma_` returns the base form of the word (e.g., 'run' for 'running'), which is unrelated to the part-of-speech tag.
Subdomain 1.10: Use Python packages (spaCy, NumPy, Keras, etc.) to implement specific traditional machine learning analyses.
7.What is the purpose of NumPy's `argsort` function?
- A.Returns the sorted array in ascending order
- B.Returns the indices that would sort an array
- C.Computes the rank of each element in the array
- D.Sorts the array in place and returns None
Show answer & explanation
Correct answer: B — Returns the indices that would sort an array
- A. Incorrect. `argsort` does not return the sorted values themselves; it returns index positions. The sorted array can be obtained using `np.sort()` or `sorted()`.
- B. Correct. `numpy.argsort` returns the indices that would arrange the array in sorted order. For example, with input `[3, 1, 2]`, it returns `[1, 2, 0]` because those indices would sort the array. These indices can then be used to reorder the original array or related arrays consistently.
- C. Incorrect. `argsort` does not directly compute ranks or ordinal positions of elements. While it can be used to derive ranks, its primary purpose is to return the permutation of indices needed to sort the array.
- D. Incorrect. `argsort` does not modify the original array in place and does not return `None`. In-place sorting is handled by `ndarray.sort()` or `np.sort()` with the `out` parameter.
Subdomain 1.9: Use prompt engineering principles to create prompts to achieve desired results.
8.Which of the following statements about prompt engineering are true?(Select 2)
- A.Ambiguous phrasing often causes off-target model responses.
- B.Models inherently grasp user context without explicit prompts.
- C.Hallucinations can still arise despite well-designed prompts.
- D.Longer prompts always yield better model performance.
- E.The model provides identical outputs every time for a given prompt.
Show answer & explanation
Correct answers: A, C — Ambiguous phrasing often causes off-target model responses.; Hallucinations can still arise despite well-designed prompts.
- A. Correct. Ambiguous phrasing leaves the model uncertain about the intended task, often resulting in vague or irrelevant responses. Clear and specific prompts are essential for guiding the model toward desired outputs.
- B. Incorrect. Models do not inherently understand the user's context; they rely solely on the information provided in the prompt. Explicit context and well-structured prompts are necessary for appropriate responses.
- C. Correct. Hallucinations, where the model generates factually incorrect information, are a known limitation of LLMs and can occur even with carefully designed prompts. Prompt engineering reduces but does not eliminate this risk.
- D. Incorrect. Longer prompts do not guarantee better performance; they can introduce noise or conflicting instructions. Effective prompts are clear, concise, and well-structured rather than simply long.
- E. Incorrect. LLMs are non-deterministic by default; outputs can vary due to sampling parameters like temperature. Identical outputs are not guaranteed unless decoding is made fully deterministic.
Subdomain 1.9: Use prompt engineering principles to create prompts to achieve desired results.
9.Which of the following is a prompt engineering technique that can improve the reliability and interpretability of a large language model's output?
- A.Asking the model to give only the final answer, no steps.
- B.Prompting the model to explain its reasoning steps before answering.
- C.Providing the model with many training examples for fine-tuning.
- D.Using a prompt composed solely of punctuation characters.
Show answer & explanation
Correct answer: B — Prompting the model to explain its reasoning steps before answering.
- A. Incorrect. While asking for only the final answer may yield concise responses, it does not improve reasoning quality or reliability. It hides the model's thought process, making it harder to assess correctness.
- B. Correct. Prompting the model to explain its reasoning steps before answering encourages structured thinking, increases interpretability, and often leads to more reliable outputs by reducing careless errors. This is a common zero-shot prompt engineering technique.
- C. Incorrect. Providing many training examples is part of fine-tuning or few-shot learning, which are different from inference-time prompt engineering. While it can improve performance, it is not a prompt engineering technique per se.
- D. Incorrect. A prompt consisting solely of punctuation characters lacks meaningful task instruction or context, and will not produce a useful or reliable response from the model.
Subdomain 1.7: Read research papers (articles, conference papers, etc.) to identify emerging LLM trends and technologies.
10.Which evaluation metric is specifically designed to assess the quality of abstractive text summarization?
- A.BLEU
- B.ROUGE
- C.Perplexity
- D.Accuracy
Show answer & explanation
Correct answer: B — ROUGE
- A. Incorrect. BLEU (Bilingual Evaluation Understudy) is primarily used for evaluating machine translation quality by comparing n-gram overlaps with reference translations. It is not specifically designed for abstractive summarization.
- B. Correct. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is specifically designed for evaluating text summarization, including abstractive summarization. It measures n-gram, word sequence, and word pair overlaps between candidate and reference summaries.
- C. Incorrect. Perplexity measures how well a probabilistic language model predicts a sample. It is a language modeling metric and does not directly assess summarization quality, such as content overlap or faithfulness.
- D. Incorrect. Accuracy is a classification metric that measures the proportion of correct predictions. It is not suitable for evaluating free-form generated text like abstractive summaries.
Subdomain 1.7: Read research papers (articles, conference papers, etc.) to identify emerging LLM trends and technologies.
11.A team notices that a recently published model achieves state-of-the-art results on the GLUE benchmark but performs poorly on their internal customer service dataset. After reading the paper's methodology and experiments, what is the most likely limitation they should investigate?
- A.The model was trained on an insufficient amount of data.
- B.The model likely overfits the specific benchmark distribution.
- C.The tokenization method used in the model is outdated.
- D.The model does not incorporate an attention mechanism.
Show answer & explanation
Correct answer: B — The model likely overfits the specific benchmark distribution.
- A. Incorrect. While insufficient training data can limit generalization, the model's strong results on GLUE suggest it was trained on a substantial and relevant dataset for that benchmark. The issue is more likely related to overfitting to the benchmark distribution rather than data quantity. The team should focus on domain mismatch and transferability.
- B. Correct. The model may have overfit to the GLUE benchmark's specific distribution, meaning it performs well on data similar to GLUE but fails to generalize to out-of-distribution data like the internal customer service dataset. This is a common limitation in benchmark-focused models, where models are heavily tuned to the benchmark's characteristics, leading to poor domain transfer. The team should investigate whether the paper's methodology involved benchmark-specific tuning or narrow data coverage.
- C. Incorrect. An outdated tokenization method could affect performance, especially for domain-specific text, but it is unlikely to cause such a stark discrepancy between GLUE and a different dataset unless the tokenization is fundamentally incompatible with the internal data. Overfitting to the benchmark distribution is a more likely explanation for the observed mismatch.
- D. Incorrect. Modern LLMs, especially those achieving state-of-the-art results on GLUE, almost certainly incorporate attention mechanisms. This option is implausible given the context, as attention-based architectures are foundational for such models. The mismatch is better explained by overfitting to the benchmark distribution.
Subdomain 1.2: Awareness of the process of extracting insights from large datasets using data mining, data visualization, and similar techniques.
12.A data analyst examines a correlation matrix and finds that variables X and Y have a correlation coefficient of -0.95, while variables X and Z have a coefficient of 0.10. What can they conclude?
- A.Strong negative for X-Y; no correlation for X-Z
- B.Weak positive for X-Y; strong negative for X-Z
- C.Strong positive for X-Y; weak negative for X-Z
- D.Strong linear relationship for both X-Y and X-Z
Show answer & explanation
Correct answer: A — Strong negative for X-Y; no correlation for X-Z
- A. Correct. A correlation coefficient of -0.95 indicates a very strong negative linear relationship between X and Y, meaning as one variable increases, the other tends to decrease. A coefficient of 0.10 is very close to zero, indicating little to no linear relationship between X and Z.
- B. Incorrect. The correlation between X and Y is strong negative, not weak positive. The correlation for X and Z is near zero (no correlation), not strong negative.
- C. Incorrect. The sign and magnitude for X-Y are wrong: -0.95 is strongly negative, not positive. For X-Z, 0.10 is weakly positive, not weakly negative.
- D. Incorrect. X-Y does show a strong linear relationship, but it is negative, not just 'strong.' X-Z does not show a strong linear relationship because 0.10 is very weak and close to zero.
Subdomain 1.5: Familiarity with the fundamentals of machine learning (e.g., feature engineering, model comparison, cross validation).
13.You are comparing two models on an imbalanced test set. Model X has higher accuracy, while Model Y has higher precision. Which model should you choose?
- A.Model X, because higher accuracy indicates better overall performance on the test set, a standard metric.
- B.Model Y, because higher precision indicates fewer false positives, essential for correctly identifying the minority class.
- C.Model X after threshold tuning might improve precision but not guaranteed to outperform Model Y.
- D.Neither; use the F1-score for comparison, as accuracy is misleading for imbalanced data.
Show answer & explanation
Correct answer: D — Neither; use the F1-score for comparison, as accuracy is misleading for imbalanced data.
- A. Incorrect. Higher accuracy alone is not reliable for imbalanced datasets; a model can achieve high accuracy by predicting the majority class, hiding poor performance on the minority class.
- B. Incorrect. While higher precision indicates fewer false positives, it does not consider false negatives. For imbalanced data, recall is equally important, and a single metric like precision is insufficient to compare models.
- C. Incorrect. Adjusting the threshold can change precision and recall, but there is no guarantee that Model X will outperform Model Y after tuning. The outcome depends on the trade-off and specific use case.
- D. Correct. The F1-score balances precision and recall, making it a more reliable metric for imbalanced datasets. Accuracy can be misleading due to class imbalance, so using F1-score provides a better basis for model comparison.
Subdomain 1.5: Familiarity with the fundamentals of machine learning (e.g., feature engineering, model comparison, cross validation).
14.Which of the following are examples of feature engineering techniques?(Select 3)
- A.One-hot encoding of categorical variables to create binary columns
- B.Applying dropout during neural network training as regularization
- C.Normalizing numeric features to a common scale such as min-max scaling
- D.Using gradient descent optimization to update model parameters iteratively
- E.Selecting the learning rate, a hyperparameter, for model training
- F.Creating polynomial features from numeric features to model interactions
Show answer & explanation
Correct answers: A, C, F — One-hot encoding of categorical variables to create binary columns; Normalizing numeric features to a common scale such as min-max scaling; Creating polynomial features from numeric features to model interactions
- A. Correct. One-hot encoding converts categorical variables into binary columns, enabling machine learning models to process them. This is a classic feature engineering technique that transforms input representation to improve learning.
- B. Incorrect. Dropout is a regularization method used during neural network training to reduce overfitting. It does not modify input data features, so it is not a feature engineering technique.
- C. Correct. Normalizing numeric features rescales inputs to a common range (e.g., min-max scaling), which is a standard preprocessing and feature engineering step. It improves model training stability and performance.
- D. Incorrect. Gradient descent is an optimization algorithm for updating model parameters during training. It affects how the model learns, not how features are constructed or transformed, so it is not feature engineering.
- E. Incorrect. Selecting the learning rate is hyperparameter tuning, not feature engineering. It controls the training process rather than the input features themselves.
- F. Correct. Creating polynomial features expands the feature set to include interactions and nonlinear relationships. This is a direct example of feature engineering because it constructs new inputs from existing ones.
Subdomain 1.1: Assist in deployment and evaluation of model scalability, performance, and reliability under the supervision of senior team members.
15.Which of the following are key metrics for monitoring the performance, scalability, and reliability of a deployed generative AI model?(Select 3)
- A.GPU utilization on the serving nodes
- B.Number of users currently logged into the application
- C.Current queue length for incoming inference requests
- D.Accuracy of the model on the latest validation data
- E.Time passed since the most recent model update
- F.Disk I/O wait time observed on the serving node
Show answer & explanation
Correct answers: A, C, F — GPU utilization on the serving nodes; Current queue length for incoming inference requests; Disk I/O wait time observed on the serving node
- A. Correct. GPU utilization directly indicates how effectively compute resources are being used for inference. High utilization suggests efficient use, while low utilization may point to underutilization or bottlenecks. Monitoring this metric helps assess serving performance and scalability.
- B. Incorrect. While user count can indicate demand, it is an application-level metric that does not directly measure model serving performance or scalability. It may correlate with load but is less actionable than inference-specific metrics.
- C. Correct. Queue length for incoming inference requests is a strong indicator of serving scalability and latency under load. A growing queue often signals that the model or serving system cannot keep up with demand, prompting scaling actions.
- D. Incorrect. Model accuracy on validation data measures model quality, not serving performance or scalability. It is important for evaluation but does not indicate whether the deployment can handle traffic reliably or efficiently.
- E. Incorrect. Time since the most recent model update relates to model freshness or versioning, not serving throughput or reliability. It may be relevant for operational tracking but is not a primary metric for scalability or performance.
- F. Correct. Disk I/O wait time on the serving node is an infrastructure performance metric that can reveal storage bottlenecks affecting inference latency and reliability. High wait times can slow down the serving system even if GPU utilization is low.
Subdomain 1.3: Build LLM use cases such as retrieval-augmented generation (RAG), chatbots, and summarizers.
16.In a conversational AI system built with an LLM, what is the purpose of maintaining a dialog state?
- A.To log all user messages for later analysis and debugging purposes.
- B.To ensure the LLM does not exceed its token limit during each turn.
- C.To track intent, slots, and conversation context across turns.
- D.To store the LLM's pre-trained knowledge for quick access during inference.
Show answer & explanation
Correct answer: C — To track intent, slots, and conversation context across turns.
- A. Incorrect. While logging user messages can be useful for analytics, auditing, or debugging, that is not the primary purpose of dialog state. Dialog state is about managing the conversation's flow and context, not just logging.
- B. Incorrect. Token limit management is a separate concern handled by prompt construction, context trimming, or memory management. Dialog state is about preserving meaningful conversational information, not primarily about controlling token usage.
- C. Correct. Dialog state tracks intent, slots, and conversation context across turns so the assistant can maintain continuity in the conversation. This helps the system understand what the user has already said, what information is missing (e.g., in task-oriented systems), and how to respond consistently and coherently.
- D. Incorrect. The LLM's pre-trained knowledge is stored in its model parameters, not in dialog state. Dialog state captures temporary, conversation-specific information rather than the model's learned knowledge.