Databricks GenAI Engineer Evaluation and Monitoring: 116 practice questions

12 of the 116 Evaluation and Monitoring questions in the Certsqill Databricks GenAI Engineer bank, shown in full below. Each one carries an explanation for every option, not just the correct one — the wrong answers are where the marks go.

Preparing for Databricks GenAI Engineer? Take the free 5-min readiness check →

1. Configure an inference table and use tracing for latency: Which implementation best meets the requirement?

1. Capture token and cost usage records only.

   Usage records support consumption analysis but do not retain complete request and response content for audits.
2. Use MLflow traces as the sole governed audit store for endpoint traffic.

   Tracing is useful for operation diagnosis but does not replace the required configured inference records under Unity Catalog.
3. Configure an inference table and use tracing for latency diagnosis. ✓

   An inference table addresses governed endpoint records, while tracing separately exposes nested operation timing.
4. Write request and response logs to an unmanaged local application file.

   A local file does not satisfy the requirement for governed records under Unity Catalog.

Use an inference table for governed endpoint records and tracing separately for nested latency diagnosis.

2. Configure an inference table for sampled endpoint requests: Select TWO actions that address the missing teleme

Select two. More than one option is correct — every correct one is ticked below.

1. Increase the endpoint rate limit to create a guaranteed monthly cost ceiling.

   Rate limits constrain request or token throughput but do not guarantee a dollar budget cap.
2. Rely on model response text to estimate usage without endpoint telemetry.

   Response text cannot reliably provide token accounting, costs, or governed request and response retention.
3. Configure an inference table for sampled endpoint requests, responses, and serving metadata. ✓

   Inference tables provide governed request and response records with serving metadata for content-oriented review.
4. Use only MLflow traces because every trace is a qualitative compliance score.

   Tracing captures operations and timing, but it is not automatically a qualitative score or complete compliance record.
5. Use Unity Gateway usage records for token and cost consumption. ✓

   Usage records are appropriate for consumption metrics such as tokens and cost by governed endpoint.

Usage records address consumption; inference tables address governed request and response content.

3. The percentage of answers satisfying the expected business: Which metric best represents task success quality?

1. The percentage of requests completed below the latency target.

   Latency compliance is an operational measure and does not show whether answers are substantively correct.
2. The median number of tokens generated per answer.

   Token volume describes usage, but it does not establish whether the assistant completed the requested task.
3. The proportion of traces containing a model response.

   A completed model response may still be irrelevant, incorrect, or unable to satisfy the user’s requested outcome.
4. The percentage of answers satisfying the expected business outcome. ✓

   Task success measures whether responses accomplish the intended outcome across representative evaluation cases.

Task success measures the share of representative cases that meet their intended outcomes.

4. Compare combined input and output tokens: Which missing check should the team perform next?

1. Evaluate answer correctness before inspecting token consumption.

   Correctness evaluation cannot explain transport or service rejection caused by exceeding token-related limits.
2. Increase the request-per-minute limit before measuring token usage.

   Request volume is already below its limit, so increasing it does not address failures correlated with long inputs.
3. Replace request-rate monitoring with a monthly dollar-budget alert.

   A spending alert may help cost governance, but it does not diagnose failures caused by token limits.
4. Compare combined input and output tokens with the applicable token limit. ✓

   Long conversations can exceed token throughput or context-related limits even when request count and endpoint health appear normal.

Long conversations require checking token throughput, not only request count and endpoint health.

5. Supply expected answers or expected facts: Which configuration supports a correctness evaluation?

1. Score whether each answer is relevant to the warranty question.

   Relevance assesses topical alignment and does not determine whether the stated warranty facts are correct.
2. Supply expected answers or expected facts with the evaluation cases. ✓

   Correctness requires reference expectations, such as an expected response or expected facts, to compare with generated answers.
3. Use only the retrieved catalog passages as correctness labels.

   Retrieved passages provide evidence, but correctness still needs explicit expected facts or a reference answer.
4. Judge every answer from its trace confidence score.

   Confidence is not a reference expectation and cannot establish that the named warranty matches the catalog.

Correctness needs expected facts or reference answers, not merely retrieved context or topical relevance.

6. Add an SME rubric with calibrated anchor examples: Which change most directly addresses this defect?

1. Add an SME rubric with calibrated anchor examples. ✓

   A shared rubric and anchor examples align judgments on expected disclosures and ambiguous cases.
2. Increase the judge model’s sampling temperature for broader opinions.

   More variation would increase inconsistency rather than establish a shared interpretation of disclosure requirements.
3. Remove borderline onboarding cases from the evaluation set.

   Removing difficult cases hides the disagreement and weakens coverage of important onboarding decisions.
4. Tune the assistant on the evaluator’s current disputed labels.

   Changing the assistant does not resolve ambiguity in the evaluation standard or align subject-matter judgments.

SME rubrics and anchor examples make borderline scoring criteria consistent and explicit.

7. Use a custom scorer implementing the identifier rule: Which design best fits this requirement?

1. Use a relevance judge to infer whether the ticket identifier is probably present.

   Relevance is not a reliable deterministic check for a required field or exact business rule.
2. Use human review only and leave the rule outside evaluation.

   Manual review can supplement evaluation, but omitting the deterministic rule prevents consistent automated scoring.
3. Use a custom scorer implementing the identifier rule and returning feedback. ✓

   A custom scorer can enforce deterministic business logic while returning an explanation for reviewers.
4. Use retrieval groundedness to verify the identifier against support documents.

   Groundedness checks support from retrieved context, not whether every response contains a required identifier.

Custom scorers suit deterministic business rules and can provide reviewer-facing feedback.

8. Add suitable quality scorers: What should it do next?

1. Use token totals as the sole quality score for logged responses.

   Token totals describe consumption and cannot determine whether answers satisfy engineering search needs.
2. Increase endpoint request limits to improve answer usefulness.

   Higher throughput limits affect capacity, not the qualitative assessment of responses already being logged.
3. Treat every logged response as a successful answer because it completed inference.

   A recorded response proves inference occurred, but not that the answer was useful, correct, or supported.
4. Add suitable quality scorers, sampling, and review of representative logged cases. ✓

   Logs provide observations; quality scorers and review are needed to assess usefulness, correctness, and groundedness.

Logging supplies evidence; scorers and representative review measure whether responses are useful.

9. Select Model B because its representative measurements: Which choice meets both constraints?

1. Select Model B because its representative measurements satisfy both hard quality and p95 latency constraints. ✓

   Model B has quality of 0.89, above 0.88, and p95 latency of 1.8 seconds, below the 2-second limit.
2. Choose Model A for quality.

   Model A meets the quality threshold but violates the hard p95 latency limit with 2.4 seconds.
3. Average all representative quality-and-latency measurements, masking each model's individual constraint violations.

   Averaging alternatives hides individual threshold violations and does not identify a deployable model.
4. Choose Model C for minimum latency.

   Model C satisfies latency but falls below the required task-quality threshold.

Model B is the only measured option satisfying both hard thresholds.

10. The trace lacks the retriever span and retrieved content: The scorer returns “insufficient evidence.” What is

1. The trace lacks the retriever span and retrieved content required by the scorer. ✓

   A retrieval-dependent scorer cannot assess support when the trace omits both the retrieval operation and its retrieved contents.
2. The evaluation should add a reference answer and ignore the missing retriever span.

   A reference may support other evaluation approaches, but it does not supply the missing retriever operation and retrieved contents required by this scorer.
3. The answer generator is defective because the score is insufficient.

   The trace does not provide enough retrieval evidence to distinguish missing observability from a generator defect.
4. The scorer only needs the final answer because source support is implicit.

   A final answer does not expose the retrieved source material required by a retrieval-dependent scorer.

The trace omits the retriever evidence required by the retrieval-dependent scorer.

11. Instrument retrieval: Select TWO changes.

Select two. More than one option is correct — every correct one is ticked below.

1. Persist complete raw inputs, outputs, and span payloads for every case indefinitely so investigators can inspect any incident later.

   Unrestricted permanent capture conflicts with the requirement to protect sensitive case content and apply controlled collection and retention.
2. Log only total request duration together with the final success or failure status.

   Request-level telemetry cannot identify whether retrieval, the diagnostic tool, or generation caused the delay or failure.
3. Instrument retrieval, tool execution, and generation as nested spans with timings and errors. ✓

   Nested spans expose each operation's duration and error status, enabling component-level diagnosis.
4. Store token totals and the names of tools selected during every request.

   Token totals and tool names provide metadata but do not show operation latency or which operation failed.
5. Capture selected span inputs and outputs using privacy-aware sampling. ✓

   Selected, sampled span content provides diagnostic context while limiting exposure and retention of sensitive case data.

Use nested operation spans and privacy-aware sampling of selected content.

12. The token limit can throttle traffic even when the request: Select TWO accurate interpretations.

Select two. More than one option is correct — every correct one is ticked below.

1. The token limit should be evaluated only after the request limit has been exceeded.

   The two limits operate independently; token throughput can be exceeded while request throughput remains below its limit.
2. Staying below the request limit proves that the assistant is within both configured limits.

   The observed throttling shows that request compliance does not establish compliance with the separate token limit.
3. The token limit can throttle traffic even when the request count remains below its limit. ✓

   Long prompts can consume token throughput and trigger throttling without reaching 60 requests per minute.
4. The configured limits establish a fixed monthly spending ceiling for the assistant.

   Request and token rate limits constrain throughput, but they do not guarantee a cumulative monthly dollar cap.
5. The request limit controls request throughput, while the token limit controls token throughput. ✓

   These are separate throughput dimensions, so a workload can remain below the request limit while exceeding the token limit.

Request and token limits independently constrain throughput; neither is a spending or authorization control.

Other Databricks GenAI Engineer domains

• Application Development — 292 questions →
• Assembling and Deploying Applications — 212 questions →
• Design Applications — 136 questions →
• Data Preparation — 133 questions →
• Governance — 78 questions →
• All 967 Databricks GenAI Engineer questions →
• Databricks GenAI Engineer certification: requirements, cost and exam format →