Databricks Data Engineer Associate Troubleshooting, Monitoring, and Optimization: 82 practice questions

12 of the 82 Troubleshooting, Monitoring, and Optimization questions in the Certsqill Databricks Data Engineer Associate bank, shown in full below. Each one carries an explanation for every option, not just the correct one — the wrong answers are where the marks go.

Preparing for Databricks Data Engineer Associate? Take the free 5-min readiness check →

1. Cluster event logs: What should the engineer inspect first?

1. Rewrite the finance joins to use a broadcast hint before examining cluster events.

   Join strategy affects running stages, while the observed failure occurs before executors are created.
2. Cluster event logs, queue time, and available capacity for the requested compute configuration. ✓

   Pending-capacity messages point to startup or capacity constraints, so cluster events and queue behavior are the first evidence.
3. Collect the input data to the driver so the job can bypass the unavailable executor startup.

   Driver collection increases driver memory pressure and cannot bypass the compute capacity required to start the job.
4. Increase shuffle partitions immediately, because startup capacity messages prove the transformations have too few tasks.

   Shuffle partition tuning affects task execution and does not explain failure to obtain initial cluster capacity.

Pending-capacity events indicate a startup or capacity problem, not a shuffle-task tuning problem.

2. Use `supplier_id` as a clustering key: What should it do?

1. Add partitions and ZORDER alongside liquid clustering.

   Liquid clustering is an alternative layout strategy rather than something to combine indiscriminately with traditional partitioning and ZORDER.
2. Use `supplier_id` as a clustering key. ✓

   The supported key aligns liquid clustering with the table's frequent filters without adding classic layout techniques.
3. Keep `event_date` permanently because clustering keys cannot change.

   Liquid-clustering keys can be changed, although existing files are not necessarily rewritten immediately.
4. Ignore frequent filters when selecting clustering keys.

   Frequent filters are important candidates when selecting supported clustering keys.

Use `supplier_id` as a clustering key.

3. Data skew: What problem is most likely?

1. Data skew. ✓

   One much larger task than the median, with disproportionate shuffle data, is characteristic of a skewed partition or hot key.
2. Insufficient SQL shuffle partitions alone.

   More partitions may help general parallelism, but they do not automatically resolve a single repeated hot key causing skew.
3. A missing Git pull.

   Source synchronization does not explain a runtime pattern where one task processes far more shuffle data.
4. Driver memory exhaustion.

   Driver memory exhaustion is associated with driver-side collection or result handling, not one unusually large executor task.

A single extreme task with disproportionate shuffle data indicates data skew.

4. Compare equivalent input: The recorded runs are: | Run | Input | Compute/config | Code | Runtime | |---|---:|-

1. Compare equivalent input, compute/configuration, and code versions, or explicitly normalize for those differences. ✓

   A defensible trend requires comparable measurements or normalization that accounts for the changed workload and execution conditions.
2. Compare elapsed times only.

   Elapsed time alone cannot distinguish degradation from changed input volume, compute, configuration, or code.
3. Use the latest successful run as the baseline for all conclusions.

   One successful run cannot establish a trend or control the differences among the recorded runs.
4. Increase every cluster size before rerunning so the existing measurements become directly comparable.

   Changing cluster size does not make historical runs comparable and introduces another execution variable.

Compare equivalent workload and execution conditions, or explicitly normalize their differences.

5. The ingest task’s run status: Which evidence should be investigated first?

1. The ingest task’s run status, logs, and earliest exception before investigating downstream skips. ✓

   The failed upstream predecessor is the earliest causal point; downstream tasks were skipped because its dependency failed.
2. The Git branch history, because every skipped task indicates a merge conflict in the latest commit.

   Skipped dependencies indicate task-flow consequences, not necessarily Git conflicts; run logs identify the actual failure.
3. The publish task’s output schema, because the final task is closest to the reported business result.

   A skipped downstream task cannot be the earliest cause when its upstream ingestion task already failed.
4. The dashboard query plan, because dashboard latency determines whether the upstream task should retry.

   Dashboard behavior does not explain an upstream task failure or determine the original exception in the pipeline.

Investigate the earliest failed upstream task and its logs before analyzing skipped downstream tasks.

6. Inspect initialization logs and verify repository access: What should be done first?

1. Add a broadcast join hint so the dependency installation completes during cluster startup.

   A join hint affects Spark planning and has no effect on dependency installation during cluster initialization.
2. Inspect initialization logs and verify repository access and dependency compatibility. ✓

   Cluster events and init-script logs can identify the startup failure, while checking repository access and dependency compatibility addresses the stated cause before retrying.
3. Retry the job repeatedly.

   Repeated retries do not diagnose a likely deterministic repository, network, or dependency problem during initialization.
4. Increase shuffle partitions and rerun the reporting task.

   Shuffle settings affect Spark processing after startup and cannot repair an init-script failure that occurs before any stage.

Inspect initialization logs and verify repository access and dependency compatibility before retrying.

7. Use liquid clustering for changing filter patterns: Which approach best meets the requirement?

1. Rely on predictive optimization to select the best layout for every future query.

   Predictive optimization automates eligible maintenance operations but does not guarantee optimal layout for every query pattern.
2. Use liquid clustering for changing filter patterns. ✓

   Liquid clustering provides adaptable data layout and data skipping without requiring a permanent classic partition design.
3. Partition permanently by warehouse_id for current reports.

   This permanently favors one current filter and may become unsuitable as access patterns evolve.
4. Partition permanently by both current filter columns.

   This still fixes the table layout around today's reporting patterns rather than allowing the layout strategy to adapt.

Use liquid clustering for changing filter patterns.

8. Confirm the skew in stage metrics: Which response best addresses this behavior?

1. Lower the broadcast threshold without checking reference-table size.

   The threshold controls broadcast eligibility and does not directly resolve a hot key in a shuffle join.
2. Confirm the skew in stage metrics, then use AQE skew handling or careful key salting. ✓

   A dominant task suggests key skew; metrics can confirm it, while supported AQE handling or salting can distribute the hot key's work.
3. Raise the shuffle-partition count for every pipeline run.

   More partitions do not reliably split one repeated key and can add scheduling overhead.
4. Increase RDD default parallelism and expect the repeated key to distribute evenly.

   spark.default.parallelism does not eliminate concentration caused by one repeatedly occurring SQL join key.

Confirm skew with stage metrics, then use supported AQE handling or carefully designed salting.

9. Compare equivalent volumes and record code: Which implementation satisfies that requirement?

1. Tune shuffle settings first, then compare the resulting production duration with last month’s run.

   Changing settings before establishing a controlled comparison prevents reliable attribution and still leaves workload differences unresolved.
2. Compare equivalent volumes and record code, compute, configuration, and task metrics. ✓

   Equivalent workloads plus execution metadata and task metrics support an evidence-based performance comparison.
3. Use the latest production run as the new baseline.

   A single run with different workload and execution conditions cannot establish a defensible baseline.
4. Compare only the two wall-clock durations.

   Duration alone is confounded by different input sizes, code, and compute conditions.

Compare equivalent volumes while recording code, compute, configuration, and task metrics.

10. The failed predecessor’s stage and executor details: What should the engineer inspect first?

1. The skipped task states.

   The skipped states are consequences of the failed prerequisite, not the primary failure evidence.
2. The previous successful run’s duration before examining current failure details.

   Historical duration does not identify the current executor-side shuffle fetch failure.
3. The failed predecessor’s stage and executor details. ✓

   The earliest failed predecessor contains the relevant stage, executor, and shuffle diagnostics.
4. The write task’s output schema and downstream metrics.

   The write task never ran, so its schema and runtime metrics cannot explain the upstream shuffle failure.

Inspect the earliest failed predecessor and its executor and shuffle diagnostics.

11. A cluster library failure requiring startup-log inspection: How should the failure be classified?

1. A cluster library failure requiring startup-log inspection. ✓

   The startup event and absent application stage identify dependency initialization as the failure boundary.
2. A driver memory failure caused by collecting too many application-log records.

   Driver collection occurs during application execution, whereas the incompatible library prevented startup.
3. A user-code exception from log parsing.

   The workload subject does not determine the failure type; the application did not reach parsing code.
4. A shuffle-skew failure.

   No application stage exists, so shuffle skew cannot explain a dependency failure during startup.

Treat it as a startup library failure and inspect cluster events and library logs.

12. Verify that the target table supports the selected layout: Select TWO actions that satisfy these requirements.

Select two. More than one option is correct — every correct one is ticked below.

1. Add classic partitioning and ZORDER automatically alongside liquid clustering.

   Liquid clustering is an alternative layout strategy and should not be combined arbitrarily with partitioning and ZORDER.
2. Verify that the target table supports the selected layout feature. ✓

   Layout capabilities vary by supported table type, so applicability must be verified before relying on the design.
3. Run VACUUM first with reduced retention to force layout improvement.

   VACUUM removes eligible old files and does not perform the layout or compaction work provided by OPTIMIZE.
4. Assume changing clustering keys instantly rewrites every existing file.

   Changing keys does not instantly rewrite all existing files; layout work occurs through subsequent maintenance operations.
5. Select clustering columns using representative filters and available data-skipping statistics. ✓

   Clustering keys should reflect filtering behavior and supported statistics rather than an arbitrary universal choice.

Verify feature support and choose clustering columns from representative filters and supported statistics.

Other Databricks Data Engineer Associate domains

• Data Transformation and Modeling — 180 questions →
• Data Ingestion and Loading — 172 questions →
• Working with Lakeflow Jobs — 129 questions →
• Governance and Security — 123 questions →
• Implementing CI/CD — 82 questions →
• All 817 Databricks Data Engineer Associate questions →
• Databricks Data Engineer Associate certification: requirements, cost and exam format →