Free Databricks Certified Associate Developer for Apache Spark practice test: 30 questions with answers

SparkSession.builder configures the session and getOrCreate returns an existing session or creates one, which is the standard entry point. The other options use methods that do not exist on SparkSession or are not valid Python.

withColumn adds or replaces a column computed from an expression and returns a new DataFrame. The select form drops the other columns, while addColumn and the withColumns syntax are not valid DataFrame calls.

In PySpark column expressions, conditions are combined with & and each condition is wrapped in parentheses. Python's and does not work on Column objects, select returns boolean columns rather than filtering rows, and single equals is an assignment.

printSchema prints the schema as a tree with names, types and nullability. Count returns a row count, describe computes statistics, and head(0) returns no rows.

collect() brings every row to the driver, so large results exhaust its memory no matter how much is added. Writing the output to storage or limiting it with show(n) keeps the data on executors, while caching or repartitioning do not reduce what collect() pulls to the driver.

Spark records transformations in a logical plan and executes them only when an action requires a result, which lets the optimizer improve the whole plan. Eager execution, cluster-idle scheduling and driver-first execution all contradict how Spark evaluates DataFrames.

The driver hosts the SparkSession, converts the query into a DAG of stages and tasks, and schedules work on executors. The cluster manager allocates resources, executors run tasks, and the NameNode is a storage component rather than a Spark scheduler.

Wide transformations such as groupBy and joins need data with the same key to be co-located, which triggers a shuffle across the network. Narrow transformations and column selections process each partition independently, so they do not shuffle.

spark.read.csv with header=True and inferSchema=True reads the header and infers types from the data. The text format does not parse columns, load.csv is not a reader method, and the json reader cannot parse CSV.

overwrite replaces the data at the path with the DataFrame's rows. Append adds rows, ignore skips the write when data exists, and error raises an exception rather than replacing the output.

spark.table returns a registered table as a DataFrame by name. Reading CSV from a table name, calling sql.read and calling load on a DataFrame are not valid ways to read a metastore table.

Parquet embeds its schema and is columnar, and Spark writes one part file per partition into a directory. CSV output also follows partitioning, JSON does not require a header, and append adds rows rather than overwriting.

createOrReplaceTempView registers the DataFrame under a name, and spark.sql runs a SQL query against it. The other methods such as saveAsSQL, query, registerTable and execute do not exist as used.

Both SQL and the DataFrame API are translated to the same logical plan and optimized by Catalyst, so performance is generally comparable. The other statements reflect misconceptions about the optimizer, languages and clusters.

GROUP BY with COUNT and a WHERE clause with AND are standard Spark SQL. WITHIN is not valid syntax, a JOIN requires a table name before ON, and HAVING without GROUP BY on amount is not valid as written.

WITH defines a named subquery that the main query can reference, which is the common table expression pattern. The other statements misplace aliases, use CTE as a keyword, or filter on aggregates without a subquery.

cache or persist stores the computed partitions so that later actions do not recompute the lineage. Collect, coalesce(1) and high repartitioning either move data to the driver or add cost without avoiding recomputation.

Early filtering and column selection reduce the data that later stages handle, and broadcast joins avoid shuffling large tables for small lookups. Collecting to the driver, row-level Python UDFs and a single shuffle partition for all jobs all create bottlenecks.

repartition performs a full shuffle and can increase or decrease the partition count. Coalesce reduces partitions without a full shuffle, cache does not change partitioning, and explain only prints the plan.

explain prints the plans, including the physical plan, for analyzing how Spark will execute the query. Print schema shows types, describe computes statistics, and show prints rows.

groupBy by customer_id followed by agg with sum of amount, aliased as total, gives one total per customer. Selecting then summing does not group, grouping by amount swaps the keys, and orderBy does not aggregate.

An inner join keeps only keys present in both DataFrames, so customers without orders are excluded. A left join keeps them, a full outer join keeps unmatched rows from both sides, and a cross join creates every combination.

A broadcast join sends the small table to every executor, which avoids shuffling the large table. Repartitioning to one partition creates a bottleneck, and caching the result does not change how the join is executed.

A left anti join returns left rows without a match, and joining on column names keeps a single key column. A semi join returns only left columns, an inner join is the default rather than a cross join, and outer joins produce nulls for unmatched values.

A window partitioned by department and ordered by salary descending, applied with rank over, gives the rank per department. The other forms swap partition and order, call withColumn on grouped data, or pass arguments that rank does not accept.

Python UDFs serialize rows between the JVM and Python and bypass the Catalyst optimizer, so built-in functions are usually faster. The other statements are wrong about where UDFs run, whether built-ins work with DataFrames, and caching.

lag returns the value from the previous row within the window, and the window partitions by customer and orders by date. Lead returns the next row, first returns the first value in the partition, and a sum over rows does not return the previous amount.

count is an action because it triggers execution and returns a value to the driver. Filter, groupBy and withColumn are transformations that only build a plan.

Wide transformations such as join need data from many partitions, which causes a shuffle. Filter and select are narrow, and groupBy is the wide transformation that needs key co-location.

countDistinct counts distinct values, and collect returns the single result row to the driver. Count after select counts all rows including duplicates, and the other forms do not return a distinct count in the way the requirement describes.