Free Aug-2025 Associate-Developer-Apache-Spark-3.5 Certification Sample Questions certification Exam [Q21-Q38]

QUESTION 21
Which feature of Spark Connect is considered when designing an application to enable remote interaction with the Spark cluster?

It provides a way to run Spark applications remotely in any programming language

It can be used to interact with any remote cluster using the REST API

It allows for remote execution of Spark jobs

It is primarily used for data ingestion into Spark from external sources

QUESTION 22
A developer initializes a SparkSession:

spark = SparkSession.builder
.appName(“Analytics Application”)
.getOrCreate()
Which statement describes thesparkSparkSession?

ThegetOrCreate()method explicitly destroys any existing SparkSession and creates a new one.

A SparkSession is unique for eachappName, and callinggetOrCreate()with the same name will return an existing SparkSession once it has been created.

If a SparkSession already exists, this code will return the existing session instead of creating a new one.

A new SparkSession is created every time thegetOrCreate()method is invoked.

QUESTION 23
A data engineer is working ona Streaming DataFrame streaming_df with the given streaming data:

Which operation is supported with streaming_df?

streaming_df.select(countDistinct(“Name”))

streaming_df.groupby(“Id”).count()

streaming_df.orderBy(“timestamp”).limit(4)

streaming_df.filter(col(“count”) < 30).show()

QUESTION 24
A data engineer is building an Apache Spark™ Structured Streaming application to process a stream of JSON events in real time. The engineer wants the application to be fault-tolerant and resume processing from the last successfully processed record in case of a failure. To achieve this, the data engineer decides to implement checkpoints.
Which code snippet should the data engineer use?

query = streaming_df.writeStream
.format(“console”)
.option(“checkpoint”, “/path/to/checkpoint”)
.outputMode(“append”)
.start()

query = streaming_df.writeStream
.format(“console”)
.outputMode(“append”)
.option(“checkpointLocation”, “/path/to/checkpoint”)
.start()

query = streaming_df.writeStream
.format(“console”)
.outputMode(“complete”)
.start()

query = streaming_df.writeStream
.format(“console”)
.outputMode(“append”)
.start()

QUESTION 25
A data engineer observes that an upstream streaming source sends duplicate records, where duplicates share the same key and have at most a 30-minute difference inevent_timestamp. The engineer adds:
dropDuplicatesWithinWatermark(“event_timestamp”, “30 minutes”)
What is the result?

It is not able to handle deduplication in this scenario

It removes duplicates that arrive within the 30-minute window specified by the watermark

It removes all duplicates regardless of when they arrive

It accepts watermarks in seconds and the code results in an error

QUESTION 26
A data scientist is working on a large dataset in Apache Spark using PySpark. The data scientist has a DataFramedfwith columnsuser_id,product_id, andpurchase_amountand needs to perform some operations on this data efficiently.
Which sequence of operations results in transformations that require a shuffle followed by transformations that do not?

df.filter(df.purchase_amount > 100).groupBy(“user_id”).sum(“purchase_amount”)

df.withColumn(“discount”, df.purchase_amount * 0.1).select(“discount”)

df.withColumn(“purchase_date”, current_date()).where(“total_purchase > 50”)

df.groupBy(“user_id”).agg(sum(“purchase_amount”).alias(“total_purchase”)).repartition(10)

QUESTION 27
A developer wants to test Spark Connect with an existing Spark application.
What are the two alternative ways the developer can start a local Spark Connect server without changing their existing application code? (Choose 2 answers)

Execute their pyspark shell with the option–remote “https://localhost”

Execute their pyspark shell with the option–remote “sc://localhost”

Set the environment variableSPARK_REMOTE=”sc://localhost”before starting the pyspark shell

Add.remote(“sc://localhost”)to their SparkSession.builder calls in their Spark code

Ensure the Spark propertyspark.connect.grpc.binding.portis set to 15002 in the application code

QUESTION 28
A data engineer has been asked to produce a Parquet table which is overwritten every day with the latest data.
The downstream consumer of this Parquet table has a hard requirement that the data in this table is produced with all records sorted by themarket_timefield.
Which line of Spark code will produce a Parquet table that meets these requirements?

final_df
.sort(“market_time”)
.write
.format(“parquet”)
.mode(“overwrite”)
.saveAsTable(“output.market_events”)

final_df
.orderBy(“market_time”)
.write
.format(“parquet”)
.mode(“overwrite”)
.saveAsTable(“output.market_events”)

final_df
.sort(“market_time”)
.coalesce(1)
.write
.format(“parquet”)
.mode(“overwrite”)
.saveAsTable(“output.market_events”)

final_df
.sortWithinPartitions(“market_time”)
.write
.format(“parquet”)
.mode(“overwrite”)
.saveAsTable(“output.market_events”)

QUESTION 29
The following code fragment results in an error:
@F.udf(T.IntegerType())
def simple_udf(t: str) -> str:
return answer * 3.14159
Which code fragment should be used instead?

@F.udf(T.IntegerType())
def simple_udf(t: int) -> int:
return t * 3.14159

@F.udf(T.DoubleType())
def simple_udf(t: float) -> float:
return t * 3.14159

@F.udf(T.DoubleType())
def simple_udf(t: int) -> int:
return t * 3.14159

@F.udf(T.IntegerType())
def simple_udf(t: float) -> float:
return t * 3.14159

QUESTION 30
You have:
DataFrame A: 128 GB of transactions
DataFrame B: 1 GB user lookup table
Which strategy is correct for broadcasting?

DataFrame B should be broadcasted because it is smaller and will eliminate the need for shuffling itself

DataFrame B should be broadcasted because it is smaller and will eliminate the need for shuffling DataFrame A

DataFrame A should be broadcasted because it is larger and will eliminate the need for shuffling DataFrame B

DataFrame A should be broadcasted because it is smaller and will eliminate the need for shuffling itself

QUESTION 31
A data engineer needs to write a Streaming DataFrame as Parquet files.
Given the code:

Which code fragment should be inserted to meet the requirement?
A)

B)

C)

D)

Which code fragment should be inserted to meet the requirement?

.format(“parquet”)
.option(“location”, “path/to/destination/dir”)

CopyEdit
.option(“format”, “parquet”)
.option(“destination”, “path/to/destination/dir”)

.option(“format”, “parquet”)
.option(“location”, “path/to/destination/dir”)

.format(“parquet”)
.option(“path”, “path/to/destination/dir”)

QUESTION 32
A Spark application is experiencing performance issues in client mode because the driver is resource- constrained.
How should this issue be resolved?

Add more executor instances to the cluster

Increase the driver memory on the client machine

Switch the deployment mode to cluster mode

Switch the deployment mode to local mode

QUESTION 33
A data scientist of an e-commerce company is working with user data obtained from its subscriber database and has stored the data in a DataFrame df_user. Before further processing the data, the data scientist wants to create another DataFrame df_user_non_pii and store only the non-PII columns in this DataFrame. The PII columns in df_user are first_name, last_name, email, and birthdate.
Which code snippet can be used to meet this requirement?

df_user_non_pii = df_user.drop(“first_name”, “last_name”, “email”, “birthdate”)

df_user_non_pii = df_user.dropfields(“first_name”, “last_name”, “email”, “birthdate”)

df_user_non_pii = df_user.dropfields(“first_name, last_name, email, birthdate”)

QUESTION 34
A Data Analyst is working on the DataFramesensor_df, which contains two columns:
Which code fragment returns a DataFrame that splits therecordcolumn into separate columns and has one array item per row?
A)

B)

C)

D)

exploded_df = sensor_df.withColumn(“record_exploded”, explode(“record”)) exploded_df = exploded_df.select(“record_datetime”, “sensor_id”, “status”, “health”)

exploded_df = exploded_df.select(
“record_datetime”,
“record_exploded.sensor_id”,
“record_exploded.status”,
“record_exploded.health”
)
exploded_df = sensor_df.withColumn(“record_exploded”, explode(“record”))

exploded_df = exploded_df.select(“record_datetime”, “record_exploded”)

QUESTION 35
A data engineer wants to process a streaming DataFrame that receives sensor readings every second with columnssensor_id,temperature, andtimestamp. The engineer needs to calculate the average temperature for each sensor over the last 5 minutes while the data is streaming.
Which code implementation achieves the requirement?
Options from the images provided:

QUESTION 36
What is the relationship between jobs, stages, and tasks during execution in Apache Spark?
Options:

A job contains multiple stages, and each stage contains multiple tasks.

A job contains multiple tasks, and each task contains multiple stages.

A stage contains multiple jobs, and each job contains multiple tasks.

A stage contains multiple tasks, and each task contains multiple jobs.

QUESTION 37
A developer needs to produce a Python dictionary using data stored in a small Parquet table, which looks like this:

The resulting Python dictionary must contain a mapping of region-> region id containing the smallest 3 region_idvalues.
Which code fragment meets the requirements?
A)

B)

C)

D)

The resulting Python dictionary must contain a mapping ofregion -> region_idfor the smallest
3region_idvalues.
Which code fragment meets the requirements?