Get Associate-Developer-Apache-Spark-3.5 Braindumps & Associate-Developer-Apache-Spark-3.5 Real Exam Questions [Q43-Q60]

QUESTION 43
Given the code fragment:

import pyspark.pandas as ps
psdf = ps.DataFrame({‘col1’: [1, 2], ‘col2’: [3, 4]})
Which method is used to convert a Pandas API on Spark DataFrame (pyspark.pandas.DataFrame) into a standard PySpark DataFrame (pyspark.sql.DataFrame)?

psdf.to_spark()

psdf.to_pyspark()

psdf.to_pandas()

psdf.to_dataframe()

QUESTION 44
Which UDF implementation calculates the length of strings in a Spark DataFrame?

df.withColumn(“length”, spark.udf(“len”, StringType()))

df.select(length(col(“stringColumn”)).alias(“length”))

spark.udf.register(“stringLength”, lambda s: len(s))

df.withColumn(“length”, udf(lambda s: len(s), StringType()))

QUESTION 45
A data analyst builds a Spark application to analyze finance data and performs the following operations:filter, select,groupBy, andcoalesce.
Which operation results in a shuffle?

groupBy

filter

select

coalesce

QUESTION 46
Given this code:

.withWatermark(“event_time”,”10 minutes”)
.groupBy(window(“event_time”,”15 minutes”))
.count()
What happens to data that arrives after the watermark threshold?
Options:

Records that arrive later than the watermark threshold (10 minutes) will automatically be included in the aggregation if they fall within the 15-minute window.

Any data arriving more than 10 minutes after the watermark threshold will be ignored and not included in the aggregation.

Data arriving more than 10 minutes after the latest watermark will still be included in the aggregation but will be placed into the next window.

The watermark ensures that late data arriving within 10 minutes of the latest event_time will be processed and included in the windowed aggregation.

QUESTION 47
What is the difference betweendf.cache()anddf.persist()in Spark DataFrame?

Bothcache()andpersist()can be used to set the default storage level (MEMORY_AND_DISK_SER)

Both functions perform the same operation. Thepersist()function provides improved performance asits default storage level isDISK_ONLY.

persist()- Persists the DataFrame with the default storage level (MEMORY_AND_DISK_SER) andcache()- Can be used to set different storage levels to persist the contents of the DataFrame.

cache()- Persists the DataFrame with the default storage level (MEMORY_AND_DISK) andpersist()- Can be used to set different storage levels to persist the contents of the DataFrame

QUESTION 48
A developer wants to test Spark Connect with an existing Spark application.
What are the two alternative ways the developer can start a local Spark Connect server without changing their existing application code? (Choose 2 answers)

Execute their pyspark shell with the option–remote “https://localhost”

Execute their pyspark shell with the option–remote “sc://localhost”

Set the environment variableSPARK_REMOTE=”sc://localhost”before starting the pyspark shell

Add.remote(“sc://localhost”)to their SparkSession.builder calls in their Spark code

Ensure the Spark propertyspark.connect.grpc.binding.portis set to 15002 in the application code

QUESTION 49
A data scientist wants each record in the DataFrame to contain:
The first attempt at the code does read the text files but each record contains a single line. This code is shown below:

The entire contents of a file
The full file path
The issue: reading line-by-line rather than full text per file.
Code:
corpus = spark.read.text(“/datasets/raw_txt/*”)
.select(‘*’,’_metadata.file_path’)
Which change will ensure one record per file?
Options:

Add the option wholetext=True to the text() function

Add the option lineSep=’n’ to the text() function

Add the option wholetext=False to the text() function

Add the option lineSep=”, ” to the text() function

QUESTION 50
A data engineer is building a Structured Streaming pipeline and wants the pipeline to recover from failures or intentional shutdowns by continuing where the pipeline left off.
How can this be achieved?

By configuring the optioncheckpointLocationduringreadStream

By configuring the optionrecoveryLocationduring the SparkSession initialization

By configuring the optionrecoveryLocationduringwriteStream

By configuring the optioncheckpointLocationduringwriteStream

QUESTION 51
Given the following code snippet inmy_spark_app.py:

What is the role of the driver node?

The driver node orchestrates the execution by transforming actions into tasks and distributing them to worker nodes

The driver node only provides the user interface for monitoring the application

The driver node holds the DataFrame data and performs all computations locally

The driver node stores the final result after computations are completed by worker nodes

QUESTION 52
A Data Analyst is working on the DataFramesensor_df, which contains two columns:
Which code fragment returns a DataFrame that splits therecordcolumn into separate columns and has one array item per row?
A)

B)

C)

D)

exploded_df = sensor_df.withColumn(“record_exploded”, explode(“record”)) exploded_df = exploded_df.select(“record_datetime”, “sensor_id”, “status”, “health”)

exploded_df = exploded_df.select(
“record_datetime”,
“record_exploded.sensor_id”,
“record_exploded.status”,
“record_exploded.health”
)
exploded_df = sensor_df.withColumn(“record_exploded”, explode(“record”))

exploded_df = exploded_df.select(“record_datetime”, “record_exploded”)

QUESTION 53
A developer notices that all the post-shuffle partitions in a dataset are smaller than the value set forspark.sql.
adaptive.maxShuffledHashJoinLocalMapThreshold.
Which type of join will Adaptive Query Execution (AQE) choose in this case?

A Cartesian join

A shuffled hash join

A broadcast nested loop join

A sort-merge join

QUESTION 54
A data engineer is building an Apache Spark™ Structured Streaming application to process a stream of JSON events in real time. The engineer wants the application to be fault-tolerant and resume processing from the last successfully processed record in case of a failure. To achieve this, the data engineer decides to implement checkpoints.
Which code snippet should the data engineer use?

query = streaming_df.writeStream
.format(“console”)
.option(“checkpoint”, “/path/to/checkpoint”)
.outputMode(“append”)
.start()

query = streaming_df.writeStream
.format(“console”)
.outputMode(“append”)
.option(“checkpointLocation”, “/path/to/checkpoint”)
.start()

query = streaming_df.writeStream
.format(“console”)
.outputMode(“complete”)
.start()

query = streaming_df.writeStream
.format(“console”)
.outputMode(“append”)
.start()

QUESTION 55
A data engineer wants to process a streaming DataFrame that receives sensor readings every second with columnssensor_id,temperature, andtimestamp. The engineer needs to calculate the average temperature for each sensor over the last 5 minutes while the data is streaming.
Which code implementation achieves the requirement?
Options from the images provided:

QUESTION 56
What is the risk associated with this operation when converting a large Pandas API on Spark DataFrame back to a Pandas DataFrame?

The conversion will automatically distribute the data across worker nodes

The operation will fail if the Pandas DataFrame exceeds 1000 rows

Data will be lost during conversion

The operation will load all data into the driver’s memory, potentially causing memory overflow

QUESTION 57
A data engineer observes that an upstream streaming source sends duplicate records, where duplicates share the same key and have at most a 30-minute difference inevent_timestamp. The engineer adds:
dropDuplicatesWithinWatermark(“event_timestamp”, “30 minutes”)
What is the result?

It is not able to handle deduplication in this scenario

It removes duplicates that arrive within the 30-minute window specified by the watermark

It removes all duplicates regardless of when they arrive

It accepts watermarks in seconds and the code results in an error

QUESTION 58
What is a feature of Spark Connect?

It supports DataStreamReader, DataStreamWriter, StreamingQuery, and Streaming APIs

Supports DataFrame, Functions, Column, SparkContext PySpark APIs

It supports only PySpark applications

It has built-in authentication

QUESTION 59
A data scientist is working on a project that requires processing large amounts of structured data, performing SQL queries, and applying machine learning algorithms. The data scientist is considering using Apache Spark for this task.
Which combination of Apache Spark modules should the data scientist use in this scenario?
Options:

Spark DataFrames, Structured Streaming, and GraphX

Spark SQL, Pandas API on Spark, and Structured Streaming

Spark Streaming, GraphX, and Pandas API on Spark

Spark DataFrames, Spark SQL, and MLlib

QUESTION 60
A data engineer is working with a large JSON dataset containing order information. The dataset is stored in a distributed file system and needs to be loaded into a Spark DataFrame for analysis. The data engineer wants to ensure that the schema is correctly defined and that the data is read efficiently.
Which approach should the data scientist use to efficiently load the JSON data into a Spark DataFrame with a predefined schema?

Use spark.read.json() to load the data, then use DataFrame.printSchema() to view the inferred schema, and finally use DataFrame.cast() to modify column types.

Use spark.read.json() with the inferSchema option set to true

Use spark.read.format(“json”).load() and then use DataFrame.withColumn() to cast each column to the desired data type.

Define a StructType schema and use spark.read.schema(predefinedSchema).json() to load the data.