Compare Spark RDDs, DataFrames, and SQL Performance Gains

SparkJobs

+---------+---------+---------------------+----------+
| job_id  | user_id | submit_time         | status   |
+---------+---------+---------------------+----------+
| 1001    | 17      | 2023-10-01 10:15:00 | running  |
| 1002    | 21      | 2023-10-01 10:20:00 | failed   |
| 1003    | 17      | 2023-10-01 10:25:00 | success  |
| 1004    | 42      | 2023-10-01 10:30:00 | running  |
+---------+---------+---------------------+----------+

Scenario

Discussion of distributed data processing tools used on big-data projects.

Question

Compare Spark RDDs, DataFrames, and Spark SQL. What performance gains come from Spark’s lazy evaluation model? When would you choose each abstraction?

Hints

Highlight memory management, query optimization, and developer ergonomics.

Spark Job Counts and Submit-Time Range by Status

You are given SparkJobs, a table of submitted Spark jobs.

Return one row per job status with:

• status
• job_count: number of jobs in that status
• first_submit_time: earliest submit_time, formatted as YYYY-MM-DD HH24:MI:SS
• last_submit_time: latest submit_time, formatted as YYYY-MM-DD HH24:MI:SS

Order the result alphabetically by status.

Table SparkJobs

column	type	constraints
job_id	INTEGER	PRIMARY KEY, NOT NULL
user_id	INTEGER	NOT NULL
submit_time	TIMESTAMP	NOT NULL
status	VARCHAR(10)	NOT NULL

Sample data:

job_id	status	submit_time	user_id
1001	running	2023-10-01 10:15:00	17
1002	failed	2023-10-01 10:20:00	21
1003	success	2023-10-01 10:25:00	17
1004	running	2023-10-01 10:30:00	42

Expected output

One row per status with job count and formatted earliest/latest submission times.

first_submit_time	job_count	last_submit_time	status
2023-10-01 10:20:00	1	2023-10-01 10:20:00	failed
2023-10-01 10:15:00	2	2023-10-01 10:30:00	running
2023-10-01 10:25:00	1	2023-10-01 10:25:00	success

Hints

1. Group by status.
2. Use COUNT(*), MIN(submit_time), and MAX(submit_time).
3. Use TO_CHAR for PostgreSQL timestamp formatting.