Zipher | When is Photon Worth It?

TL;DR – This blog is a detailed dive into our experiments testing Photon on different data sets, in order to determine when Photon is cost effective. On small and medium-sized jobs, it’s almost never worth it. Only on certain types of larger jobs does Photon offer meaningful benefits.

One of the major problems global data teams are struggling with is query performance. Faster processing can significantly reduce compute time and costs, but the impact goes beyond infrastructure bills: query latency directly affects the speed at which teams can deliver insights; In business-critical dashboards and reporting pipelines, slow queries can lead to missed SLAs and delayed decisions.

For analytics and data science teams, performance bottlenecks translate into longer development cycles, reduced productivity, and frequent context switching. That’s why in June 2020, Databricks introduced Photon, a next-generation query engine designed to accelerate SQL and DataFrame workloads. Photon became generally available in July 2022 and is supported on AWS, Azure, and GCP.

Photon promises speed, but it comes with a cost multiplier that can catch teams off guard. Databricks positions Photon as a drop-in accelerator for SQL and DataFrame workloads, especially for large datasets. But is it really worth the premium?

In this blog, we wanted to explore when it is useful to use Photon. To do so we benchmarked Photon across real-world workloads and scales to understand when it pays off, when it doesn’t, and what trade-offs to consider. We also break down how it works, how to enable it, what kinds of jobs benefit most from using it, and whether there are other ways of achieving comparable results.

What Is Databricks Photon?

Photon is a next-gen query engine built in C++ to replace Spark’s traditional JVM-based execution layer. It’s optimized for modern hardware, like AVX-512 instruction sets and NVMe SSDs, and uses vectorized execution to process data in batches instead of row-by-row.

That means aiming for faster processing for SQL, Spark DataFrame, and Koalas workloads, particularly for queries with Large joins, Aggregations, Filtering and sorting, and Complex window functions.

Databricks Photon

Databricks claims Photon can achieve up to 12× speedups on certain workloads. For Databricks SQL Warehouses, Photon is the default engine with no extra DBU cost. However, for Job Compute clusters, enabling Photon consumes 2.5× to 2.9× more DBUs depending on your cloud provider (Azure/GCP: 2.5×, AWS: 2.9×).

How to Enable Databricks Photon

Enabling Photon depends on the environment:

SQL Warehouses & Serverless Compute – Photon is enabled by default. No changes needed.

Interactive & Job Clusters – Photon is available in runtimes 9.1 LTS and above.

Databricks UI: To enable it, check the “Use Photon Acceleration” box under the cluster’s Performance tab.
API provisioning, change the “runtime_engine” field in the cluster configuration to “PHOTON”.

Photon Settings

Note: ML runtimes (e.g., 15.2 ML) may require manual Photon activation.

Benchmark Setup: Comparing Photon vs. Non-Photon Jobs

To evaluate Photon performance and cost trade-offs on Databricks, we designed a set of benchmarks spanning multiple workload types and data scales. Our testing covered a range of analytical queries, including joins, filters, aggregations, and time-based operations, representative of real-world ETL and BI patterns.

We generated synthetic datasets at three distinct scales to assess how Photon behaves under different data volumes and complexity levels. Each test was run with identical cluster configurations except for the Photon toggle, ensuring fair comparisons while capturing how performance and cost vary with workload size, query type, and cluster settings.

All synthetic data generation scripts, benchmark queries, cluster configs, and results are available in our GitHub repository if you’d like to reproduce these benchmarks.

Cluster Configuration

We ran performance (job completion time) and cost benchmarks using Databricks Job Clusters with identical specs, toggling only the Photon setting. We used on-demand instances to create a benchmark that won’t be impacted by sporadic spot termination events. Here is the setup we used:

Runtime: 16.4 LTS (Spark 3.5.2)
Worker type: m6id.large (Memory 8 GiB, Disk Space 118 GiB, 2 vCPU)
Driver type: Same as worker
Enabled Autoscaling:
- Min number of workers: 1
- Max number of workers: 4

Synthetic Datasets

We generated datasets at three scales (A, B and C, see table below) and stored them in Delta format. Each dataset was used to run a consistent suite of three benchmark queries designed to mimic common analytical workloads:

Query 1: Total spending per country (join + group by + sum)
Query 2: Top 10 Products by Total Sales (join + group by + sum + limit)
Query 3: Top 5 Spenders per Country (join + group by + window function + rank + filter)
Query 4: Average high-value transaction per product category (join + filter + aggregation)
Query 5: Monthly sales trend by category (join + date functions + group by)

These queries included joins across user, product, and transaction tables, aggregation logic, and time-based partitioning to simulate typical ETL and BI workloads. We ran queries in SQL, covering joins, aggregations, filters, and window functions.

	Users	Products	Transactions
Scale A	100M	1M	500M
Scale B	10M	10M	500M
Scale C	10M	1M	5B

Databricks Photon Performance: Key Findings

Databricks Photon Experiment

Scale A: Photon Is Faster, But Costlier

The expected pattern of high price but shorter job completion times is observed in queries 1,2 and more substantially in query 4, where Photon reduced runtime from 6.7 to 3.7 minutes but increased the cost more than 4x, from $0.07 to $0.3. On the other hand, query 5 had both lower cost and shorter job completion time.

Databricks Photon Results B

Scale B: Faster, But Sometimes Much More Expensive

While Photon consistently reduced runtime across most queries, the cost impact was mixed. In query 2, Photon slightly improved execution times but increased the cost more than 6X, from $0.02 to $0.13. In query 1, the runtime dropped from 5.7 minutes to 3.6 minutes, keeping roughly the same cost. In queries 3 and 4, Photon delivered both faster results and lower costs. But in query 5, runtime dropped from 5.5 to 4.1 minutes, but the cost more than tripled, from $0.07 to $0.24.

Scale C: Photon Starts Paying Off

In high-scale scenarios, Photon consistently improved execution times, cutting runtimes by 40–70% across all queries. In query 1, Photon ran ~50% faster while also lowering cost. In query 2, Photon reduced job duration in nearly half the time and at less than half the cost. In query 5, Photon cut runtime from 24 minutes to 8 minutes, while reducing cost from $0.42 to $0.09. However, while Photon lowered the costs from $0.19 to $0.17 in query 4, it also increased the runtime from 6 minutes to 7.2 minutes.

Note on Testing Methodology: These tests were performed with autoscaling enabled. In future experiments, we plan to repeat the analysis with fixed cluster sizes and a larger variety of data scales to better isolate the impact of Photon on performance and cost.

Databricks Photon on Moderate Workloads: Bronze-to-Silver Transformations

After testing Photon on large-scale analytical queries, we turned to a more common scenario: transforming data from Bronze to Silver.

In contrast to our previous tests involving large-scale datasets, we intentionally worked with smaller, moderate-sized datasets to evaluate Photon’s impact on typical pipeline workloads. Our goal was to see how Photon performs when tasks are not CPU-intensive, highlighting that it may offer little benefit for smaller or I/O-bound jobs.

Data & Transformations Preparation

We used two synthetic datasets simulating real-world data volumes that were stored into the bronze layer:

Small dataset: 100,000 users and 1 million transactions
Large dataset: 1 million users and 10 million transactions

Each dataset included a users table and a transactions table, containing randomized user names, countries, transaction amounts, and dates.

To evaluate performance under different processing scenarios, we created 3 variations of transformation logic for converting bronze data into the silver layer:

Basic enrichment and filtering
Enrichment with categorization and windowing
Full enrichment with rolling metrics and dense ranks

Each dataset was processed with each transformation variation, with and without Photon, resulting in a total of 12 distinct runs per scenario.

Photon Performance: Key Findings

Small Dataset Photon

Small Dataset

For small workloads, Photon is not cost-effective. Photon slightly reduced execution times for T2 and T3, while making no change for T1. However, cost was significantly higher with Photon in all cases – in T1, the cost increased by more than x5, from $0.02 to $0.11, and in T3, the cost increased by x7, from $0.02 to $0.14.

Large Dataset Photon

Conclusion – When Should You Use Databricks Photon?

Cost-effectiveness depends on data scale. For small, daily workloads, performance gains did not justify the significantly higher costs. At medium scale, Photon accelerated queries but often at a steep increase in cost, making the trade-off less compelling. For large-scale workloads with high CPU demands, results were inconclusive.

Ultimately Photon is best for transform-heavy jobs, scenarios where execution time is a bottleneck, and BI dashboards or SQL warehouses that need fast responses. It’s not ideal for small/medium sized jobs, simple append-only pipelines, Python heavy pipelines, or low priority tasks. Photon also doesn’t support UDFs, Datasets, RDDs, or many Python operations, and structured streaming support is limited to stateless queries with certain sinks. Unsupported operations silently fall back to standard Spark, meaning Photon might not actually run, even when you’re paying extra for it.

Because Photon is not always the correct choice, we built Zipher to dynamically enable or disable Photon based on the run level workload characteristics. Zipher analyzes Spark logs, job code, and cluster configurations – including worker and driver node types – to determine whether Photon will deliver meaningful speedups relative to its cost.

If Photon is unlikely to help, it stays off. When a job has tight SLAs or benefits from SQL acceleration, we enable Photon automatically to meet performance goals while avoiding unnecessary spend.

When is Photon Worth It?

What Is Databricks Photon?

How to Enable Databricks Photon