The NYC Taxi & Limousine Commission publishes monthly trip records — pickup/dropoff times and locations, distances, fares, passenger counts — and January 2017 alone is millions of rows across yellow and green cabs. It’s a good dataset for learning what Spark is actually doing under the hood, because the transformations you’d reach for instinctively (group-by hour, join lookup tables, compute averages) are exactly the ones that expose Spark’s lazy-evaluation model.
Setting up Spark from scratch
Running Spark outside a managed cluster means owning the JVM setup yourself:
!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q https://dlcdn.apache.org/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz
!tar xf spark-3.1.2-bin-hadoop3.2.tgz
!pip install -q findsparkOnce findspark locates the install and SparkSession is up, the DataFrame API looks almost identical to pandas — which is exactly where the intuition gap starts. A pandas .groupby().mean() runs immediately. The Spark equivalent builds a logical plan and does nothing until an action (.collect(), .show(), .write()) forces evaluation.
Measuring instead of guessing
For every transformation I cared about — average speed by hour of day, most common trip routes, fare distribution by payment type — I timed the wall-clock execution, checked the amount of data shuffled, and computed a rough processing-speed figure (rows/sec). That habit surfaces two things pandas users don’t usually have to think about:
- Which operations trigger a shuffle. A
groupByon high-cardinality keys (pickup location × hour) redistributes data across partitions — visibly slower than agroupByon a low-cardinality column like payment type. - Where lazy evaluation hides cost. Chaining five
.withColumn()calls looks cheap until the first.collect(), at which point all five run at once. If you’re debugging performance, instrument at the action, not the transformation.
The takeaway
The biggest speed wins didn’t come from clever Spark tricks — they came from reducing the amount of data being shuffled before the expensive stage: filtering early, projecting only the columns actually needed, and pre-aggregating where possible. Spark rewards exactly the same discipline that makes SQL queries fast; it just makes the cost of skipping that discipline much more visible, because the row counts are large enough that a bad plan takes minutes instead of milliseconds.
Full notebook: PySpark_Applications.md