Estimate project
← All posts

New York on a Map in Seconds: Geospatial Wizardry with Polar

See how Polars (pola.rs), a fascinating library for Rust, can dramatically speed up your script, illustrated with geospatial indexing of NYC Taxi data.

8 min read
DataRustlibaries

In this post, we will take a look at a fascinating library for Rust fans called Polars (pola.rs). Polars is a DataFrame library built for high performance (if you have used comparable libraries before - Pandas from the Python ecosystem is probably the best known - you will immediately understand what it is and what it is good for). Polars is naturally written in Rust, but it also offers Python bindings, so you can use it in either world. It was built for efficient data processing and data manipulation with outstanding performance.

Unlike Pandas, which relies on NumPy arrays, Polars is based on the Apache Arrow columnar memory format and is tuned for analytics workloads. Its execution model favors lazy execution, which enables query optimization, and it usually outperforms Pandas, particularly in operations such as groupBy, joins and merges, and in filtering large data sets (some benchmarks are available here).

Being written in Rust, Polars uses memory far more efficiently than Pandas. Its columnar storage is more efficient, and its data structures carry much less overhead. In addition, many operations are multithreaded out of the box, unlike in Pandas (unless you deliberately bring in extra Python libraries such as Dask).

Short tutorial on Pola.rs

For this blog post, and to try Polars on a dataset closer to real-world usage, I chose the NYC Taxi dataset, made up of 3 files totaling 5.4 GB of data (the first quarter of the year 2016).

To begin, we have to load the data and combine it into a single lazy dataset:

   let file_paths = vec!["nyc_taxi_data/yellow_tripdata_2016-01.csv",
   "nyc_taxi_data/yellow_tripdata_2016-02.csv",
   "nyc_taxi_data/yellow_tripdata_2016-03.csv",
];

// Create a vector to store individual LazyFrames
let mut lazy_frames: Vec<LazyFrame> = Vec::new();
println!("loading data...");
// Read each CSV file into a LazyFrame
for path in file_paths {
   let lf = LazyCsvReader::new(path)
       .with_has_header(true)
       .with_try_parse_dates(false)
       .with_low_memory(true)
       .finish()?;
   lazy_frames.push(lf);
}

This code loads several CSV files into LazyFrames rather than plain DataFrames. As a result, computations such as parsing, filtering, etc., are not run right away. Instead, we build a logical plan that the Pola.rs engine will optimize and execute later on.

Polars lets us load data either eagerly or lazily. The eager approach is simpler and more direct, which suits small datasets and fast prototyping. It does have downsides, though, such as the lack of optimizations and potentially higher memory usage (Polars will not drop columns that your queries never actually use).

// Create UnionArgs for concat function
let union_args = UnionArgs {
   parallel: false,
   ..Default::default()
};

// Concatenate all LazyFrames vertically
let combined_lf = concat(lazy_frames, union_args)?;

// Collect data
let df = combined_lf.collect()?;

With the data from several sources loaded into our lazy frames, we can merge them into one combined data frame. To do that, we use the UnionArgs struct, which tells Polars how to concatenate the frames.

Right, all our data is now collected and ready for processing. For this short tutorial, I chose to look at just 2 columns, dropoff_latitude and dropoff_longitude, which show where NYC taxis let passengers out. There are certainly other interesting columns, such as tip amount, totals, etc., but we are after location so that we can experiment with it using the H3 library and display something on the map.

We select the columns and attempt to obtain a typed f64 representation of each.

let lat_series = df.column("dropoff_latitude")?;
let lng_series = df.column("dropoff_longitude")?;

let lat_vec = lat_series.f64()?;
let lng_vec = lng_series.f64()?;

Our data is now ready to be manipulated. In this simple example, we will turn the lat/lng points into H3 cells and count how many records land in each H3 cell.

// HashMap to count frequencies of each H3 cell
let mut h3_counts: HashMap<String, u32> = HashMap::new();

// Convert lat/lng to H3 cells and count occurrences
for i in 0..lat_vec.len() {
   if let (Some(lat), Some(lng)) = (lat_vec.get(i), lng_vec.get(i)) {
       let latlng = LatLng::new(lat, lng).expect("Invalid coordinates");
       let h3_index = latlng.to_cell(Resolution::Nine);
       let h3_str = h3_index.to_string();

       // Store the H3 index as a string
       h3_cells.push(h3_str.clone());

       // Increment count for this H3 cell
       *h3_counts.entry(h3_str).or_insert(0) += 1;
   } else {
       h3_cells.push(String::new()); // Handle missing values
   }
}

To finish this simple data manipulation, we append the h3_index column to our data frame, so it can be used later when generating the map with JavaScript.

// Add H3 cell indices to the dataframe
let h3_series = Series::new(PlSmallStr::from_str("h3_index"), &h3_cells);
let mut df_with_h3: DataFrame = df.clone();
df_with_h3.with_column(h3_series)?;

With the index in place, we can fill a simple html page using leaflet.js and h3-js.umd.js to display our polygons on a map of New York City.

I also added a frequency heat map, so it is easy to spot the most frequently used drop-off locations (marked in red).

The H3 frequencies are turned into json and inserted into a js object that the page uses later:

// Convert HashMap to a JSON object
let h3_frequency_json = h3_counts
   .iter()
   .map(|(h3, count)| format!("\"{}\":{}", h3, count))
   .collect::<Vec<_>>()
   .join(",\n        ");

We also perform a few basic calculations needed for the legend, such as the highest frequency value across all generated h3 polygons:

let max_count = h3_counts.values().max().cloned().unwrap_or(1);

The script writes the generated html to a new file, and the result is a simple leaflet map on an html page, with tooltips (showing frequency) and a legend explaining the heatmap colors.

Speeding things up

So far so good, and Pola.rs offers a great developer experience, but it looks like things could run a little faster. With the code as it stands, the entire generation takes roughly 5 minutes (00:05:00.105). Considering it processes 5.4 GB of data, that is still impressive, but there is room for improvement.

The first step is to drop every column we will not need as early as possible, before we call .collect() on our lazy data frame.

Lazy data frames in Polars come with some optimizations applied automatically. Even so, getting rid of unused columns as early as possible is always good practice.

// Read each CSV file into a LazyFrame
for path in file_paths {
   let lf = LazyCsvReader::new(path)
       .with_has_header(true)
       .with_try_parse_dates(false)
       .with_low_memory(true)
       .finish()?;
   let lf_lat_lng = lf.select([
       col("dropoff_latitude"),
       col("dropoff_longitude"),
   ]);
   lazy_frames.push(lf_lat_lng);
}

This simple trick saved us roughly 16 seconds, bringing the total end-to-end run time of our script to 00:04:43.905

Attentive readers have probably spotted that our union_args struct had the parallel: false field set when we merged the lazy frames into one entity. That was intentional: switching this flag to true gives us a further 19 seconds of improvement, because the Polars engine can then use several cores of our machine for this operation.

UnionArgs is only a basic example; when working with Polars, keep an eye out for other operations that can take advantage of parallelism and make your code much faster, for example:

1. .par_apply() on ChunkedArrays
let result = some_chunked_array.par_apply(|v| expensive_fn(v));
2. .with_streaming(true) on joins and groupbys
lazy_df
    .join(other, ...)
    .with_streaming(true)
3. CsvReader::with_n_threads(n) (Eager only)
let df = CsvReader::from_path("file.csv")?
    .with_n_threads(4)
    .finish()?;

For most internal operations such as .groupby(), .join(), .agg(), etc, Polars runs in parallel by default unless you turn it off explicitly, but it is always worth checking the docs and enabling parallel options where they exist.

With parallel turned on in our union args, we are down to 00:04:24.839

When you run the script on your own, you will soon see that converting the lat/lng pairs into H3 cells takes up most of the time. That is because we iterate over every record in the data frame with a plain for loop, which can certainly be improved.

If you have some experience with Polars, you probably know this already - under the hood, Polars relies on the Rayon data-parallelism library. Instead of a plain for loop that processes records one at a time, we can easily use Rayon to parallelize our own code and speed it up significantly.

Rayon is a high-level abstraction on top of threads. It runs parallel for-loops and map/filter/collect chains across multiple CPU cores. It is also very safe and pleasant to use: no manual thread management, no Mutex’es, no RwLocks, no unsafe and no Pinning.

The minor complication in parallelizing our code is that we count the occurrences of each cell along the way. To handle this, we have to divide the computation into two separate steps: one that converts the data into cells and another that counts the occurrences.

// Combine into (index, lat, lng) tuples and process in parallel
let results: Vec<Option<String>> = (0..lat_vec.len())
   .into_par_iter()
   .map(|i| {
       match (lat_vec.get(i), lng_vec.get(i)) {
           (Some(lat), Some(lng)) => {
               let latlng = LatLng::new(lat, lng).ok()?;
               let h3_index = latlng.to_cell(Resolution::Nine);
               Some(h3_index.to_string())
           }
           _ => None,
       }
   })
   .collect();

Then we count the occurrences and gather the data into h3_cells as before:

let mut h3_cells:Vec<String> = Vec::with_capacity(results.len());
let mut h3_counts: HashMap<String, u32> = HashMap::new();

for maybe_h3 in results {
   if let Some(h3_str) = maybe_h3 {
       *h3_counts.entry(h3_str.clone()).or_insert(0) += 1;
       h3_cells.push(h3_str);
   } else {
       h3_cells.push(String::new());
   }
}

We now stand at: Elapsed time: 00:00:50.339, which is 3 minutes 24 seconds faster than the previous result and 4 minutes and 10 seconds faster than where we started.

Final thoughts

With this blog post, I wanted to demonstrate that:

  • Pola.rs is enjoyable and can be used much like Pandas, but with considerably faster, type-safe Rust scripting.
  • Whatever data manipulation library you choose, consider possible optimizations right from the beginning – they can make your script far faster and sometimes are what makes it run at all.
  • h3 is worth exploring further - it is a powerful geospatial indexing system that makes handling location data surprisingly elegant and efficient.