Our Approach to Google Drive Backups Across Our Workspace Organization
Have you ever thought about the danger of losing access to your own data? To reduce that danger, we set up Google Drive backups covering our whole domain.
Most readers of this blog probably know about vendor lock-in – when services are integrated so tightly that swapping one out becomes almost impossible. But have you ever thought about the danger of being locked out of your own data? Google pinky promises not to use your Workspace data for advertising or training. Even so, because of legal requirements, Google will probably scan your files for intellectual property (IP) infringement or CSAM. As with any automated system, false positives are inevitable. If your company depends on data stored in the cloud, losing access for days (or even weeks!) because of a suspension is a risk you simply can’t take.
At our company, we rely on Google Workspace because it offers a smoother and more intuitive collaboration experience than Microsoft 365. We’re just as exposed to the risk of being locked out as anyone. One misclassification or automated flag could get our access revoked. To reduce that risk, we set out to build Google Drive backups covering our whole domain.
Incidentally, did you know you can start new docs, sheets, and meetings by typing docs.new, sheet.new, meet.new into the address bar? A rather clever use of a TLD.
Current state
Before we committed to reinventing the wheel with a tool of our own, we looked into the solutions already out there:
- Google’s Data Export – Google Workspace customers can ask for a data export through the Data Export tool. It behaves much like Google Takeout and exports data from a broad set of Google services, well beyond Google Drive alone. Once you request it, the exported data lands in a Google-managed Cloud Storage bucket, from which you have 60 days to copy it wherever you like. The only catch is that an export can be requested at most once every 30 days, and it’s a manual process. Still, it’s a sensible starting point for companies that can’t afford to build their own backups.
- rclone – rclone is a mature project that lets you work with many remote storage systems, Google Drive included, as though they were local file systems. It copes quite well with Drive-specific features such as shortcuts and exports from Google Docs, Sheets, and other tools. However, treating an object storage system like Google Drive as an ordinary file system has its limits. A major drawback is that rclone can’t deal with several files sharing the same name and location – it just regards them as duplicates. At most, it can “deduplicate” them by removing the extras. This stems from the fact that Google Drive works as object storage with no real folder structure; folders are only simulated in the interface, and files are identified by unique IDs rather than by name and path. Another big downside is that rclone is configured separately for each drive.
Since Google’s Export Tool can’t be automated and rclone would have required us to write custom tooling to generate its configuration automatically, we chose to build our own backup solution.
What does it take to create a custom backup tool?
Listing drives
Step one was getting a full list of personal and shared drives. That’s fairly easy, because Google offers both the Google Drive API and the Admin SDK Directory API. Shared drives have unique IDs that must be retrieved from the Drive API, while personal drive IDs are the same as the user’s primary email. Normally, an application built on these APIs is intended for end-users (picture an alternative UI for drive, e.g. a fancy file manager), and you get permissions by sending the user to Google’s OAuth screen. For our scheduled backups, though, the permissions had to be granted in advance and apply to the whole domain. Google anticipated this: you can manage Domain Wide Delegation from the Admin Dashboard. It’s thoroughly documented by Google support as well as in our application’s README.md. Permissions are referred to as scopes, and the Client ID is the Unique Name of the GCP service account (a 21-characters long number, not the email that’s usually mentioned). In our case, we needed these scopes:
- https://www.googleapis.com/auth/admin.directory.user.readonly
- https://www.googleapis.com/auth/drive.readonly
Fetching file list
Google Drive is object storage, and files are identified by their IDs. Before you can download a file, you need its ID. Google’s Drive SDK has a method for listing files, so all it takes is iterating over the paginated API responses. You can also choose which metadata fields you want. In addition to id and name, we wanted md5Cheksum, mimeType, and exportLinks to determine whether a file could be downloaded or needed exporting (more on that later), parents to construct the file path, shortcutDetails to deal with shortcuts, and permissions/permissionIds in case we ever needed to restore files.
How file permissions work depends on the type of drive. For personal drives, the permissions field gives you exactly what you’d expect – the user’s email and access level. For files in shared drives, however, permissions isn’t available; only permissionIds is. You then need to call a separate endpoint with the permission ID to get the actual permissions, including user email and access level. Although the list endpoint can be called with a page_size of 1000 to fetch a batch of files in one response, firing off a thousand (or more) requests (for every batch!) just to get permissions would be a nightmare. We solved this by simply caching the responses and building a dictionary of known permissionIds
Remember how I mentioned that folders in Google are simulated? Files don’t know their own path. The parent field is the only one that relates to the path. Because we wanted our backups to keep the folder structure, we had to construct each file’s path ourselves by walking up its parent tree. Looking up a file’s parent by ID in the array returned by the Google API, though, has O(n) time complexity. With some drives holding tens of thousands of files, building paths would take forever. Luckily, the fix is simple: file IDs are unique. We converted the file array into a set keyed by file ID. This lets us fetch file information by ID in O(1), which speeds things up considerably.
Downloading and exporting
Downloading binary files (those that have an md5Checksum) is pretty simple – you just call the get_media method with the file ID and save the response to a file. To keep memory usage down, though, you may want to download the file in chunks.
Non-binary files are somewhat trickier. For shortcuts (recognizable by the application/vnd.google-apps.shortcut mime type), we decided to write a .txt file containing the path to the original source instead of downloading the original file again and again (once per shortcut). Other Google-specific applications such as Docs, Sheets, and Presentations can be exported to the formats listed in this table using the export_media method. At least in theory. export_media is capped at 10MB, and a document containing just a few images will easily go over that. We had to fall back on the file’s exportLinks field. It’s normally used in the Google Drive web-ui and is simply a link. Because this method isn’t part of Google’s SDK, we had to implement our own authentication flow (luckily, the SDK exposes the bearer token) and our own chunker. The limits of exportLinks aren’t documented, but we haven’t hit any problems with it so far. The one caveat is that it could be depreciated at any moment, since it isn’t part of the Drive v3 API.
Oh, the lovely duplicates
Because Google Drive identifies files by unique IDs, it was only a question of time before we ran into files sharing the same name and path. This was a serious problem, as our implementation downloaded files using multiple threads (and yours should as well!). With that setup, one thread might begin writing to a local file right after another thread had written a chunk of a different file with the same name, causing overwrites and corrupted data. We fixed this by introducing a data structure with O(1) lookup time – a set recording every locked (in-use) path. Before downloading a file, we checked whether its path was locked. If it wasn’t, we locked it, guaranteeing an uninterrupted download. If the path was already taken, we added the file ID and a random number to the filename to prevent clashes.
One important point: make sure your data structures are thread-safe. Otherwise the set itself could end up corrupted, just like the files.
Compressing
After every file had been downloaded, compressing them was simple. We benchmarked two compression algorithms: lz4 and pzstd.
For roughly 700GB of data:
- lz4 (which always runs on a single thread) needed about 114 minutes and produced a 599GB archive (a compression ratio of 1.16x).
- pzstd (on 2 cores) finished in 88 minutes – 22% faster than lz4 – and produced a 587GB archive (a compression ratio of 1.19x).
As 700GB was only one of our drives, and a few hours of compute cost less than keeping large volumes of data for several months, we went with pzstd.
Deployment and first gotcha
Because we wanted the data in S3, it made sense to deploy the application inside AWS to avoid paying for data transfer. We decided to run the backup application as an ECS Task triggered periodically by CloudWatch. With the full backup taking about 6 hours when processing multiple drives, Fargate was a much better fit than an always-on EC2 instance.
Our first implementation downloaded all the data up front, so the container needed a scratch disk of at least several terabytes. Fortunately, since January 2024, AWS has let you create an EBS volume at deployment time (when invoking the RunTask API method). You set this up through the volumeConfigurations parameter, next to other settings such as cluster and taskDefinition.
We then hit an unexpected obstacle, though: more than a year on, the official Terraform aws_scheduler_schedule resource still offered no volume configuration in the ecs_parameters field, even though it’s meant to reflect the RunTask API.
That left us in a tough spot. We could either abandon IaC or find a way around the 200GB-max ephemeral_storage set at the task definition level. Being an IaC-first company, dropping Terraform was out of the question. We hurried to change our code so that each file would be uploaded to S3 and removed from local storage right after being downloaded. This drastically cut the disk space needed at any moment down to just the files currently being processed.
The code
We’re open-sourcing the code of our backup application. It’s written in Python, supports black & whitelisting of drives, processes drives in parallel, and downloads using multiple threads. You can also choose whether Shared-with-me files should be included. At the moment it exports Google-specific application files to standard Microsoft Office files, but with some help from the community, it may soon handle other formats too. You can also decide whether to download every file and then compress them or, as we do, upload each file as soon as it’s downloaded. Any errors are logged and uploaded to S3 together with the files as well.
Conclusion
To sum up, although Google Workspace provides a powerful and convenient environment for collaboration, the possibility of losing access to essential data is a real concern. Existing backup options, such as Google’s Data Export and rclone, have shortcomings that make them unsuitable for fully automated, frequent backups. By building our own backup tool, we gained more control and efficiency and became more resilient to potential disruptions. Our open-source solution offers a scalable and flexible way to protect Google Drive data, helping organizations reduce risk and keep uninterrupted access to vital information.