#GIT ANNEX DROPBOX CODE#
#GIT ANNEX DROPBOX FREE#

We created a small fork of Git that measured timing of operations like git status and git pull, had it automatically provisioned and installed on developer machines, and munged $PATH so that developers use our Git over the ones installed by the system or Homebrew.Īt the time, git status took over two seconds on average. Additionally, we had to measure performance. In order to tackle local Git performance, we needed to control the version of Git that developers on macOS used. In practice, most of our server engineers use macOS, and that’s where we focused our efforts. For example, an engineer working on the Dropbox desktop client might work on Windows, and some server engineers prefer Linux.

#GIT ANNEX DROPBOX FREE#

MacOS is the supported development platform for Dropbox engineers, but they’re free to use other platforms as they see fit.

Fortunately, we didn’t have workflows that depended on this, and we set up a pre-receive hook to limit the size of new files pushed to our repositories and prevent regressions. To start off with, checking in large files severely impacts the performance of many operations in Git. So in late 2017, we started investigating the various options to speed up Git for our users. Common operations like git status were getting slower with time. Specifically, it seemed to degrade linearly with the number of files being added to the repository. This worked for us for a few years, but as expected, Git performance started degrading.

#GIT ANNEX DROPBOX CODE#

Eventually, we’ve seen even more benefits, like simple code sharing, easy large scale refactoring, good operability with monorepo centric build tools like Bazel, and simple automatic bisects and reverts. This merge, combined with various other initiatives to improve testing infrastructure and quality, helped us keep master much stabler with much less work, and smoothed our release processes. The combined repository size was not that large (~50,000 files), and we estimated that Git performance would be acceptable for at least a few years. Eventually we realized it would be simplest to merge all relevant repositories.

This provided a global ordering on changes and test results, and helped narrow down the repository that caused a breakage. We either needed a polyglot tool (or a set of tools per language) to pin versions for each dependency directly, a repository-based pinning mechanism like git subtree, or to merge our repositories into one.įor a while, we had a “super repo” that received a check-in every time one of the server-related repositories changed (via a Git pre-receive hook). To solve this, we needed a single commit identifier to reproducibly determine the state of the code we were testing. Due to the indeterminate nature of these test failures, engineers didn’t trust CI test results and inspect failures, which led to more problems. It caused a lot of work for the release engineers and made the release process slow and inconsistent. Debugging test failures, tracking down changes, and fixing the build before the daily release was a painful process. The large frequency of changes meant there would be multiple failures a week. These tests often failed when the service’s code changed, and since our continuous integration (CI) process simply ran all tests at HEAD for each repository (with no notion of pinning), it was hard to diagnose the root cause of problems. Consequently, developers would write integration tests for these services in the monolith’s repository. In practice, this meant the monolith was the largest and often the only consumer of a smaller service or some code in a separate repository. For example, Magic Pocket, our custom block storage system was built separately from day one, while our metadata store, Edgestore, started off as a simple client side Python library that eventually spun off into a sophisticated service. Over time, we extracted targeted components from the monolith into separate services, but a large number of our engineers still contribute to the monolith. The backend business logic for Dropbox at the time lived mainly in a monolithic Python web application, with infrastructural components built independently. So, over the course of a few Hack Weeks, a small group of engineers migrated many of our repositories from Mercurial to Git, and planned to migrate the rest. More importantly, Git had become an industry standard tool that most new engineers had already used. But around 2014, we tested and found that we’d have better local performance with Git. Originally, our code was distributed across several dozen Mercurial repositories.

Git annex dropbox