Pinterest has significantly improved its Android development pipeline by reducing Android CI build times by more than 36 percent. The improvement comes from a shift to runtime-aware test sharding and the rollout of a custom internal testing platform designed to eliminate bottlenecks and uneven test execution.
The changes target a common pain point for large mobile teams: slow and unpredictable continuous integration feedback that delays development and increases engineering toil.
Why Android CI Pipelines Were Slowing Down
Before the optimization, Pinterest relied on Firebase Test Lab to run Android end-to-end tests. Tests were split by package name, which often caused severe workload imbalance. As a result, one slow shard routinely delayed the entire build, even when other shards finished much earlier.
In addition, emulator startup time and flaky test behavior introduced further delays, making build durations inconsistent and difficult to predict. These issues compounded as the Android test suite continued to grow.
Moving Away from Vendor-Managed Test Sharding
After evaluating third-party alternatives, the engineering team decided that external platforms could not deliver the level of control they needed. This led to the creation of PinTestLab, an internal testing system running Android emulators on EC2.
By managing the infrastructure directly, Pinterest gained full visibility into scheduling, environment setup, and test orchestration. This shift laid the groundwork for deeper CI optimization.
Runtime-Aware Test Distribution
The key breakthrough came from adopting a runtime-based sharding strategy. Instead of grouping tests by count or package, the system uses historical execution data stored in Pinterest’s Metro test management service.
Tests are sorted by expected runtime and distributed using a greedy assignment algorithm. This approach minimizes the difference between the fastest and slowest shards, dramatically reducing tail latency.
As a result, the slowest shard runtime dropped by roughly 55 percent, while overall Android CI build times fell by about nine minutes per run.
Balancing Speed and Reliability in CI
To ensure reliability, the sharding logic runs within Buildkite and gracefully falls back to round-robin distribution if runtime data becomes unavailable. This design prevents CI outages while still delivering consistent performance improvements under normal conditions.
The platform also prioritizes stability by reducing emulator churn and limiting unnecessary test retries, further smoothing CI execution.
What This Means for Mobile Engineering Teams
Pinterest’s results highlight a broader trend in CI optimization: historical execution data is one of the most effective tools for improving parallel test performance. By aligning workloads based on real runtime behavior, teams can unlock deterministic speed gains without adding complexity.
The case also shows why some organizations are moving away from vendor-managed CI abstractions. Custom orchestration enables deeper optimization, especially for large Android codebases with diverse and uneven test suites.
As mobile applications scale, strategies like runtime-aware sharding are becoming essential for keeping CI fast, predictable, and developer-friendly.
Read also
Join the discussion in our Facebook community.