portage-cli

I’m experimenting with an alternative implementation of portage in Rust.

It is quite different from pkgcraft even if both projects have similar targets.

Main differences:

• separation of concerns: portage-cli is based on a set of layered crates, that let people reuse specific parts of it. pkgcraft on the other hand is a bit monolithic right now.
• pure-rust: pkgcraft for correctness wraps bash, portage-repo leverages brush fixing issues as they are found.
• focus on different tooling: pkgcraft offers already extensive linting support via pkgcruft, portage-cli offers an emerge workalike (em) with a pubgrub-based dependency resolver.

Some numbers

Some comparisons with the current implementations.

metadata regen

All 3 tools are able to do that and we have a script to do that in a simple way.

GENTOO_REPO=/var/db/repos/gentoo 
EM=target/release/em 
PK=../pkgcraft/target/release/pk 
INCLUDE_EGENCACHE=1 
ITERATIONS=1 
./benchmarks/scripts/compare-regen.sh 8 16 20 24

Enter fullscreen mode Exit fullscreen mode

tool j run real user sys files em 8 1 0m18.008s 2m12.733s 0m9.387s 31880 egencache 8 1 5m24.107s 0m0.003s 0m0.004s 31880 pk 8 1 0m48.226s 5m27.982s 1m8.484s 31880 em 16 1 0m10.115s 2m17.674s 0m11.647s 31880 egencache 16 1 4m36.731s 0m0.000s 0m0.007s 31880 pk 16 1 0m28.172s 6m0.818s 1m22.995s 31880 em 20 1 0m9.593s 2m27.314s 0m13.700s 31880 egencache 20 1 4m35.749s 0m0.000s 0m0.006s 31880 pk 20 1 0m23.707s 5m55.650s 1m33.241s 31880 em 24 1 0m9.229s 2m39.258s 0m16.244s 31880 egencache 24 1 4m56.389s 0m0.007s 0m0.000s 31880 pk 24 1 0m23.367s 5m54.775s 1m44.188s 31880

Here we see that the rust implementation are way faster and portage-repo parallel visit via jwalk has an edge.

dependency resolution

Only emerge and em have a resolver wired in and the search support.

RUNS=2 EM=target/release/em ./benchmarks/bench-em-vs-emerge.sh

Enter fullscreen mode Exit fullscreen mode

Benchmark em (mean ± σ) emerge (mean ± σ) speedup (em vs emerge) firefox -p 0.917 s ± 0.082 s 3.675 s ± 0.000 s 4.01× libreoffice -p 0.918 s ± 0.026 s 3.953 s ± 0.013 s 4.31× multi (5 pkgs) -p 1.041 s ± 0.025 s 4.724 s ± 0.196 s 4.54× gcc -s 0.154 s ± 0.011 s 5.175 s ± 0.006 s 33.63×

Since it is fun to watch, here an example of em dealing with some use changes compared with emerge.

The pubgrub solver wiring covers already plenty of the niceties of emerge, but I need to test it more.

Progress so far

To make updating simpler all the crates are now in the same portage-cli monorepo and most are already released, with few on hold waiting on having all the fixes I prepared for brush landing (help reviewing and polishing very welcome).

Right now it is possible to enjoy a quite faster dependency resolution, cache regeneration and search. There is some essential support for building and merging packages, but it is far from being battle tested.

In the coming weeks I’ll try to fill more gaps and make it more functional and hopefully blog more often.