hckrnws
Show HN: Attune - Build and publish APT repositories in seconds
by ilikebits
Hey HN, we're Eliza and Xin, and we’ve been working on Attune. Attune is a tool for publishing Linux packages.
Previously, we worked at other startups building open source developer tools that ran on our customers’ CI and development machines. For many of them, being able to `apt-get install` our tools was a requirement.
When we went to actually set up APT repositories, we were really surprised by the state of tooling around package publishing. The open source tools we found were old, slow, and difficult to figure out how to run in CI. The commercial tools we found were not much better. The cloud-hosted vendors required us to provide our signing keys to a cloud vendor (which was a non-starter), while the self-hosted vendors required us to operate our own specialized hosting servers.
We just wanted something simple: sign locally, run quickly, be easy to use, and deploy to managed object storage.
We couldn’t find it, so we built it. If you want to try it out, you can create a repository with three commands:
attune repo create --uri https://apt.releases.example.com
attune repo pkg add --repo-id 123 package.deb
attune repo sync --repo-id 123
It’s fully open source under Apache 2. We’re also working with some early customers to build enterprise features like audit logging, RBAC, and HSM integrations, and we’re thinking about building a managed cloud hosting service as well.
We’d love your feedback on whether this is useful for you, and what you’d like to see next. We’re well aware that publishing is a small piece of CI/CD, but we think a lot of the tooling in this area (publishing, artifact registries, package repositories) could really use some love.
What do you think? Comment here, or email us at founders@attunehq.com.
It's probably intuitive to anybody who's tried to do this the hard way, but if your target is people without that experience, you might want to dumb it down even more. So this is from your readme:
attune repo create -u 'http://localhost:9000/debian' -d bookworm
attune repo pkg -r 1 add -c stable/v17 ~/Downloads/teleport_17.4.4_arm64.deb
Also, from your post here (and why are the quick start steps different here?),
attune repo create --uri https://apt.releases.example.com
Totally agree here. The current target is people who are familiar with the process, and we plan to write more documentation as we start thinking more about accessibility to a broader audience. We actually ran into this issue ourselves while building the tool - the Debian documentation is all there, but not the easiest to approach. These docs are one of the rough edges (sorry!) since we wanted to get this out earlier for feedback.
To answer your direct question: on self-hosted repositories, the URI right now is just a string identifier. We are working right now on a cloud hosted service, and there the URI will help us figure out which repository to actually serve.
Lots more options listed on the Debian wiki, I've added Attune to the list.
Thanks, appreciate it!
How is this different to aptly, especially if one already has object storage?
One of the tools we tried was Aptly! A couple issues we had with Aptly:
1. Aptly needs to rebuild the entire repository before it can do any changes. One of our customers builds packages for every Linux OS, and every architecture, and each release contains 10-something separately packaged tools, and they do regular releases. This means their repository of packages is _huge_. When they try to do small maintenance actions like removing a package that has a bug, they need to rebuild their entire repository to do this, which takes upwards of 30 minutes. On Attune, since we have a centralized control plane, it takes about 10 seconds.
2. Aptly doesn't have concurrency controls. This makes it annoying to run in CI. Imagine you're setting up a CI pipeline that runs on every commit and publishes a package. You need to make sure you don't have two different instances of Aptly running at the same time, because their updates will clobber each other if they publish at the same time. Can you ensure that your CI only ever runs one publish at a time? This is easy in some setups, but surprisingly hard in GitHub Actions - concurrency keys don't quite work the way you'd expect them to.
3. Aptly uses its storage as its source of truth for the current state of packages. This is a great design choice for simplicity! However, if your CI job dies halfway through during an Aptly publish, this can leave your repository in a corrupted (in this case, half-updated) state. To recover from this, you need to rebuild the repository state... which you can't, because the ground truth is contained in the storage which was just corrupted! We mitigate this issue by tracking metadata in a separate transactional database so CI crashes don't cause corruption.
These are some examples of little cuts that build up over time with all of the tools we've found so far. Lots of the open source tooling is great at low scale, but the cuts start to build up once you are pushing a meaningful amount of packages to production.
Thanks for the detailed response!
Maybe I'm wrong, but aren't apt repositories just files in a web root? There are many scripts in the Debian ecosystem to generate and maintain such web directories. Maybe the actual package building would be interesting, but I don't know about just "we'll host your .deb files and keep the index updated"
As a FOSS project which publishes debs, it's a real hassle using current tooling. The tools are heavily biased towards the needs of distro maintainers, forcing projects into overly complex workflows which are overkill for a small numbers of packages. Features that would benefit smaller repositories with more frequent updates are limited or missing.
The other options is to DIY without tooling (essentially, write custom tooling as scripts or whatever), which is messy and full of pitfalls to the inexperienced.
You are correct. The reason we built this tool is because all of the scripts we found had sharp edges when we tried to use them in production at scale (I talk about some of them here: https://news.ycombinator.com/item?id=43730136).
Most of these scripts were designed for a world where there was A Blessed Deployment Machine that acted as its own de facto centralized control plane. We're designed for a newer world where publishing is just another piece of your CI, so you need more features to handle concurrency control, distributed signing, incremental index rebuilds, etc.
What is an example of something that would require "distributed signing"?
Our use case is customers who (1) want to use a managed cloud hosting service that we provide, but (2) are not willing to give us their signing keys. Our design allows them to keep all of their signing keys local to their environment.
To our knowledge, we have not found another provider who supports both of these requirements. It's not some amazing technical innovation, but it is one of those annoying paper cuts that builds up with all the others.
> when we tried to use them in production at scale
Debian and Ubuntu have been using them in production and "at scale" for decades. What are the "sharp edges" that you're trying to solve?
That's true! The Debian and Ubuntu folks are also experts at this. In our experience, the sharp edges generally affect teams that don't have a lot of in-house expertise in this, and where release engineering is not a core engineering competency that they want to invest in.
Signing an unsigned apt repository is about three gpg commands though. I don't see how this kind of thing requires a whole separate product.
Yeah, solving this locally for one repository definitely isn't that hard at all. Most of the features we're building become useful when you're trying to build CI integrations for a larger team while also complying with enterprise security requirements (e.g. audit logging, HSM key protections, etc.).
Comment was deleted :(
Curious how different is this from mise and ubi? https://github.com/jdx/mise
I have a few pain points with that for installing cua (https://github.com/trycua/cua/issues/27), so if it can remove the initial friction happy to chat!
I'm not familiar with mise and ubi, but these look like they're tools that the end user runs to install binaries. In contrast, we're building something that sets up a server so that end users can use their existing `apt-get install`.
I've never heard of mise used for creating and managing APT repositories. Where does that figure in?
Looks neat!
What would be really valuable for us would be future support for other distros/repos.
A more unified interface for synchronized publishing across say dep/rpm/archlinux/alpine when we have multi-distro packages to build and publish.
Yeah, multi-distro and multi-arch is something on our roadmap! We're especially interested in automating the more annoying OS-specific parts of releasing, like MacOS notarization.
very cool stuff!
Yes, it's Rust ;). Sign of quality to me.
Excited to try. I ran a reprepo for a couple of years & it wasn't the worst to run, but not fun either. My confidence in efficiently publishing packages was super low, never improved much.
It also mandated running a web server (configuring my nginx to point at the repo filesystem), not very batteries included.
Thanks for trying, and sorry in advance for the rough edges :)
We're also working on a hosted service! If you'd like a sneak peek, send us a message at founders@attunehq.com (or email me directly at eliza@attunehq.com). I'm happy to talk about your specific needs and see if we can build something for them.
(And yes, it is Rust. I keep trying to find projects where I get to stretch my Haskell wings again, but unfortunately I keep working on things with silly requirements such as "easily understandable performance" and "a cross-compilation story" and "not making my collaborators sit through another monad tutorial".)
Will you be rewriting the Golang part in Rust?
We don't plan to do that right now. The CLI isn't that complicated, and the choice for that was driven more by Go's excellent cross-compilation story than anything else.
It would be a fun thing to do if we had the resources to get equally good cross-compilation in Rust, but we're focused on building functionality right now.
It looks like Golang to me?
Edit: the CLI part is Golang and the rest(?) is Rust.
.. for the repo github pages can be used as well, although the disgust when pushing huge .deb files up there is real.
Comment was deleted :(
[dead]
Crafted by Rajat
Source Code