this post was submitted on 09 Aug 2023
27 points (100.0% liked)

Linux

45595 readers
675 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago
MODERATORS
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
27
Nix 3rd party repos and Binary packages (feddit.nl)
submitted 11 months ago by [email protected] to c/[email protected]
12 comments fedilink hide all child comments
 

I've read a lot, but didn't get a simple answer for 3 topics I'm interested about:

  1. Are there any 3rd party repos for NixOS? It seems some people overlay one package, but I cannot find entire packagesets such as the one in Nixpkgs. Are channels used for that? Would I be able to have my own repo with several packages? Can I have more than 1 repo in my Nix system, and set which package install each program from?
  2. Would it be possible, for propietary software compiled directly for NixOS, to have a binary-only (as opposed to source, or binary+patched) repository? Everything I see is either source+cache or binary+patched, but nothing like binary especifically for NixOS.
  3. Let's say I want to add some extra, propietary codecs unavailable to NixOS. How would I be able to change a dependency so that each package dependent on it automatically picked it? Could I switch a dependency for a certain repo from one of another, different repo?

Thank you for your support guys!

all 13 comments
sorted by: hot top controversial new old
[–] [email protected] 8 points 11 months ago* (last edited 11 months ago) (1 children)

Are there any 3rd party repos for NixOS? It seems some people overlay one package, but I cannot find entire packagesets such as the one in Nixpkgs. Are channels used for that? Would I be able to have my own repo with several packages? Can I have more than 1 repo in my Nix system, and set which package install each program from?

Yes, the most common one I think is the NUR, the Nix User Repository, the packages have their own namespace. https://github.com/nix-community/NUR

Would it be possible, for propietary software compiled directly for NixOS, to have a binary-only (as opposed to source, or binary+patched) repository? Everything I see is either source+cache or binary+patched, but nothing like binary especifically for NixOS.

I don't see anything prohibiting that

Let’s say I want to add some extra, propietary codecs unavailable to NixOS. How would I be able to change a dependency so that each package dependent on it automatically picked it? Could I switch a dependency for a certain repo from one of another, different repo?

The issue is that packages (or derivations) only depend on something because it's declared in their definition. That means if you need to add dependencies to existing packages, that can be done via overrideAttrs. However, I'm not certain that you can easily add attributes to functions that don't have a mechanism for it (I don't know derivations that allow for more attributes than specified and I'm not very good at the nix language) and dependencies (buildInputs and nativeBuildInputs) can only use packages that were supplied as attributes. Derivations usually declare their attributes via { x, y ? "bar" }:, what you would need to make use of additional attributes is something like attrs @ { x, y ? "bar", ... }: so that the additionally supplied attributes can be accessed. But again am not an expert in the language so take that with a grain of salt.

[–] [email protected] 1 points 11 months ago

You are confusing override and overrideAttrs. override is for what you said - overriding the derivation arguments. But overrideAttrs is for overriding whatever is passed to mkDerivation by the derivation code.

[–] [email protected] 4 points 11 months ago (2 children)

Are there any 3rd party repos for NixOS?

Define "repository". For most definitions of a repository the answer would technically be "yes" but yours might differ.

Would it be possible, for propietary software compiled directly for NixOS, to have a binary-only (as opposed to source, or binary+patched) repository?

No. For a "native" Nix output path, you must have the full build definitions; Nix expressions.

In the case of most proprietary software in Nixpkgs, the BLOB of the actual thing is a FOD (fixed-output-derivation) which obfuscates everything behind it; meaning it can't depend on other Nix output paths. Only the the dependency DAG for the wrapping around it is known.

Let’s say I want to add some extra, propietary codecs unavailable to NixOS. How would I be able to change a dependency so that each package dependent on it automatically picked it?

This is what overlays are for. They allow you to change the meaning of some package name in the context of an entire package set in which other packages may depend on the meaning of that package name. Those packages will have the change in meaning propagated to them.

Note that changing a low-level package with many dependants such as i.e. mesa will cause a change in those dependants derivations; a change that is not present in the expression used for the central binary cache and therefore requires you to build the packages yourself. With a decently fast computer, building a typical desktop closure from such a low point onwards will take at least half a day.

[–] [email protected] 2 points 11 months ago* (last edited 11 months ago) (1 children)

No. For a “native” Nix output path, you must have the full build definitions; Nix expressions.

I think I originally misunderstood the question. But I'm still not sure.

This is what overlays are for. They allow you to change the meaning of some package name in the context of an entire package set in which other packages may depend on the meaning of that package name. Those packages will have the change in meaning propagated to them.

The issue with how OP stated it though is that it basically means adding a dependency to a multitude of packages, not changing an existing one. I mean sure I could overlay ffmpeg-proprietary for ffmpeg and all other packages depending on ffmpeg would happily pick it up (as you said probably needing a rebuild) but how would you add my-proprietary-codec to all packages supporting it? Or how would you add JPEG-XL to ffmpeg and other packages that make use of it independently (e.g. so that Gwenview supports JPEG-XL)? I'm not saying it can't be done, but I personally don't know how. The latter might need a separate derivation for the plugin and a change in the derivation for Gwenview itself that can not be achieved through overrideAttrs for buildInputs alone. Or is my understanding incorrect?

[–] [email protected] 2 points 11 months ago (1 children)

The issue with how OP stated it though is that it basically means adding a dependency to a multitude of packages

How would I be able to change a dependency so that each package dependent on it automatically picked it?

how would you add my-proprietary-codec to all packages supporting it?

Gather the list of attribute names and then define an overlay like this:

final: prev: lib.genAttrs (n: prev."${n}".overrideAttrs (old: {
  buildInputs = old.buildInputs or [ ] ++ [ my-proprietary-codec ];
})) packageNamesToModify

It should be said that packages usually don't work like this however (they usually depend on one central thing which then depends on some proprietary thing) and if they do, there's usually an override setting for the package in question with the dep already wired up.

What you're actually looking for here is a Gentoo USE-flags like system. We have a precursor of that in i.e. config.cudaSupport but it's not widely used.

how would you add JPEG-XL to ffmpeg

You'd do that upstream because that's something that every instance of ffmpeg should be able to do. And also ping me because I maintain ffmpeg ;)

and other packages that make use of it independently (e.g. so that Gwenview supports JPEG-XL)?

I'd expect most packages to make use of such a feature dependently. I don't know how gwenview or QT stuff works but from a quick Google I gather that KDE stuff has the kimageformats library which then in turn depends on jxl. Gwenview should depend on that but only depends on qtimageformats currently? I'd give adding kimageformats to its buildInputs a spin if you're curious. If it works, make sure to submit a patch against Nixpkgs.

This isn't the sort of thing you should solve in an overlay (better just do it upstream) but you certainly could.

[–] [email protected] 1 points 11 months ago (1 children)

Thanks for the thorough answer.

final: prev: lib.genAttrs (n: prev."${n}".overrideAttrs (old: {
  buildInputs = old.buildInputs or [ ] ++ [ my-proprietary-codec ];
})) packageNamesToModify

Thanks. I didn't know that worked, however I'd say this is rather advanced nix. I know genAttrs "generate[s] an attribute set by mapping a function over a list of attribute names", however for this piece of code I'd have trouble locating the list of attribute names. But I guess this is because I don't understand overrideAttrs fully / correctly. In my defense, locating the documentation for that one isn't trivial – I think this requires having some more experience with the language and the needed libraries. While I know that genAttrs is defined and explained in nixpkgs/lib/attrsets.nix, I'm not so sure about overrideAttrs– I'd guess it's in nixpkgs/pkgs/stdenv/generic/make-derivation.nix? Even https://ryantm.github.io/nixpkgs/using/overrides/#sec-pkg-overrideAttrs is not very verbose.

Please don't take this as criticism, I know nix is a full language and I'm not trying to bash anyone here, however there's quite the learning curve.

What you’re actually looking for here is a Gentoo USE-flags like system. We have a precursor of that in i.e. config.cudaSupport but it’s not widely used.

In my uneducated opinion, full use of USE-flags are rather complicated with package managers providing binaries, no? It would require a big number of builds for all possible options and their combinations.

You’d do that upstream because that’s something that every instance of ffmpeg should be able to do. And also ping me because I maintain ffmpeg ;)

Thanks for your work, I don't really need jpeg-xl in ffmpeg, I played around with it when I was still using Arch (i.e. take screenshots in the format in mpv) but it wasn't urgent enough for me to file an issue on Github.

I’d expect most packages to make use of such a feature dependently. I don’t know how gwenview or QT stuff works but from a quick Google I gather that KDE stuff has the kimageformats library which then in turn depends on jxl. Gwenview should depend on that but only depends on qtimageformats currently? I’d give adding kimageformats to its buildInputs a spin if you’re curious. If it works, make sure to submit a patch against Nixpkgs.

I had a quick glance at the definitions myself; also since 2021, kimageformats should provide support for jxl. My knowledge here was outdated that it needs a seperate plugin (Arch has one here https://aur.archlinux.org/packages/qt5-jpegxl-image-plugin) but that seems to no longer be the case. Also I don't actually have jxl files to test stuff with, since the major browsers don't support it atm. I tried finding an easy example to OPs problem, though it seems it is kind of moot now, and Gwenview in nixpkgs does support JPEG-XL (just tested it by creating one via cjxl).

[–] [email protected] 0 points 11 months ago (1 children)

Thanks. I didn't know that worked, however I'd say this is rather advanced nix.

Sorry, I assumed you were an advanced user/the code was easier to understand. It's easy to get caught up in the imposter syndrome.

I know genAttrs "generate[s] an attribute set by mapping a function over a list of attribute names", however for this piece of code I'd have trouble locating the list of attribute names.

You mean packageNamesToModify? You have to provide those. I don't know which packages could be built with my-proprietary-codec and it's not feasible to figure that out automatically.

But I guess this is because I don't understand overrideAttrs fully / correctly. In my defense, locating the documentation for that one isn't trivial – I think this requires having some more experience with the language and the needed libraries.

Yes, certainly. overrideAttrs allows you to "modify" the argument originally passed to mkDerivation such as src, pname, version, buildInputs and many more. To understand overrideAttrs, you must know what these common mkDerivation arguments mean. From then on, overrideAttrs is trivial to understand.
It's like writing a derivation but "updating" an existing derivation to be slightly different.

While I know that genAttrs is defined and explained in nixpkgs/lib/attrsets.nix, I'm not so sure about overrideAttrs– I'd guess it's in nixpkgs/pkgs/stdenv/generic/make-derivation.nix? Even https://ryantm.github.io/nixpkgs/using/overrides/#sec-pkg-overrideAttrs is not very verbose.

Please don't take this as criticism, I know nix is a full language and I'm not trying to bash anyone here, however there's quite the learning curve.

Oh absolutely. There's quite the learning curve, almost everyone knows that. It's just not an easy thing to fix and you don't tend to think about learning things you already know, so it's not constantly in people's minds.

What you’re actually looking for here is a Gentoo USE-flags like system. We have a precursor of that in i.e. config.cudaSupport but it’s not widely used.

In my uneducated opinion, full use of USE-flags are rather complicated with package managers providing binaries, no? It would require a big number of builds for all possible options and their combinations.

I'ma let you in on a little secret: We're not actually a binary distribution. We're source-based.

It's integral to the functional approach. Every artefact is a more-or-less direct result of a realisation which is the direct result of an evaluation of Nix expressions. The latter is pure and the former we strive to make as pure as possible via sandboxing.
This allows us to treat artefacts as replaceable objects which means that instead of building locally, we can substitute the (theoretically) same artefact from somewhere else and that's the binary cache.

You’d do that upstream because that’s something that every instance of ffmpeg should be able to do. And also ping me because I maintain ffmpeg ;)

Thanks for your work, I don't really need jpeg-xl in ffmpeg, I played around with it when I was still using Arch (i.e. take screenshots in the format in mpv) but it wasn't urgent enough for me to file an issue on Github.

If you're at all interested, go ahead and do so. That'd make for a great first PR. JXL support is something you'd expect of ffmpeg in the near future (if not already), so this will happen anyways.

[–] [email protected] 0 points 11 months ago (1 children)

I know genAttrs “generate[s] an attribute set by mapping a function over a list of attribute names”, however for this piece of code I’d have trouble locating the list of attribute names.

You mean packageNamesToModify? You have to provide those. I don’t know which packages could be built with my-proprietary-codec and it’s not feasible to figure that out automatically.

Not really. Looking at genAttrs, it says that it's genAttrs :: [ String ] -> (String -> Any) -> AttrSet, and the example code is

genAttrs [ "foo" "bar" ] (name: "x_" + name)
=> { foo = "x_foo"; bar = "x_bar"; }

So I'd expect a list of strings and the mapping function. However, from my first naive glance at the code you posted, there's no list of strings there, and rather directly a function definition that makes use of overrideAttrs, for which for me the documentation is unclear if it can only set attributes, create attributes or what else; also buildInputs at first naive glance isn't an attribute of the packages, but rather of stdenv (sorry if all of this is wrong), all of which isn't exactly intuitive if you've only worked with imperative languages.

I’ma let you in on a little secret: We’re not actually a binary distribution. We’re source-based.

I do know that, that's why I worded it like I did. In fact I have created a single package myself before and as such, did read a bit about FOB and all that, how the cache works (not that I can fully recall everything or claim that I understood it all) with input-hashes etc. I tripped over some stuff back then when I found out that which is not part of the sandbox. The program however is a bit niche and there's another component it needs that I didn't write a derivation for so I also never wrote a pull request, plus the build process is somewhat ugly. So I wouldn't call myself a beginner exactly but I definitely have trouble with advanced usage of the language and the intricacies of the system.

However, while source-based, I think it's a common understanding that the binary cache is a huge appeal, otherwise the discussion about the cost it wouldn't be had.

[–] [email protected] 0 points 11 months ago (1 children)

Not really. Looking at genAttrs, it says that it's genAttrs :: [ String ] -> (String -> Any) -> AttrSet, and the example code is

genAttrs [ "foo" "bar" ] (name: "x_" + name)
=> { foo = "x_foo"; bar = "x_bar"; }

So I'd expect a list of strings and the mapping function. However, from my first naive glance at the code you posted, there's no list of strings there, and rather directly a function definition

That's my bad. I didn't actually look at how the function works, so I got the arguments flipped. This is just a rough outline of how I'd imagine it to work; pseudo-code. 100% untested. Only there for the sake of argument.

packageNamesToModify is the list of strings you're looking for here. As I said though, I can't know which packages are supposed to be changed. That's a piece of info you'd need to research and define yourself. This is just how you'd apply that knowledge through a mechanism.

that makes use of overrideAttrs, for which for me the documentation is unclear if it can only set attributes, create attributes or what else

A function cannot "set" or "create" attributes. For better of for worse, there's no real meta-programming in Nix.

A function can only ever return a value. That value may be any of the primitive types such as booleans, strings, numbers, lists, attrsets etc. or even another function but it's always a value.

In the case of overrideAttrs { ... }, the return value is a derivation. In the case of genAttrs, it's an attrset.

buildInputs at first naive glance isn't an attribute of the packages, but rather of stdenv (sorry if all of this is wrong)

That is correct. However note how I said that overrideAttrs is about overriding arguments to mkDerivation? The canonical path to mkDerivation is stdenv.mkDerivation ;)

mkDerivation is the way you set arguments for the stdenv. .overrideAttrs is how you "modify" the arguments to the stdenv of an existing package.

all of which isn't exactly intuitive if you've only worked with imperative languages.

I can absolutely see that. You'll get used to it though. When I was new to Nix, it took me quite a while to realise how you'd even do something like creating loops (spoiler: you don't).

I'd highly recommend familiarising yourself with basic functional programming concepts such as immutability, everything being a value (including functions, see lambdas), recursion, basic functional list comprehension (head/tail, map, filter, reduce) and perhaps even currying.

Nix is a great learning ground for the basics of functional programming as it's a pure expression language which is quite a bit more limited than an actual functional programming language.

I tripped over some stuff back then when I found out that which is not part of the sandbox.

That's what buildInputs are for. Add which to buildInputs and it's available inside the sandbox. The stdenv takes care of putting its binaries into $PATH and making its libraries discoverable.

In the case of which, you'd probably need it in order to execute its binary during the build process though, so nativeBuildInputs is more appropriate but that only truly matters for cross-compilation.

[–] [email protected] 0 points 11 months ago (1 children)

Thanks for the explanation. I didn't make the connection that buildInputs is an attribute itself as it is an attribute of stdenv instead of the function describing the derivation directly. Or at least I think that's where my confusion comes from.

I tripped over some stuff back then when I found out that which is not part of the sandbox.

That’s what buildInputs are for. Add which to buildInputs and it’s available inside the sandbox. The stdenv takes care of putting its binaries into $PATH and making its libraries discoverable.

In the case of which, you’d probably need it in order to execute its binary during the build process though, so nativeBuildInputs is more appropriate but that only truly matters for cross-compilation.

Small correction, it wasn't which, but rather env, I had those mixed up. The "issue" is described here: https://github.com/NixOS/nixpkgs/issues/6227 What created more problems was that patchShebangs wouldn't work here because it appeared in a configure script that was created and run during the actual build process (I think the build process is horrible, but here it is in case you're inclined: https://github.com/BSI-Bund/TaSK/tree/master/tlstesttool the stuff in the 3rdparty directory gets downloaded, configured and linked against in the main program's build phase so you have no opportunity to actually follow the solution in that issue, similar to what is described here https://github.com/NixOS/nixpkgs/issues/6227#issuecomment-73410536. I got it to work in the end and like to tell myself that it's elegant but the project's build process is just bad in my opinion.

https://pastebin.com/0GwLk1wP if you want to see an example of my level of nix-fu. I have programming basics but the nix language can be confusing sometimes. I'd say I have a basic understanding of things but as said before the more intricate stuff still escapes me.

[–] [email protected] 1 points 11 months ago

it wasn't which, but rather env, I had those mixed up. The "issue" is described here: https://github.com/NixOS/nixpkgs/issues/6227

The problem there isn't that env isn't available (it is, we have coreutils in stdenv) but rather that /usr/bin/env (that specific path) does not exist in the sandbox.

What created more problems was that patchShebangs wouldn't work here because it appeared in a configure script that was created and run during the actual build process (I think the build process is horrible, but here it is in case you're inclined: https://github.com/BSI-Bund/TaSK/tree/master/tlstesttool the stuff in the 3rdparty directory gets downloaded, configured and linked against in the main program's build phase

Yikes, you don't want that.

so you have no opportunity to actually follow the solution in that issue

I didn't read the issue in full but, since everything is pre-downloaded, you can always fix these scripts.

For the auto-generated configure script for example, you'd have to patch the build step which generates the configure script to put in ${coreutils}/bin/env rather than /usr/bin/env. Simple as that.

3rd party repositories can be patched during their respective fetches. You have to inject them anyways since there's no way to download 3rd party repos during a build.

https://pastebin.com/0GwLk1wP if you want to see an example of my level of nix-fu. I have programming basics but the nix language can be confusing sometimes. I'd say I have a basic understanding of things but as said before the more intricate stuff still escapes me.

You have to differentiate between the Nix expression language and using Nixpkgs' frameworks to define packages here. These are two entirely different skillsets with only a slight dependence on the former by the latter.

If you take a look at your expression, it only required fairly basic Nix syntax: Simple function calls, declaring recursive attrsets, string interpolation and attribute access. Most packages are like this.
Figuring out which attributes need to be set how and what that means is an entirely different skillset. Defining patchPhase in that attrset is trivial syntax-wise but figuring out what needs to be substituted using which stdenv "library" function is something else entirely.

Looks pretty good btw. I'd look into whether the build could be coerced to link against Nixpkgs' versions of those libraries though instead of vendoring dependencies. Especially security-critical stuff like openssl. That'd probably also save you the trouble with the interpreter.
If you take a look at the build definition, there's this handy dandy USE_3RDPARTYBUILD option.

I'd wager if you did cmakeFlags = [ ... "-DUSE_3RDPARTYBUILD=OFF" ]; and buildInputs = [ asio zlib openssl ... ]; (from pkgs, not your fetched sources) it'd just work. (Might need pkg-config in nativeBuildInputs but I don't think it uses that and will instead discover those deps via cmake.)

If you can get rid of the vendoring, feel free to submit that to Nixpkgs ;)

[–] [email protected] 1 points 11 months ago* (last edited 11 months ago)

Are there any 3rd party repos?

Nix has "flakes", which allow you to share Nix code in a Git repository, it's like repos on steroids. There are many Git projects that offer new packages (such as nix-gaming) or NixOS modules (such as my project nixos-router), or even just Nix code (such as my projects notlua and notnft, which allow you to write Lua code and nftables rules in Nix), or any combination thereof.

Would it be possible [for proprietary software to be compiled for NixOS]?

Kind of. You first have to understand what Nix derivations are - builders that take certain inputs (such as certain versions of libraries) and produce some outputs.

What happens if the inputs (such as a library version) change? The outputs change as well - previously it was /nix/store/abcdefgh-libfoo/lib/libfoo.so, now it's /nix/store/ijklmnop-libfoo/lib/libfoo.so - the path to libfoo changes, and the binary's RPATH reflects that.

So if you want to package binary software for NixOS, you either have to pin library versions (so the paths don't ever change), or patch the binary.

...proprietary codecs...

It depends on what those codecs are.

Let's say they are a binary. In that case, you install them and they get added to your PATH - easy.

Let's say they are some data files. In that case you install it and it gets put into XDG_DATA_DIRS - easy.

Let's say it's a shared library (.so). First question - how is that .so loaded? By which program? From where?

Depending on the answer, what you have to do changes as well. You may have to override some core media library, or ffmpeg, or maybe you can override VLC, or VLC's ffmpeg, but not system ffmpeg. Or, it may be the case that a simple LD_LIBRARY_PATH change will do it for you.

Basically - it depends. That's why NixOS requires a deeper knowledge of Linux, or forces you to learn.