Allow `github:` fetcher to fetch subtrees using the trees API

[arianvp]

Is your feature request related to a problem?

It is overkill to pull in all of nixpkgs if you just want nixpkgs/lib. Currently we work around this by vendoring out lib out of the nixpkgs monorepo but I am proposing a more elegant solution:

Proposed solution

% sha=$(curl https://api.github.com/repos/NixOS/nixpkgs/git/trees/master:lib | jq -r .sha)
% curl -L https://github.com/NixOS/nixpkgs/archive/$sha.tar.gz -o $sha.tar.gz
% tar xzvf $sha.tar.gz
% ls nixpkgs-$sha
README.md		deprecated		generators.nix		network			tests
ascii-table.nix		derivations.nix		gvariant.nix		options.nix		trivial.nix
asserts.nix		fetchers.nix		kernel.nix		path			types.nix
attrsets.nix		fileset			licenses.nix		source-types.nix	versions.nix
cli.nix			filesystem.nix		lists.nix		sources.nix
customisation.nix	fixed-points.nix	meta.nix		strings-with-deps.nix
debug.nix		flake-version-info.nix	minfeatures.nix		strings.nix
default.nix		flake.nix		modules.nix		systems

We now have a tarball with just the lib subfolder!

Alternative solutions

The current solution is https://github.com/nix-community/nixpkgs.lib which is complex and complicated and doesn't work for arbitrary refs. It relies on github actions to keep things manually in sync

Additional context

Sparked out of Matrix discussions with @emilazy and @alyssais

Checklist

• checked latest Nix manual (source)
• checked open feature issues and pull requests for possible duplicates

---

Add 👍 to issues you find important.

[arianvp]

This works:

let
  ref = "master";
  subdir = "lib";
  tree = builtins.fromJSON (
    builtins.readFile (
      builtins.fetchTree {
        type = "file";
        url = "https://api.github.com/repos/NixOS/nixpkgs/git/trees/${ref}:${subdir}";
      }
    )
  );
  lib = import (
    builtins.fetchTree {
      type = "tarball";
      url = "https://github.com/nixos/nixpkgs/archive/${tree.sha}.tar.gz";
    }
  );
in
lib

[arianvp]

From the same discussion we also discovered that

https://github.com/NixOS/nixpkgs/archive/master^{tree}.tar.gz 

gives you a tarball that doesnt have export-subst reproducibility problems

[roberth]

Yes please :)

Prior discussions and work by DavHau have explored shallow git: fetches instead, but those have not been without trouble, so this is definitely still worth pursuing.

References

• Prior discussion fetchGit/fetchTree reproducibility problems #5313 (comment)
• Non-API url https://github.com/hercules-ci/flake-parts/pull/250/files

[acid-bong]

  builtins.fromJSON (
    builtins.readFile (
      builtins.fetchTree {

isn't it IFD?

[arianvp]

No that's not IFD. non of those are derivations. it's all eval-time fetches. fetchTree is what flakes uses under the hood.

If you want to be able to run in pure evaluation mode we need to add native flake support for the same pattern

[acid-bong]

Semi-related, but the same API schema works for Forgejo and Gitea. Welp, these two also support downloading tarballs like REF:PATH.tar.gz, keeping the same locking manner that was introduced this spring/summer