Mon 26 Aug 2024

Previous: Mon 12 Aug 2024 Next: Mon 2 Sep 2024

Mon

ok, so we’re going to try and modify opam-0install-solver to have different `roles` per implementation for every cargo package so there’s no restrictions on a single version of a package.

where are the roles defined? Model.role

where are the implementations defined? Model.implementations

Okay, so what’s the problem? If we have a role as an OpamPackage (name + version) how do we specify the versions to solve in the first instance? Can we just say any of them? Maybe as constraints? Or do you know what, let’s just specify a version up front. How do we minimise duplicated dependancies?

How do we do express version constraints with each role a different version?

This is trickier than I anticipated. How do we express dependencies when we have each package version as a distinct ‘role’? We need a way to go from a role to a list of dependency roles in 0install. We could express this at the SAT solving (for 0install) or CUDF layer as e.g. package.1.1 OR package.1.2 OR package.1.3, but this would require modifying the 0install solver directly

This is trickier than I anticipated. Namely how we express dependencies when we have each package version as a distinct ‘role’. 0install uses a ‘requires’ list of dependency roles, which when using a separate role for each dependency means we need to pick a version at the dependency specifying stage. Instead of depending on a role with a concrete version, we could express this at the SAT solving (for 0install) or CUDF layer as e.g. package.1.1 OR package.1.2 OR package.1.3, but this would require modifying the 0install solver directly.

For reference, the solver function, which builds a SAT problem which adds an expression for dependencies, which we could modify to enumerate multiple roles.

Wed

In the first instance let’s try and get it working spraying versions all over the place.

Actually, maybe we can do something with restrictions?

with meets_restriction?

and implementations

Thu

  • Got a prototype solver for multiple versions of a dependancy
  • But it’s very slow, and the solver will return all implementations of a role for a single dependance relation.
$ dune exec -- bin/main.exe --repo /dev/shm/opam-giga-repository/packages cargo-sauron-core.0.57.0
cargo-bumpalo.3.2.1 cargo-cfg-if.0.1.0 cargo-cfg-if.0.1.2 cargo-cfg-if.1.0.0 cargo-doc-comment.0.3.0 cargo-dtoa.0.4.0 cargo-itoa.0.3.0 cargo-js-sys.0.3.0 cargo-json.0.12.0 cargo-jss.0.6.1 cargo-lazy_static.1.0.0 cargo-log.0.4.0 cargo-longest-increasing-subsequence.0.1.0 cargo-mt-dom.0.55.1 cargo-num-traits.0.1.32 cargo-once_cell.1.8.0 cargo-once_cell.1.12.0 cargo-phf.0.10.0 cargo-phf_shared.0.10.0 cargo-proc-macro2.0.4.0 cargo-proc-macro2.0.4.4 cargo-proc-macro2.0.4.8 cargo-proc-macro2.0.4.9 cargo-proc-macro2.1.0.0 cargo-proc-macro2.1.0.23 cargo-quote.0.3.8 cargo-quote.0.6.0 cargo-quote.1.0.0 cargo-sauron-core.0.57.0 cargo-serde.1.0.0 cargo-serde-wasm-bindgen.0.5.0 cargo-serde_derive.1.0.0 cargo-serde_derive_internals.0.15.0 cargo-serde_json.1.0.0 cargo-siphasher.0.3.0 cargo-syn.0.11.0 cargo-syn.0.11.10 cargo-syn.0.15.0 cargo-syn.1.0.0 cargo-syn.1.0.67 cargo-synom.0.11.0 cargo-thiserror.1.0.0 cargo-thiserror-impl.1.0.0 cargo-unicode-xid.0.0.4 cargo-unicode-xid.0.1.0 cargo-unicode-xid.0.2.0 cargo-wasm-bindgen.0.2.23 cargo-wasm-bindgen.0.2.83 cargo-wasm-bindgen-backend.0.2.23 cargo-wasm-bindgen-backend.0.2.83 cargo-wasm-bindgen-macro.0.2.23 cargo-wasm-bindgen-macro.0.2.83 cargo-wasm-bindgen-macro-support.0.2.23 cargo-wasm-bindgen-macro-support.0.2.83 cargo-wasm-bindgen-shared.0.2.23 cargo-wasm-bindgen-shared.0.2.83
- cargo-bumpalo.3.2.1
- cargo-cfg-if.0.1.0
- cargo-cfg-if.0.1.2
- cargo-cfg-if.1.0.0
- cargo-doc-comment.0.3.0
- cargo-dtoa.0.4.0
- cargo-itoa.0.3.0
- cargo-js-sys.0.3.0
- cargo-json.0.12.0
- cargo-jss.0.6.1
- cargo-lazy_static.1.0.0
- cargo-log.0.4.0
- cargo-longest-increasing-subsequence.0.1.0
- cargo-mt-dom.0.55.1
- cargo-num-traits.0.1.32
- cargo-once_cell.1.8.0
- cargo-once_cell.1.12.0
- cargo-phf.0.10.0
- cargo-phf_shared.0.10.0
- cargo-proc-macro2.0.4.0
- cargo-proc-macro2.0.4.4
- cargo-proc-macro2.0.4.8
- cargo-proc-macro2.0.4.9
- cargo-proc-macro2.1.0.0
- cargo-proc-macro2.1.0.23
- cargo-quote.0.3.8
- cargo-quote.0.6.0
- cargo-quote.1.0.0
- cargo-sauron-core.0.57.0
- cargo-serde.1.0.0
- cargo-serde-wasm-bindgen.0.5.0
- cargo-serde_derive.1.0.0
- cargo-serde_derive_internals.0.15.0
- cargo-serde_json.1.0.0
- cargo-siphasher.0.3.0
- cargo-syn.0.11.0
- cargo-syn.0.11.10
- cargo-syn.0.15.0
- cargo-syn.1.0.0
- cargo-syn.1.0.67
- cargo-synom.0.11.0
- cargo-thiserror.1.0.0
- cargo-thiserror-impl.1.0.0
- cargo-unicode-xid.0.0.4
- cargo-unicode-xid.0.1.0
- cargo-unicode-xid.0.2.0
- cargo-wasm-bindgen.0.2.23
- cargo-wasm-bindgen.0.2.83
- cargo-wasm-bindgen-backend.0.2.23
- cargo-wasm-bindgen-backend.0.2.83
- cargo-wasm-bindgen-macro.0.2.23
- cargo-wasm-bindgen-macro.0.2.83
- cargo-wasm-bindgen-macro-support.0.2.23
- cargo-wasm-bindgen-macro-support.0.2.83
- cargo-wasm-bindgen-shared.0.2.23
- cargo-wasm-bindgen-shared.0.2.83

Can't find all required versions.
Selected: cargo-doc-comment.0.3.3 cargo-js-sys.0.3.70 cargo-json.0.12.4
          cargo-jss.0.6.2 cargo-log.0.4.22
          cargo-longest-increasing-subsequence.0.1.0 cargo-mt-dom.0.59.2
          cargo-once_cell.1.19.0 cargo-phf.0.11.2 cargo-phf_shared.0.11.2
          cargo-proc-macro2.1.0.86 cargo-quote.1.0.37 cargo-serde.1.0.209
          cargo-serde-wasm-bindgen.0.6.5 cargo-serde_derive.1.0.209
          cargo-syn.2.0.76 cargo-thiserror.1.0.63 cargo-thiserror-impl.1.0.63
          cargo-unicode-ident.1.0.12
- cargo-cfg-if -> cargo-cfg-if.0.1.10
    cargo-sauron-core 0.57.0 requires >= 0.1 & < 1.0.0
- cargo-sauron-core -> cargo-sauron-core.0.57.0
    User requested = 0.57.0
- cargo-siphasher -> cargo-siphasher.0.3.11
    cargo-phf_shared 0.11.2 requires >= 0.3 & < 1.0.0
- cargo-wasm-bindgen -> (problem)
    cargo-js-sys 0.3.70 requires >= 0.2.93 & < 1.0.0
    Rejected candidates:
      cargo-wasm-bindgen.0.2.93: Requires cargo-cfg-if >= 1.0.0 & < 2.0.0
      cargo-wasm-bindgen.0.2.92: Incompatible with restriction: >= 0.2.93 & < 1.0.0
      cargo-wasm-bindgen.0.2.91: Incompatible with restriction: >= 0.2.93 & < 1.0.0
      cargo-wasm-bindgen.0.2.90: Incompatible with restriction: >= 0.2.93 & < 1.0.0
      cargo-wasm-bindgen.0.2.89: Incompatible with restriction: >= 0.2.93 & < 1.0.0
      ...
Failed: cargo-sauron-core.0.57.0
  • We want to restrict solving a single version of a package per dependancy.
  • This may describe some of the performances issues as well.
  • core calculus: depends, deopts, restricts
  • ah, ha! we just needed to reverse the dependancies list to get the right order
dune exec -- bin/main.exe --repo /dev/shm/opam-giga-repository/packages cargo-sauron-core.0.57.0
cargo-bumpalo.3.16.0 cargo-cfg-if.0.1.10 cargo-cfg-if.1.0.0 cargo-doc-comment.0.3.3 cargo-js-sys.0.3.70 cargo-json.0.12.4 cargo-jss.0.6.2 cargo-log.0.4.22 cargo-longest-increasing-subsequence.0.1.0 cargo-mt-dom.0.59.2 cargo-once_cell.1.19.0 cargo-phf.0.11.2 cargo-phf_shared.0.11.2 cargo-proc-macro2.1.0.86 cargo-quote.1.0.37 cargo-sauron-core.0.57.0 cargo-serde.1.0.209 cargo-serde-wasm-bindgen.0.6.5 cargo-serde_derive.1.0.209 cargo-siphasher.0.3.11 cargo-syn.2.0.76 cargo-thiserror.1.0.63 cargo-thiserror-impl.1.0.63 cargo-unicode-ident.1.0.12 cargo-wasm-bindgen.0.2.93 cargo-wasm-bindgen-backend.0.2.93 cargo-wasm-bindgen-macro.0.2.93 cargo-wasm-bindgen-macro-support.0.2.93 cargo-wasm-bindgen-shared.0.2.93
- cargo-bumpalo.3.16.0
- cargo-cfg-if.0.1.10
- cargo-cfg-if.1.0.0
- cargo-doc-comment.0.3.3
- cargo-js-sys.0.3.70
- cargo-json.0.12.4
- cargo-jss.0.6.2
- cargo-log.0.4.22
- cargo-longest-increasing-subsequence.0.1.0
- cargo-mt-dom.0.59.2
- cargo-once_cell.1.19.0
- cargo-phf.0.11.2
- cargo-phf_shared.0.11.2
- cargo-proc-macro2.1.0.86
- cargo-quote.1.0.37
- cargo-sauron-core.0.57.0
- cargo-serde.1.0.209
- cargo-serde-wasm-bindgen.0.6.5
- cargo-serde_derive.1.0.209
- cargo-siphasher.0.3.11
- cargo-syn.2.0.76
- cargo-thiserror.1.0.63
- cargo-thiserror-impl.1.0.63
- cargo-unicode-ident.1.0.12
- cargo-wasm-bindgen.0.2.93
- cargo-wasm-bindgen-backend.0.2.93
- cargo-wasm-bindgen-macro.0.2.93
- cargo-wasm-bindgen-macro-support.0.2.93
- cargo-wasm-bindgen-shared.0.2.93
  • Next up is supporting both of these in one dependency tree, which might require adding a new flag to the cargo opam files
$ time dune exec -- bin/main.exe --repo /dev/shm/opam-giga-repository/packages cargo-ripgrep
cargo-aho-corasick.0.7.20 cargo-aho-corasick.1.1.3 cargo-bitflags.2.6.0 cargo-bstr.1.10.0 cargo-cfg-if.1.0.0 cargo-crossbeam.0.8.4 cargo-crossbeam-utils.0.8.20 cargo-dirs-next.2.0.0 cargo-dirs-sys-next.0.1.2 cargo-docopt.0.9.0 cargo-encoding_rs.0.8.34 cargo-encoding_rs_io.0.1.7 cargo-env_filter.0.1.2 cargo-env_logger.0.11.5 cargo-getrandom.0.2.15 cargo-globset.0.4.14 cargo-grep.0.3.1 cargo-grep-cli.0.1.10 cargo-grep-matcher.0.1.7 cargo-grep-printer.0.2.1 cargo-grep-regex.0.1.12 cargo-grep-searcher.0.1.13 cargo-hermit-abi.0.4.0 cargo-kernel32-sys.0.2.2 cargo-lazy_static.0.2.11 cargo-lazy_static.1.5.0 cargo-libc.0.2.158 cargo-libredox.0.1.3 cargo-log.0.4.22 cargo-memchr.0.1.11 cargo-memchr.2.7.4 cargo-memmap.0.7.0 cargo-memmap2.0.9.4 cargo-num_cpus.1.16.0 cargo-proc-macro2.1.0.86 cargo-quote.1.0.37 cargo-redox_users.0.4.6 cargo-regex.0.2.11 cargo-regex.1.10.6 cargo-regex-automata.0.4.7 cargo-regex-syntax.0.8.4 cargo-ripgrep.0.1.0 cargo-rustc-serialize.0.3.25 cargo-rustversion.1.0.17 cargo-serde.1.0.209 cargo-serde_derive.1.0.209 cargo-strsim.0.11.1 cargo-syn.2.0.76 cargo-term.0.7.0 cargo-termcolor.1.4.1 cargo-thiserror.1.0.63 cargo-thiserror-impl.1.0.63 cargo-thread_local.0.3.6 cargo-unicode-ident.1.0.12 cargo-utf8-ranges.1.0.5 cargo-walkdir.0.1.8 cargo-wasi.0.13.2+wasi-0.2.1 cargo-winapi.0.3.9 cargo-winapi-i686-pc-windows-gnu.0.4.0 cargo-winapi-util.0.1.9 cargo-winapi-x86_64-pc-windows-gnu.0.4.0 cargo-windows-sys.0.52.0 cargo-windows-targets.0.52.6 cargo-windows_aarch64_gnullvm.0.52.6 cargo-windows_aarch64_msvc.0.52.6 cargo-windows_i686_gnu.0.52.6 cargo-windows_i686_gnullvm.0.52.6 cargo-windows_i686_msvc.0.52.6 cargo-windows_x86_64_gnu.0.52.6 cargo-windows_x86_64_gnullvm.0.52.6 cargo-windows_x86_64_msvc.0.52.6 cargo-wit-bindgen-rt.0.30.0
- cargo-aho-corasick.0.7.20
- cargo-aho-corasick.1.1.3
- cargo-bitflags.2.6.0
- cargo-bstr.1.10.0
- cargo-cfg-if.1.0.0
- cargo-crossbeam.0.8.4
- cargo-crossbeam-utils.0.8.20
- cargo-dirs-next.2.0.0
- cargo-dirs-sys-next.0.1.2
- cargo-docopt.0.9.0
- cargo-encoding_rs.0.8.34
- cargo-encoding_rs_io.0.1.7
- cargo-env_filter.0.1.2
- cargo-env_logger.0.11.5
- cargo-getrandom.0.2.15
- cargo-globset.0.4.14
- cargo-grep.0.3.1
- cargo-grep-cli.0.1.10
- cargo-grep-matcher.0.1.7
- cargo-grep-printer.0.2.1
- cargo-grep-regex.0.1.12
- cargo-grep-searcher.0.1.13
- cargo-hermit-abi.0.4.0
- cargo-kernel32-sys.0.2.2
- cargo-lazy_static.0.2.11
- cargo-lazy_static.1.5.0
- cargo-libc.0.2.158
- cargo-libredox.0.1.3
- cargo-log.0.4.22
- cargo-memchr.0.1.11
- cargo-memchr.2.7.4
- cargo-memmap.0.7.0
- cargo-memmap2.0.9.4
- cargo-num_cpus.1.16.0
- cargo-proc-macro2.1.0.86
- cargo-quote.1.0.37
- cargo-redox_users.0.4.6
- cargo-regex.0.2.11
- cargo-regex.1.10.6
- cargo-regex-automata.0.4.7
- cargo-regex-syntax.0.8.4
- cargo-ripgrep.0.1.0
- cargo-rustc-serialize.0.3.25
- cargo-rustversion.1.0.17
- cargo-serde.1.0.209
- cargo-serde_derive.1.0.209
- cargo-strsim.0.11.1
- cargo-syn.2.0.76
- cargo-term.0.7.0
- cargo-termcolor.1.4.1
- cargo-thiserror.1.0.63
- cargo-thiserror-impl.1.0.63
- cargo-thread_local.0.3.6
- cargo-unicode-ident.1.0.12
- cargo-utf8-ranges.1.0.5
- cargo-walkdir.0.1.8
- cargo-wasi.0.13.2+wasi-0.2.1
- cargo-winapi.0.3.9
- cargo-winapi-i686-pc-windows-gnu.0.4.0
- cargo-winapi-util.0.1.9
- cargo-winapi-x86_64-pc-windows-gnu.0.4.0
- cargo-windows-sys.0.52.0
- cargo-windows-targets.0.52.6
- cargo-windows_aarch64_gnullvm.0.52.6
- cargo-windows_aarch64_msvc.0.52.6
- cargo-windows_i686_gnu.0.52.6
- cargo-windows_i686_gnullvm.0.52.6
- cargo-windows_i686_msvc.0.52.6
- cargo-windows_x86_64_gnu.0.52.6
- cargo-windows_x86_64_gnullvm.0.52.6
- cargo-windows_x86_64_msvc.0.52.6
- cargo-wit-bindgen-rt.0.30.0
dune exec -- bin/main.exe --repo /dev/shm/opam-giga-repository/packages   1159.56s user 19.92s system 99% cpu 19:40.02 total

$ time dune exec -- bin/main.exe --repo /dev/shm/opam-giga-repository/packages cargo-ripgrep.14.1.0
cargo-aho-corasick.1.1.3 cargo-anyhow.1.0.86 cargo-bstr.1.10.0 cargo-cfg-if.1.0.0 cargo-crossbeam-deque.0.8.5 cargo-crossbeam-epoch.0.9.18 cargo-crossbeam-utils.0.8.20 cargo-encoding_rs.0.8.34 cargo-encoding_rs_io.0.1.7 cargo-globset.0.4.14 cargo-grep.0.3.1 cargo-grep-cli.0.1.10 cargo-grep-matcher.0.1.7 cargo-grep-printer.0.2.1 cargo-grep-regex.0.1.12 cargo-grep-searcher.0.1.13 cargo-ignore.0.4.22 cargo-itoa.1.0.11 cargo-jemalloc-sys.0.5.4+5.3.0-patched cargo-jemallocator.0.5.4 cargo-lexopt.0.3.0 cargo-libc.0.2.158 cargo-log.0.4.22 cargo-memchr.2.7.4 cargo-memmap2.0.9.4 cargo-proc-macro2.1.0.86 cargo-quote.1.0.37 cargo-regex-automata.0.4.7 cargo-regex-syntax.0.8.4 cargo-ripgrep.14.1.0 cargo-ryu.1.0.18 cargo-same-file.1.0.6 cargo-serde.1.0.209 cargo-serde_derive.1.0.209 cargo-serde_json.1.0.127 cargo-syn.2.0.76 cargo-termcolor.1.4.1 cargo-te
xtwrap.0.16.1 cargo-unicode-ident.1.0.12 cargo-walkdir.2.5.0 cargo-winapi-util.0.1.9 cargo-windows-sys.0.52.0 cargo-windows-targets.0.52.6 cargo-windows_aarch64_gnullvm.0.52.6 cargo-windows_aarch64_msvc.0.52.6 cargo-windows_i686_gnu.0.52.6 cargo-windows_i686_gnullvm.0.52.6 cargo-windows_i686_msvc.0.52.6 cargo-windows_x86_64_gnu.0.52.6 cargo-windows_x86_64_gnullvm.0.52.6 cargo-windows_x86_64_msvc.0.52.6
- cargo-aho-corasick.1.1.3
- cargo-anyhow.1.0.86
- cargo-bstr.1.10.0
- cargo-cfg-if.1.0.0
- cargo-crossbeam-deque.0.8.5
- cargo-crossbeam-epoch.0.9.18
- cargo-crossbeam-utils.0.8.20
- cargo-encoding_rs.0.8.34
- cargo-encoding_rs_io.0.1.7
- cargo-globset.0.4.14
- cargo-grep.0.3.1
- cargo-grep-cli.0.1.10
- cargo-grep-matcher.0.1.7
- cargo-grep-printer.0.2.1
- cargo-grep-regex.0.1.12
- cargo-grep-searcher.0.1.13
- cargo-ignore.0.4.22
- cargo-itoa.1.0.11
- cargo-jemalloc-sys.0.5.4+5.3.0-patched
- cargo-jemallocator.0.5.4
- cargo-lexopt.0.3.0
- cargo-libc.0.2.158
- cargo-log.0.4.22
- cargo-memchr.2.7.4
- cargo-memmap2.0.9.4
- cargo-proc-macro2.1.0.86
- cargo-quote.1.0.37
- cargo-regex-automata.0.4.7
- cargo-regex-syntax.0.8.4
- cargo-ripgrep.14.1.0
- cargo-ryu.1.0.18
- cargo-same-file.1.0.6
- cargo-serde.1.0.209
- cargo-serde_derive.1.0.209
- cargo-serde_json.1.0.127
- cargo-syn.2.0.76
- cargo-termcolor.1.4.1
- cargo-textwrap.0.16.1
- cargo-unicode-ident.1.0.12
- cargo-walkdir.2.5.0
- cargo-winapi-util.0.1.9
- cargo-windows-sys.0.52.0
- cargo-windows-targets.0.52.6
- cargo-windows_aarch64_gnullvm.0.52.6
- cargo-windows_aarch64_msvc.0.52.6
- cargo-windows_i686_gnu.0.52.6
- cargo-windows_i686_gnullvm.0.52.6
- cargo-windows_i686_msvc.0.52.6
- cargo-windows_x86_64_gnu.0.52.6
- cargo-windows_x86_64_gnullvm.0.52.6
- cargo-windows_x86_64_msvc.0.52.6
dune exec -- bin/main.exe --repo /dev/shm/opam-giga-repository/packages   11.95s user 2.18s system 101% cpu 13.850 total