+
|
|
|
|
+

An earlier post announced Updo. From the Updo Examples, let’s walk through the Cabal example to see how to introduce Updo.

A few of benefits of Updo for haskell/cabal are:

• Package Groups

  Packages go into publish-to-hackage, test and benchmark groups. Helpful to keep a project like this, with a lot of local packages, organized.

• Dual Projects

  Contributors get to pick Cabal or Stack as their build tool.

• Validated Dependencies

  With only a few exceptions listed explicitly as constraints, we get all dependencies from stackage, a set of packages that build together.

Projects, Before and After

There are many projects in the haskell/cabal repository:

$ tree -P 'cabal.project*|stack.yaml*' -L 1 --prune
.
├── cabal.project
├── cabal.project.buildinfo
├── cabal.project.coverage
├── cabal.project.doctest
├── cabal.project.libonly
├── cabal.project.meta
├── cabal.project.release
├── cabal.project.validate
├── cabal.project.validate.libonly
├── cabal.project.weeder
└── stack.yaml

0 directories, 11 files

Before conversion, the Stack project doesn’t build¹ but Updo conversion will fix that by keeping both default projects in sync with an upstream configuration.

After conversion we can build Cabal with Cabal itself as before but now we can also build it with Stack!

$ cabal clean
$ cabal build all --enable-tests --enable-benchmarks
Resolving dependencies...
Build profile: -w ghc-9.4.7 -O1
...
$ cabal build all --enable-tests --enable-benchmarks
Up to date

$ stack purge
$ stack build --test --no-run-tests --bench --no-run-benchmarks
...
Completed 17 action(s).

Conversion Steps

The steps of converting a project to Updo, using conversion of Cabal for example, are:

1. Ignores

   Ignore the working (.updo) and bootstrap (updo) folders in .gitignore:

   +.updo
   +updo

2. Versions

   Put stackage resolver and GHC version into project-versions.mk², not bothering with separate upgrade versions for now. The process for adding an upgrade version is the same as for adding an initial current version.

   GHC_VERSION ?= 9.4.7
   STACKAGE_VERSION ?= lts-21.19
   GHC_UPGRADE ?= 9.4.7
   STACKAGE_UPGRADE ?= lts-21.19

3. Stackage Config

   Download a cabal.config file from stackage matching the resolver version and save it to project-stackage/${STACKAGE-VERSION}.config. This likely won’t work as-is. No worries, we’ll comment out version constraints that clash later.

   $ mkdir -p project-stackage
   $ curl -sSL https://www.stackage.org/lts-21.19/cabal.config \
       > project-stackage/lts-21.19.config

4. Group Packages

   Add configuration under project-dhall/ghc-${GHC-VERSION}. We’ll break the packages up into groups and as we’re not yet upgrading we’ll use an empty list for upgrades yet to do.

   -- project-dhall/pkg-groups.dhall
   [ "benchmarks", "hackage", "tests" ]
   
   -- project-dhall/pkgs/benchmarks.dhall
   [ "cabal-benchmarks", "solver-benchmarks" ]
   
   -- project-dhall/pkgs/hackage.dhall
   [ "Cabal", "Cabal-syntax", "cabal-install", "cabal-install-solver" ]
   
   -- project-dhall/pkgs/tests.dhall
   [ "Cabal-QuickCheck"
   , "Cabal-described"
   , "Cabal-tests"
   , "Cabal-tree-diff"
   , "cabal-testsuite"
   ]
   
   -- project-dhall/pkgs-upgrade-todo.dhall
   [] : List Text

5. Source Repositories

   Cabal doesn’t use any source repository packages so we can leave all of these empty³.

   -- project-dhall/ghc-9.4.7/deps-external.dhall
   -- project-dhall/ghc-9.4.7/deps-internal.dhall
   -- project-dhall/ghc-9.4.7/forks-external.dhall
   -- project-dhall/ghc-9.4.7/forks-internal.dhall
   [] : List { loc : Text, tag : Text, sub : List Text }

6. Text Templates

   Add text templates for the ways we want to generate projects. Pasted verbatim, the following dhall2config⁴ template for Cabal and dhall2stack template for Stack put the snippet content before the default template content.

   -- project-dhall/ghc-9.4.7/text-templates/dhall2config.dhall
   \(stackage-resolver : Text) ->
   \(ghc-version : Text) ->
     let project-dhall2config = ../../../updo/text-templates/dhall2config.dhall
   
     in  ''
         ${./cabal-snippet.dhall}
         ${project-dhall2config stackage-resolver ghc-version}
         ''

   -- project-dhall/ghc-9.4.7/text-templates/dhall2stack.dhall
   let TYPES = ./../../../updo/types.dhall
   
   let null = https://prelude.dhall-lang.org/List/null
   
   in  \(pkgs-done : List Text) ->
       \(stackage-resolver : Text) ->
         let pkgs-todo = ../../pkgs-upgrade-todo.dhall
   
         let pkg-config =
               { constraints = ./../constraints.dhall
               , source-pkgs =
                 { deps-external = ./../deps-external.dhall
                 , deps-internal = ./../deps-internal.dhall
                 , forks-external = ./../forks-external.dhall
                 , forks-internal = ./../forks-internal.dhall
                 }
               }
   
         in  ''
             ${./stack-snippet.dhall (None Text)}
             ${../../../updo/text-templates/dhall2stack.dhall
                 stackage-resolver
                 ( if    null Text pkgs-todo
                   then  TYPES.PkgSet.AllPkgs pkgs-done
                   else  TYPES.PkgSet.PkgUpgrade
                           { todo = pkgs-todo, done = pkgs-done }
                 )
                 pkg-config}
             ''

   Note

   The dhall2stack template is more complicated than the dhall2config template because everything generated goes into one ghc-x.y.z.dhall2stack.yaml file so it has to handle upgrades whereas the root ghc-x.y.z-dhall2config.project imports generated project-config/pkgs/*.config package groups indirectly through project-config/pkgs.config.

   In project-config/pkgs/*.config files, partitioning of packages into those included in the upgrade project and those yet to do is done by the installed updo-pkg-groups executable or the ./updo/project-dhall2config/pkg-groups.hs script invoked by a make recipe and not by the dhall2config template.

7. Snippets

   Snippets are used to add extra configuration to the generated projects, configuration unknown to Updo. Compare generated projects with those same files before the conversion to see what’s missing.

   -- project-dhall/ghc-9.4.7/text-templates/cabal-snippet.dhall
   ''
   tests: True
   optional-packages: ./vendored/*/*.cabal
   constraints: rere -rere-cfg
   program-options
     ghc-options: -fno-ignore-asserts
   ''
   
   -- project-dhall/ghc-9.4.7/text-templates/stack-snippet.dhall
   \(stackage-resolver : Optional Text) ->
     let resolver =
           merge
             { None = ""
             , Some =
                 \(r : Text) ->
                   ''
   
                   resolver: ${r}''
             }
             stackage-resolver
   
     in  ''
         user-message: "WARNING: This stack project is generated."
         allow-newer: true
         flags:
           rere:
             rere-cfg: false
         ghc-options:
           "$locals": -fhide-source-paths
         ${resolver}
         ''

   Note

   We need allow-newer: true because cabal-testsuite has a custom setup depending on 3.10.* of Cabal and Cabal-syntax while the rest of the package depends on 3.11.*.

8. Bootstrap

   Add the entry and bootstrapping Updo makefile, project-files.mk:

   # project-files.mk
   # To use installed executables instead of *.hs scripts, set these to true.
   PKG_GROUPS_HS_EXE ?= false
   PKGS_SORTED_HS_EXE ?= false
   PKGS_UPGRADE_DONE_HS_EXE ?= false
   
   include project-versions.mk
   include updo/Makefile
   
   project-nix/ghc-%/sha256map.nix: ghc-%.sha256map.nix
   	mkdir -p $(@D) && cp $^ $@
   
   .PHONY: all
   all: \
     projects \
     project-nix/ghc-$(GHC_VERSION)/sha256map.nix \
     project-versions.nix
   
   # To make stack.yaml or cabal.project and no other, mark the file we copy from
   # as intermediate. This is all we want when not doing a GHC upgrade.
   #
   # Comment out these .INTERMEDIATE targets to allow these files to be kept.
   .INTERMEDIATE: ghc-$(GHC_VERSION).$(CABAL_VIA).project
   .INTERMEDIATE: ghc-$(GHC_UPGRADE).$(CABAL_VIA).project
   .INTERMEDIATE: ghc-$(GHC_VERSION).$(STACK_VIA).yaml
   .INTERMEDIATE: ghc-$(GHC_UPGRADE).$(STACK_VIA).yaml
   
   .DEFAULT_GOAL := all
   
   UPDO_VERSION ?= 1.0.0
   HACKAGE := http://hackage.haskell.org/package
   UPDO_URL := ${HACKAGE}/updo-${UPDO_VERSION}/updo-${UPDO_VERSION}.tar.gz
   
   updo/Makefile:
   	rm -rf updo
   	curl -sSL ${UPDO_URL} | tar -xz
   	mv updo-${UPDO_VERSION} updo
   	chmod +x $$(grep -RIl '^#!' updo)

9. Constrain Versions

   Try to generate projects with make. If this fails, Stack will complain the loudest.

   $ make -f project-files.mk
   ...
     * directory must match >=1.2 && <1.4, but this GHC boot package has been
       pruned from the Stack configuration.  You need to add the package
       explicitly to extra-deps. (latest matching version is 1.3.8.1).
     * process must match >=1.2.1.0 && <1.7, but this GHC boot package has
       been pruned from the Stack configuration. You need to add the package
       explicitly to extra-deps. (latest matching version is 1.6.17.0).
     * directory must match >=1.2 && <1.4, but this GHC boot package has
       been pruned from the Stack configuration. You need to add the package
       explicitly to extra-deps. (latest matching version is 1.3.8.1).
     * process must match >=1.2.1.0 && <1.7, but this GHC boot package has
       been pruned from the Stack configuration. You need to add the package
       explicitly to extra-deps. (latest matching version is 1.6.17.0).

   Use the suggestions from Stack to add version equality constraints:

   -- project-dhall/ghc-9.4.7/constraints.dhall
   [ { dep = "directory", ver = "1.3.8.1" }
   , { dep = "filepath", ver = "1.4.100.4" }
   , { dep = "process", ver = "1.6.17.0" }
   , { dep = "rere", ver = "0.2" }
   , { dep = "semaphore-compat", ver = "1.0.0@rev:1" }
   , { dep = "unix", ver = "2.8.2.1" }
   ]

   Note

   All the recommendations from Stack match cabal freeze versions before the conversion, except for process-1.6.18.0 and unix-2.8.3.0.

10. Fixup Unsatisfiable Version Constraints

    Where there are unsatisfiable version constraints with the Cabal solver, comment out the relevant line from the stackage-sourced cabal.config that we saved locally:

    -- project-stackage/lts-21.19.config
    -- NOTE: Due to revisions, this file may not work. See:
    -- https://github.com/fpco/stackage-server/issues/232
    
    -- Stackage snapshot from: http://www.stackage.org/snapshot/lts-21.19
    -- Please place this file next to your .cabal file as cabal.config
    -- To only use tested packages, uncomment the following line:
    -- remote-repo: stackage-lts-21.19:http://www.stackage.org/lts-21.19
    with-compiler: ghc-9.4.7
    constraints:
    ...
        -- Cabal installed,
        -- cabal-install ==3.8.1.0,
        -- cabal-install-solver ==3.8.1.0,
        -- Cabal-syntax installed,
        -- directory installed,
        -- filepath installed,
        -- process installed,
        -- unix installed,