Dune: A composable build system for OCaml

Keywords:

Build system?
For OCaml?
Composable?

Dune: A composable build system for OCaml

dune builds things. This is done through the basic concept of rules.

Dune rules

A set of dependencies

→

An action to execute

→

A target file

(rule
  (deps ...)
  (action ...)
  (target ...))

An action to execute

→

Create hello.txt

→

hello.txt

(rule
 (deps)
 (action
   (write-file hello.txt "print_endline \"Hello world\""))
 (target hello.txt))

hello.txt

→

Copy hello.txt to hello.ml

→

hello.ml

(rule
 (deps hello.txt)
 (action
   (copy hello.txt hello.ml))
 (target hello.ml))

hello.ml

→

ocamlopt -o hello.exe hello.ml

→

hello.exe

(rule
 (deps hello.ml)
 (action
  (run ocamlopt -o hello.exe hello.ml))
 (target hello.exe))

Combining rules

By combining rules, we can build targets from scratch.

Create hello.txt

→

hello.txt

→

Copy hello.txt to hello.ml

→

hello.ml

→

ocamlopt -o hello.exe hello.ml

→

hello.exe

(rule
 (deps)
 (action (write-file hello.txt "print_endline \"Hello world\""))
 (target hello.txt))

(rule
 (deps hello.txt)
 (action (copy hello.txt hello.ml))
 (target hello.ml))

(rule
 (deps hello.ml)
 (action (run ocamlopt -o hello.exe hello.ml))
 (target hello.exe))

OK, but how de we generate rules?

The `dune` files

Dune finds its rules in dune files. Those files are written using the sexp syntax:

{ "menu": {
   "id": "foo",
   "value": "This is a string"
  },
  "popup": [ "A", "B", "C" ]
}

(menu
  (id foo)
  (value "This is a string"))

(popup "A" "B" "C")

(parentheses) for scope,
atoms for names,
"strings" for strings,
to separate.

A (toplevel?) pair of parentheses and its content is called a stanza.

Example: The "menu stanza", the "popup stanza".

A complete documentation for the rule stanza can be found in Dune's documentation!

Building a target with `dune`

$ dune build <target>

Collect all rules,
Find the one that builds the target,
Build the dependencies (cf 2.) then execute the action.

Any question? What does "all rules" mean?

Rules are collected from all dune files below the root directory.

The root directory is given by the presence of a dune-project file.

The dune-project file also uses S-exp syntax. It indicates the root of a project, as well as some metadata about it.
It is versioned and must start with the version: eg (lang dune 3.7).

(lang dune 3.7)
(name odoc)
(documentation "https://ocaml.org/p/odoc")
(source (github ocaml/odoc))
(license ISC)
(authors
 "Anton Bachin <antonbachin@yahoo.com>"
  ...)

(cram enable)

(using mdx 0.3)

The full list of dune-package stanza is available in the documentation!

We can finally create a minimal dune project!

In a fresh directory, create a minimal dune-project file (eg (lang dune 3.9)),
Create a dune file, add some rules,
Call dune build <target> --verbose where <target> is the target of a rule,
Look for the produced target!

(rule
 (deps)
 (action (write-file hello.txt "print_endline \"Hello world\""))
 (target hello.txt))

(rule
 (deps hello.txt)
 (action (copy hello.txt hello.ml))
 (target hello.ml))

(rule
 (deps hello.ml)
 (action (run ocamlopt -o hello.exe hello.ml))
 (target hello.exe))

The file above is redundant. Simplify it, until dune complains!

(rule (action (write-file hello.txt "print_endline \"Hello world\"")))

(rule (action (copy hello.txt hello.ml)))

(rule
 (deps hello.ml)
 (action (run ocamlopt -o %{target} %{deps}))
 (target hello.exe))

Remarks

The files are built "out of tree", in <root>/_build/default.
Any file "in the tree" are copied "out of the tree".
dune rules outputs the list of (lowest-level) rules! dune build --verbose displays the commands run.

$ dune rules
((deps ((File (In_source_tree dune))))
 (targets ((files (_build/default/dune)) (directories ())))
 (context default)
 (action (copy dune _build/default/dune)))

((deps ((File (In_source_tree dune-project))))
 (targets ((files (_build/default/dune-project)) (directories ())))
 (context default)
 (action (copy dune-project _build/default/dune-project)))

((deps ((File (External /home/panglesd/.opam/5.2.0/bin/ocamlopt.opt))))
 (targets ((files (_build/default/hello.exe)) (directories ())))
 (context default)
 (action
  (chdir
   _build/default
   (run /home/panglesd/.opam/5.2.0/bin/ocamlopt.opt -o hello.exe hello.ml))))

((deps ((File (In_build_dir _build/default/hello.txt))))
 (targets ((files (_build/default/hello.ml)) (directories ())))
 (context default)
 (action (chdir _build/default (copy hello.txt hello.ml))))

((deps ())
 (targets ((files (_build/default/hello.txt)) (directories ())))
 (context default)
 (action
  (chdir
   _build/default
   (write-file hello.txt "print_endline \"Hello world\""))))
$ dune build --verbose hello.exe
[...]
Actual targets:
- _build/default/hello.exe
Running[1]: (cd _build/default && /home/panglesd/.opam/5.2.0/bin/ocamlopt.opt -o hello.exe hello.ml)

Targets, and more

$ dune build <target-or-some-other-thing>

A target is the righthand side of a rule. It belongs to a directory. It can be:

The name of the target of a rule in the dune file in the current directory.
```
$ dune build hello.exe
```
The name of the target of a rule in the dune file in a different directory.
```
$ dune build src/bin/hello.exe
$ dune build ./src/bin/hello.exe
$ dune build ../hello.exe
```
All paths is dune are unix-style, to work on all OSes.

$ dune show targets     # OR dune show targets <dir>
dune
dune-project
hello.exe
hello.ml
hello.txt

You may have seen:

$ dune build .
$ dune build

What does that mean, to build a directory? To build nothing?

A good approximation is that dune build . builds all targets in this directory and subdirectories! And dune build is dune build <root>.

But that's not completely true.

Aliases

Dune has a concept of "aliases". An alias is a build target that does not produce a file. It:

Has a name. Some are predefined: default, runtest, doc.
Is attached to a directory: src/default ≠ default

Has rules attached to it,

(rule
 (alias deploy)
 (action ./run-deployer.exe))

Triggers the build of all its attached rules
```
# Exec rules attached to deploy in:
$ dune build @deploy      # current dir and subdirs
$ dune build @@deploy     # current dir
$ dune build @foo/deploy  # directory foo and subdirs
$ dune build @@foo/deploy # directory  foo
```
- The default alias contains all rules from the directory.
  
  dune build . is equivalent to dune build @@default!
- The doc alias builds the docs. The runtest alias runs the tests.
- The fmt alias formats the files.

Can be (re)defined, have dependencies, be a dependency:

(alias
 (name my_own_alias)
 (deps
  (alias test-integration)))

To see all aliases, run dune show aliases!

$ dune show alias
all
default
fmt
ocaml-index

Caching

dune records the hash of dependencies of executed rules, to avoid executing them twice.

It will recompile only what is required!

h.txt

→

Extract first line

→

h2.txt

→

Copy

→

h3.txt

$ dune build h3.txt
$ echo "New line" >> h3.txt
$ dune build  # Will only execute the rule: "Extract first line"

Dune: A composable build system for OCaml

Dune has first class support for OCaml!

This is (in part) expressed by "higher level rules" specific to OCaml.

The two most important "high-level" rules are for defining libraries and executables.

Libraries

An OCaml library is a set of modules under a specific name. It includes all *.ml files from the directory as modules.

(library
  (name my_name))

It can have some dependencies on other libraries (internal or external).

(library
  (name my_name)
  (libraries lwt cmdliner))

It can be "private" or "public".

(library
  (name my_name)
  (public_name super_lib)
  (libraries lwt re))

Each of these is translated by dune into "lower-level" rules.

In a fresh folder in the hello project, declare a library in the dune file. Inspect the rules with dune rules --recursive.

Executables

(executable
  (name main)
  (public_name super_ls)
  (libraries cmdliner))

The dune init commands allows to create a blank project.

$ dune init proj test_project

Do that, inspect all files generated, verify you understand what they mean, check the output of dune rules!

What is dune exec bin/main.exe doing exactly?

Add a file with a .ml extension in test_project/lib/ and access it from test_project/bin/main.ml

The `library` stanza is complex!

The dependencies inside the library is inferred.

If multiple files are part of the same library, they can depend on each other.
```
(* lib/a.ml *)
let x = B.y

(* lib/b.ml *)
let y = 5
```

dune automatically computes the dependencies (using ocamldep) when generating the low-level rules.

$ dune build --verbose
[...]
(ocamldep.opt -modules -impl lib/b.ml) > [...]/hey__B.impl.d
[...]
(ocamldep.opt -modules -impl lib/a.ml) > [...]/hey__A.impl.d
[...]
$ cat [...]/hey__A.impl.d
lib/a.ml: B

From the test_project above, add several modules to the library. Make it fail by creating a cycle of dependencies, and understand the error message.

The modules are wrapped in a top-level module

When no file has the same name as the library, the modules are "wrapped" in a module.

├── foo.ml
├── bar.ml
└── dune
      (library
        (name test))

≈

(* test.ml generated by dune *)

module Foo = Foo

module Bar = Bar

When a file has the library name, some modules may not be accessible from the outside.

├── foo.ml
├── bar.ml
├── test.ml
└── dune
      (library
        (name test))

≈

(* test.ml handwritten by the user *)

module Foo = Foo

let y = Bar.y

Verify the above claim!

Note: Test__Foo and Test__Bar are generated because of this "wrapping".

Tests and cram tests, promotion mechanism

Dune has a concept of test: An executable that is built and run.

The high level rule is defined via the (test ...) stanza, by default attached to the runtest alias.

├── foo.ml
└── dune
      (test
        (name foo)
        (libraries fpath))

Additionally, if a <name>.expected file is in the directory, this file will be compared with the output of the execution.

The <name>.expected file can be updated with dune promote.

Use the test_project above. Add a test which calls print_endline "Hello", and make sure the output is the expected one.

Cram tests

Cram tests execute a cram file: a list of commands with an expected output.

run.t

  $ ocamlc -c -bin-annot unit.ml

  $ odoc compile --parent-id pkg --output-dir _odoc unit.cmt
  $ odoc link _odoc/pkg/page-index.odoc

Test the compilation

  $ odoc compile-index --root _odoc/pkg
  File "index.mld", line 1, characters 42-47:
  Warning: Duplicate 'dir1/' in (children).

(cram
 (deps %{bin:odoc} %{bin:odoc_print}))

Do the exercises from 3 to 7!

Dune: A composable build system for OCaml

From the repo:

Dune is composable, meaning that multiple Dune projects can be arranged together, leading to a single build that Dune knows how to execute. This allows for monorepos of projects.

Dune makes simultaneous development on multiple packages a trivial task.

Example of modularity: You often can just rename directories! But there is more.

The project world and the installed world

Dune makes a distinction between the project world and installed world.

The project world is private to the project.
The installed world is the rest of the world. Example: your switch.

They do not share the same namespace!

(library
  (name name_in_project)
  (public_name name_outside_project))

The public_name is optional.

The installed world can be accessed directly.

(library
  ...
  (libraries libname))

Dune will first search libname in the project world, then in the installed world.

Public libraries can be moved to the installed world:

$ dune build @install
$ tree _build/install/default
_build/install/default
├── bin
│   └── hello_world -> ../../../default/bin/main.exe
└── lib
    └── hello_world
        ├── dune-package -> ../../../../default/hello_world.dune-package
        ├── META -> ../../../../default/META.hello_world
        └── opam -> ../../../../default/hello_world.opam

The install alias builds all public libraries/executables in _build/install, using their public names.

$ dune install --verbose
[...]
Installing /home/panglesd/.opam/5.2.0/lib/hehehehe/META
Installing /home/panglesd/.opam/5.2.0/lib/hehehehe/dune-package
Installing /home/panglesd/.opam/5.2.0/lib/hehehehe/opam
Installing /home/panglesd/.opam/5.2.0/bin/hehehehe

The dune install command copies it to the installed world: your opam switch.

Note: The public_name must make it part of a package: pkgname or pkgname.whatyouwant.

(library
  ...
  (public_name pkgname))

But what is a package?

Packages

The "package" word in Dune refers to Opam packages.

Any public library/executable must be part of a package.

A project may have multiple packages.

Packages can be defined in two ways:

By the (package ...) stanza in dune-package, if it contains (generate_opam_files true),
By the presence of a <pkgname>.opam file, otherwise.

Libraries/executables/tests can be assigned to a package in two ways:

From the public_name: (public_name <pkgname>) and (public_name <pkgname>.<subname>).
Using the (package <pkgname>) stanza. Useful for non-public libraries, tests, ...

Vendoring

Vendoring is copying the source code of a dependency in your own project.

Sometimes to modify it,
Sometimes to remove an (opam) dependency.

Dune vendors some libraries to use them without a dependency cycle,
Merlin vendors the OCaml typer and make it more resistant to errors.

How to vendor a dune project in your own dune project?

Simple: copy-paste the vendored project into a directory from your project!

How does this work?

The multiple dune-project files helps to distinguish project roots, and what to consider "installed world" and "project world".

 ├── dune-project
 │
 ├── my_lib/
 │   ├─ file.ml
 │   └─ dune
 │
 └── vendor/vendored_project_name/
     │
     ├─ dune-project
     │
     └── their_lib
         ├─ file.ml
         └─ dune

my_lib and their_lib are not part of the same project, so they refer to each other using public names.
Dune computes the list of root from the furthest dune-project.

You can also add a (vendored_dirs) stanza to dune-project to restrict the builds in the vendored library.

Conclusion

Dune is a composable build system for OCaml!

And it has great documentation. Let's go for a tour!

Do the exercise from 8 to 10!

Dune: A composable build system for OCaml

Dune: A composable build system for OCaml

Dune rules

Combining rules

The dune files

Building a target with dune

Remarks

Targets, and more

Aliases

Caching

Dune: A composable build system for OCaml

Libraries

Executables

The library stanza is complex!

The dependencies inside the library is inferred.

The modules are wrapped in a top-level module

Tests and cram tests, promotion mechanism

Cram tests

Dune: A composable build system for OCaml

The project world and the installed world

Packages

Vendoring

How does this work?

Conclusion

The `dune` files

Building a target with `dune`

The `library` stanza is complex!