smlpkg tutorial

by Alex Nelson, 22 August 2024

There are few package managers for Standard ML. In fact, the only one I am aware of is smlpkg which lacks adequate documentation.

This post just constitutes a “note to self”, which I hope may be helpful to others.

Building smlpkg

This requires MLkit, MLton, or any other Standard ML compiler which can work with MLton’s “Basis build system”.

For MLton, you have to build it using the command:

~/src/smlpkg$ MLCOMP=mlton make clean all

Unfortunately, it does not build under Poly/ML or SML/NJ. I tried using Chris Cannam’s sml-buildscripts, but no dice. So if you want to build smlpkg using polybuild, you have extra work to do for yourself.

Installing smlpkg

Assuming you somehow succeeded in building smlpkg…now what?

Well, if you tried make install, it will simply create a subdirectory bin/ inside the smlpkg directory. So, for me, it created ~/src/smlpkg/bin/ and placed a copy of the binary smlpkg there.

You need to either place this in a place which is accessible to $PATH (e.g., in /usr/local/bin/) or you need to add this directory to the $PATH variable.

You can verify you’ve done this correctly by opening up a terminal, and running smlpkg. You should get something like the following:

alex@lisp:~$ smlpkg
Usage: smlpkg [--version] [--verbose] [--help] <command> ...:

Commands:
   add        Add another required package to sml.pkg.
   check      Check that sml.pkg is satisfiable.
   init       Create a new sml.pkg and a lib/ skeleton.
   fmt        Reformat sml.pkg.
   sync       Populate lib/ as specified by sml.pkg.
   remove     Remove a required package from sml.pkg.
   upgrade    Upgrade all packages to newest versions.
   versions   List available versions for a package.

Using smlpkg

Assuming you have now built smlpkg and its directory is accessible to PATH, how do we use it?

As I understand it, the intent is that:

You keep your Standard ML projects in a subdirectory $somewhere/lib/
You must decide the location of the parent directory $somewhere (usually it’s ~, I guess?)
Starting projects. Assuming you have a Github username username and you wish to start a new project called new-project, you run the command (in the directory $somewhere) smlpkg init github.com/username/new-project
- Example: if I (with username pqnelson on Github) wanted to start a project called indexer, then I would run the command
```
alex@lisp:~/src/$ mkdir indexer
alex@lisp:~/src/$ cd indexer
alex@lisp:~/src/indexer/$ smlpkg init github.com/pqnelson/indexer
Created directory 'lib/github.com/pqnelson/indexer'.
Wrote sml.pkg.
```
  This effectively creates a new empty directory ~/src/indexer/lib/github.com/pqnelson/indexer/ and does some book-keeping in ~/sml.pkg
  
  Observe that the directory which contains all my source code is ~/src/indexer/lib/github.com/pqnelson/indexer/ — this is analogous to the src/ directory in a Maven project. More importantly, smlpkg apparently expects to find lib/github.com/pqnelson/indexer when it tries to download a copy for any users of my code — smlpkg will throw errors if it does not find this directory.
  
  If I want to add unit tests, I would create the subdirectory ~/src/indexer/lib/github.com/pqnelson/indexer/test/ which will then contain a test.mlb file, test.sml test runner file, and a tests/ subdirectory.
- IMPORTANT: You MUST create a sml.pkg file which initially looks like:
```
package github.com/pqnelson/indexer
    
require {
}
```
  Just change the uesrname from pqnelson to whatever your handle is, and change the project name from indexer to whatever your project is called.
  
  This is an undocumented aspect to smlpkg and it is important to do, otherwise other users will receive errors when they try to add your package to their projects.
  
  (I really do not understand why smlpkg forces us to manage this file, nor why it lacks documenting the importance that this file exists.)
- IMPORTANT 2: You must version your projects with tags with prefix v — so the initial release will be v0.0.1; without the leading v, smlpkg will throw errors when other people try to use your package.
  
  Again, this is not well documented, and it seems like smlpkg should check for versions without a leading v prefix.
Using packages. If you wish to use my fancy indexer package, you need to say smlpkg add github.com/pqnelson/indexer — this will only modify the ~/sml.pkg file. You also need to run the command smlpkg sync to fetch the code.

Concluding remarks

Is this worth it? …I feat not so much for me, but hopefully for you.

But this also leaves much to be desired, since smlpkg lacks the functionality of, say, Rust’s cargo or Clojure’s lein (or even Java’s mvn).

For example, there’s no way to run tests with smlpkg, unlike the other tools I listed off. But smlpkg isn’t designed to do that.

There’s a lot of repetitive work involved which I would have expected a package manager to abstract away — for example, whenever I start a new project, I always have to remind smlpkg I use github.com, my username is pqnelson, and it always requires a lib/github.com/pqnelson/<project-name> directory storing all the source code for the project (as opposed to simply requiring a src directory).

And this is entirely hardcoded into smlpkg, by the way.

The design decision seems well-motivated in the abstract: have a lib subdirectory which stores the names of all the websites which have the relevant git repositories, then the usernames are subdirectories of these, and the library needed is the sub-subdirectory. This cleanly separates out dependencies, and prevents name collisions (if the user wanted to use both github.com/pqnelson/xunit and gitlab/some-other-user/xunit, for some reason, then these two packages are always separated from each other).

But it seems like forcing me to manage this in each package I develop is foolish. Why not have a hidden directory ~/.smlpkg/cache/ which does this for me? This avoids duplicate downloads (in case I write a unit testing framework and use it in all my packages, for example: smlpkg would download a copy for each package I develop).

I also don’t understand why smlpkg cannot keep track of those packages and add them to the .mlb build files. This would allow me to develop a package without redundantly bloated github.com/pqnelson/new-package/ subdirectories.