A reliable checklist for setting up new Scala projects
Key settings and practices to consider when you want to improve your builds or your development workflow. Several of them work well as part of templates.
Starting a fresh Scala project usually ends with copying and pasting build configurations from somewhere else. A build configuration that has been around for a while tends to collect plenty of handy settings and plugins. Some organizations move their default settings into internal plugins or rely on features such as GitHub templates. Such templates seldom make sense outside the context they were made for, so trying to build one definitive project scaffold for every purpose is not worth it.
What I’d like to offer instead is a checklist of key settings and practices to consider when you want to improve your builds or your development workflow. Several of them work well as part of templates. Others are simply essential steps I like to have in place before anyone begins contributing.
Checklist overview (TLDR)
If you’d rather skim the whole checklist without reading through dull paragraphs, here it is:
- .gitignore
- sbt-dotenv and direnv
- local.conf
- .sbtopts
- Java version
- Compiler flags
- .scalafmt.conf
- .scala-steward.conf or renovate
- publish settings
- .mergify.conf or automerge
- GitHub actions and CODEOWNERS
- Essential libraries
- TTLs for SNAPSHOTs
- Bonus: nix
.gitignore
Prepare a thorough .gitignore file up front, so irritating, unnecessary local files don’t end up in pull requests.
sbt-dotenv and direnv
Custom project configuration values are often supplied through environment variables. In Scala projects, for instance, it is common to reference them in application.conf and then set the actual values for local, test and production environments. direnv is a widely used tool that automatically loads the environment into your current shell as soon as you enter a project directory containing a .envrc file, with content such as
export KAFKA_HOST=localhost
export KAFKA_PORT=9131
Hooking such a file up to the sbt console is very handy, particularly when you run it from an IDE. The sbt-dotenv plugin is worth a look for this. It makes sense to set it up as a global plugin so that it applies to all of your local projects.
Start by creating ~/.sbt/1.0/plugins/dotenv.sbt containing:
addSbtPlugin("nl.gn0s1s" % "sbt-dotenv" % "3.0.0")
I also suggest switching the plugin’s default filename from .env to .envrc. That is the standard name used by direnv, a more popular tool that serves other purposes too. To do so, add a .sbt/1.0/envrc.sbt with:
ThisBuild / envFileName := ".envrc"
After that, every time you start an sbt console, it will load environment variables from the .envrc file located in the workspace root. The file should stay local, so make sure git ignores it; see the .gitignore section.
(Note: sbt-dotenv relies on reflection. Starting with Java 17 reflection is forbidden, so sbt-dotenv crashes at startup. The next section, about .sbtopts, explains how to fix this).
local.conf
Alongside environment variables, another useful practice is overriding configuration entries from application.conf. Add local.conf to .gitignore so that developers can create a src/main/resources/local.conf file that includes application.conf, for example:
include "application.conf"
akka {
loggers = ["akka.event.slf4j.Slf4jLogger"]
loglevel = "DEBUG"
stdout-loglevel = "DEBUG"
}
The file takes effect when the app is started with -Dconfig.file=path/to/src/main/resources/local.conf. You can pass this parameter to sbt in several ways:
- when running the plain sbt console:
sbt 'set javaOptions+="-Dconfig.file=path/to/src/main/resources/local.conf"; test:runMain com.myapp.MyMain' - in the IntelliJ sbt console: simply add
-Dconfig.file=...to “VM parameters” under “sbt settings” - in IntelliJ “Run” configurations: as with the sbt console, use the “VM parameters” field
- and likewise in other tools
.sbtopts
This is a handy place to define sbt or Java options for builds and the sbt console. To share a common set with every developer and with CI, put them into .sbtopts, for instance:
Enabling reflection in newer java versions: (needed for sbt-dotenv on Java 17+)
-J--add-opens=java.base/java.util=ALL-UNNAMED
-J--add-opens=java.base/java.lang=ALL-UNNAMED
Defining specific memory settings:
-J-Xmx5G
-J-Xms1G
-J-Xss2M
Defining specific sbt options (the complete list is here):
-Dsbt.task.timings=true
Java version
Every now and then, a lengthy investigation ends with the discovery that your local Java version differs from the one CI uses – and the version deployed with the application may be something else altogether. Aligning the Java version across every layer of your project can be done, but it is not straightforward.
Locally, developers frequently rely on sdkman or jenv. The former is general-purpose and can be combined with .envrc, but the only approach I managed to find is far from pretty. Otherwise, go with either .sdkmanrc or .java-version, the dedicated files these tools use.
Your Continuous Integration pipeline, on the other hand, most likely has its own JDK configuration (see this GitHub Action, which reads it from a file), and then there is sbt-native-packager, which requires a base image definition in build.sbt.
I had never seen all of these JDK versions brought together in one place until I came across nix, which I will describe in depth in a separate blog post.
Compiler flags
Question: can you spot the problem in this code?
def doStuff(data: Data): F[Unit] = {
Logger[F].info(s"Logging important metadata: ${data.metadata}")
processor.process(data).void
}
Answer: Clearly, the code lacks a >> that chains the F[Unit] returned by the logger with the other F[Unit] returned by the processor call, which is our end result. But is it really that clear? Most likely yes when you are told to look for bugs in two lines of code, but in a larger context it is easy to miss.
The trouble is that the Scala 2.13 compiler stays completely silent, and mistakes like this can slip through code reviews! The default compiler settings nearly always need tightening, for instance to warn about unused and discarded values and other possible problems. The sbt-tpolecat plugin provides an excellent set of sbt compiler flags. Pair it with selective warning suppression and you’re all set. For instance, put:
scalacOptions += "-Wconf:src=src_managed/.*:s"
into your build configuration to mute warnings for classes that ScalaPB generates from protobuf files.
.scalafmt.conf
Introducing formatting rules late in a project’s life leads to painful updates, so agreeing on a good rule set at the start is essential. Below are some rules I’m fond of, but keep in mind that code style conventions are highly subjective:
maxColumn = 140
// breaks long class/def definitions into multiline
verticalMultiline.atDefnSite = true
// forces newline before and after implicit params
newlines.implicitParamListModifierForce = [before, after]
// only format files tracked by git
project.git = true
// PreferCurlyFors: Replaces parentheses into curly braces in for comprehensions that contain multiple enumerator generators
// RedundantBraces, RedundantParens: Remove redundant braces, parens
rewrite.rules = [PreferCurlyFors, RedundantBraces, RedundantParens, SortImports]
// Add spaces next to curly braces in imports
spaces.inImportCurlyBraces = true
// more than standard align rules https://scalameta.org/scalafmt/docs/configuration.html#alignpresetmore
style = defaultWithAlign
// all infix operators can be exempted from applying continuation indentation
indentOperator.exemptScope = all
rewriteTokens {
"⇒" = "=>"
"→" = "->"
"←" = "<-"
}
.scala-steward.conf or renovate
Keeping dependencies up to date is an involved process that calls for regular work and often follows specific rules. These days it can be automated with tools such as Scala Steward or renovate, supplemented by dependabot. At times you may want to skip certain libraries, set PR titles, or cap the number or frequency of updates. Adjust your Scala Steward rules (see this reference and put together a clearly defined config.
publish settings
In multi-module sbt projects, the choice of which modules should be handled in which way is often quite specific:
- Published as artifacts
- Published as docker images
- Published with scaladoc
- Not published
My initial build.sbt nearly always gains from:
-
A default list of resolvers
-
A path to local ivy credentials
-
Publishing configuration
-
Basic docker settings such as the base image or package name
-
Compile / packageDoc / publishArtifact := falseincommonSettingsfor every module, which skips scaladoc and cuts build time considerably.
A set oflazy val noPublishSettings = Seq( publish := {}, publishLocal := {} ) lazy val dockerNoPublishSettings = Seq( Docker / publishLocal := {}, Docker / publish := {} )
that can later be applied as
.settings(noPublishSettings)
.settings(dockerNoPublishSettings)
to the modules that should not be published.
Set up publishing ahead of time, so that future repositories based on your template need as little extra work as possible.
.mergify.conf or automerge
There are many situations in which merging PRs automatically is useful. For instance:
- auto-merging approved PRs opened by a dependency update bot
- auto-merging PRs that touch specific files
Take a look at a sample .mergify.conf with some of the rules we apply. Bear in mind that mergify is free for open source code only, so GitHub automerge or renovate might suit your project better. A sample configuration:
# Enables automatic merge of master -> PR for Scala Steward PRs
# created by user 'github-actions', when all checks
# pass (including review approvals)
name: automerge
on:
pull_request:
types:
- labeled
- unlabeled
- synchronize
- edited
- ready_for_review
- reopened
- unlocked
pull_request_review:
types:
- submitted
check_suite:
types:
- completed
jobs:
automerge:
runs-on: ubuntu-latest
if: github.event.pull_request.user.login == 'github-actions'
steps:
- name: automerge
uses: "pascalgn/automerge-action@v0.15.3"
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
MERGE_FILTER_AUTHOR: "github-actions"
MERGE_METHOD: "squash"
MERGE_LABELS: ""
MERGE_RETRIES: "15"
MERGE_RETRY_SLEEP: "20000"
UPDATE_LABELS: ""
GitHub actions and CODEOWNERS
Above I gave one example of a GitHub action that fits nicely into a project template - automerge. Which other actions do you think are essential to have before contributors start committing?
Speaking of contributors, I like to add a .github/CODEOWNERS file, which lets pull request reviewers be assigned automatically based on its contents. You don’t have to list individuals; the file also accepts teams, such as
@my-organization/alpha-team
Essential libraries
Libraries don’t belong in project templates, but they deserve a spot on the checklist for starting a new codebase. Many of them count as essential utilities that we frequently want in our project from the outset. A fellow Scala developer, for instance, put together an excellent list: cats, refined, iron, chimney, ducktape, scala-newtype, monocle, quicklens, enumeratum, derevo, macwire, jam, PPrint, sbt-thank-you-stars.
Your starter set might also contain libraries for logging, metrics, testing and other concerns shared across your organization.
Caution! I’m not recommending a “common libraries” plugin shared between projects. On the contrary - I steer clear of that approach, because it causes awful dependency problems and produces a bloated module carrying a hundred megabytes of things you may not need. Keeping a checklist of suggestions that teams review when starting a new codebase works far better.
TTLs for SNAPSHOTs
In some situations, developing against SNAPSHOT dependencies makes sense. Picture two teams swapping an “api” artifact often during an early prototype phase while they settle on a communication contract. If that applies to you, remember that Coursier’s TTL is 24 hours, so your artifacts will not refresh even after a new version is published. To shorten that window, pick one of two options:
-
Add the TTL configuration to your template (
build.sbt)csrConfiguration := csrConfiguration.value.withTtl(Some(1.minute))
or
- Use the
.envrcfile mentioned earlier, add exportCOURSIER_TTL=60s(or whatever value you need), and then restart your sbt session.
Note that even when sbt picks up these settings and reloads SNAPSHOTs correctly, IntelliJ appears to struggle with noticing new artifacts. You may need to restart it every time a new SNAPSHOT is downloaded.
Bonus: nix
Earlier I briefly brought up nix as a tool that could extend sbt-native-packager to build docker images with exactly the same jdk used in development and CI. Nix offers more, such as defining complete environments of sdks, libraries and system packages that are loaded only while you are in your project directory (nix shells). The talk “Nix all the things” is a gentle introduction to nix for beginners.
Summary
I hope this checklist helps you put together sound build configurations early in your projects. I’m certain there are plenty of points I’m not aware of, so if you have ideas for additions or updates, please leave a comment!