Module registry

This article shows how Spacelift can help you manage Terraform Modules

Intro

In Terraform, modules help you abstract away common functionality in your infrastructure. The name of a module managed by Spacelift is of the following form:

spacelift.io/<organization>/<module_name>/<provider>

In this name we have:

• The source module registry - spacelift.io is used here;

• The organization which owns and maintains the module;

• The module name, this will usually be the best shorthand descriptor of what the module actually does, i.e. it could be starting a machine with an HTTP server running.

• The main Terraform provider this module is meant to work with, i.e. the provider for the cloud service the resources should be created on.

You can use a module in your Terraform configuration this way:

module "my-birthday-cake" {
  source  = "spacelift.io/spacelift-io/cake/oven"
  version = "4.2.0"
  
  # Inputs.
  eggs  = 5
  flour = "200g"
}

output "my-birthday-cake" {
  value = {
    weight = module.my-birthday-cake.weight
    allergens = module.my-birthday-cake.allergens
  }
}

As you can see, we've explicitly used a module which can make cakes using an oven. We can specify variables the module depends on, and finally use the outputs the cake module exports.

Spacelift obviously lets you host modules, but it also does much more, providing you with robust CI/CD for your modules, leading us to the question...

Why host your Modules on Spacelift?

Spacelift provides everything you need to make your module easily maintainable and usable. There is CI/CD for multiple specified versions of Terraform, which "runs" your module on each commit. You get an autogenerated page describing your Module and its intricacies, so your users can explore them and gather required information at a glimpse. It's also deeply integrated with all the features Stacks use which you know and love, like Environments, Policies, Contexts and Worker Pools.

Setting up a Module

Git repository structure

You will have to set up a repository for your module, the structure of the repository should be as follows:

.
├── .spacelift
│   ├── config.yml
├── README.md
├── main.tf
├── output.tf
└── variables.tf

Each module must have a config.yml file in its .spacelift directory, containing information about the module along with any test cases. Details of the format of this file can be found in the module configuration section of this page.

You can check out an example module here: https://github.com/spacelift-io/terraform-spacelift-example

The source code for a module can be stored in a subdirectory of your repository because you can specify the project root when configuring your module in Spacelift. An example of a repository containing multiple modules can be found here: https://github.com/spacelift-io/multimodule

Spacelift setup

In order to add a module to Spacelift, navigate to the Modules section of the account view, and click the Add module button:

The setup steps are pretty similar to the ones for stacks. First you you point Spacelift at the right repo and choose the "tracked" branch - note that repositories whose name don't follow the convention are filtered out:

In the behavior section there are just three settings: administrative, worker pool and project root. You will only need to set administrative to true if your module manages Spacelift resources (and most likely it does not). Setting worker pool to the one you manage yourself makes sense if the module tests will be touching resources or accounts you don't want Spacelift to access directly. Plus, your private workers may have more bandwidth than the shared ones, so you may get feedback faster. The project root let's you specify the module source code root inside of your repository:

Last but not least, you will be able to add a name, provider, labels and description.

The name and provider will be inferred from your repository name if it follows the terraform-<provider>-<name> convention. However, if it can't be inferred or you want a custom name, then you can specify them directly. The final module slug will then be based on the name.

Environment, contexts and policies

Environment and context management in modules is identical to that for stacks. The only thing worth noting here is the fact that environment variables and mounted files set either through the module environment directly, or via one of its attached stacks will be passed to each of the test cases for the module.

Attaching policies works similar, too, though there are 2 types of policies that cannot be attached to modules: task and trigger. Task policies are only applicable to tasks, which make no sense for modules that are essentially stateless. Trigger policies are highly dependent on state changes, too, and so far we haven't seen a use case for making them available to modules. That said, feel free to prove us wrong.

Module configuration

While by convention a single Git repository hosts a single module, that root module can have multiple submodules. Thus, we've created a way to create a number of test cases:

# The version of the configuration file format
version: 1
# Your module version - must be changed to release a new version
module_version: 0.1.1

# Any default settings that should be used for all test cases
test_defaults:
  before_init: ["terraform fmt -check"]
  runner_image: your/runner:image

# The set of tests to run to verify your module works correctly
tests:
  - name: Test the module with 0.12.7
    terraform_version: 0.12.7
    environment:
      TF_VAR_bacon: tasty

  - name: Test the submodule with 0.13.0
    # project_root can be set if your test case is not stored in the root directory
    project_root: submodule
    terraform_version: 0.13.0
    environment:
      TF_VAR_cabbage: awful
      
  - name: Ensure that the submodule can fail
    # You can use negative to indicate that the test case is expected to fail
    negative: true
    project_root: submodule
    terraform_version: 0.13.0

This configuration is nearly identical to the one described in the Runtime configuration section, with both test_defaults and each test case accepting the same configuration block. Note that settings explicitly specified in each test case will override those in the test_defaults section. Also, notice that each test case has a name, which is a required field.

While we don't check for name uniqueness, it's always good idea to give your test cases descriptive names, as these are then used to report job status on your commits and pull requests.

Tests

In order to verify that your module is working correctly, Spacelift can run a number of test cases for your module. Note how the configuration above allows you to set up different runtime environment (Docker image, Terraform version) etc. If you want to test the module with different inputs, these can be passed as Terraform variables (starting with TF_VAR_) through the test-level environment configuration option - see above for an example.

While coverage is not yet calculated or enforced, we suggest that tests set up all resources defined by the module and submodules. It's generally a good idea to provide examples in the examples/ directory of your repository showing users how they can use the module in practice. These examples can then become your test cases, and you can test them against multiple supported Terraform version to maximize compatibility.

While running each test case, Spacelift will - as usual, initialize, plan and apply the resource, but also destroy everything in the end, checking for errors. In the meantime, it will also validate that some resources have actually been created by the tests - though as for now it does not care what these are.

A test case can be marked as negative, which means that it is expected to fail. In an example above one of the test cases is expected to fail if one of the required Terraform variables is not set. Negative test cases are as useful as positive ones because they can prove that the module will not work under certain - unexpected or erroneous - circumstances.

Test cases will be executed in parallel (as much as worker count permits) for each of the test cases version you have specified in the module configuration.

Tests run both on proposed and tracked changes. When a tracked change occurs, we create a Version. Versions are described in more detail in the next section.

Each test case will have its own commit status in GitHub / GitLab.

Versions

Whenever tests succeed on a tracked change, a new Version is created based on the module_version in the configuration. Important thing to note is that Spacelift will not let you reuse the number of a successful version, and will require you to strictly follow semantic versioning - ie. you can't go to from 0.2.0 to 0.4.0, skipping 0.3.0 entirely.

Two proposed git flow are as follows:

The first one would be to have a main branch and create feature branches for changes. Whenever you merge to the main branch you bump the version and release it.

If you want more control over release schedules, you could go with the following:

• A release branch

• A main branch

• Feature branches

Whenever you add a new functionality, you may want to create a feature branch and open Pull Request from it to the main branch. Whenever you want to release a new version, you merge the main branch into the release branch.

You can also use Git push policies to further customize this.

If no test cases are present, the version is immediately marked green.

Modules in practice

In order to use modules, you have to source them from the Spacelift module registry. You can generate the necessary snippet, by opening the page of the specific module version, and clicking show instructions.

Sharing modules

Unlike Stacks, modules can be shared between Spacelift accounts in a sense that while they're always managed by a single account, they can be made accessible to an arbitrary number of other accounts.

In order to share the module with other accounts, please add their names in subdomain form (all lowercase) in the module settings' Sharing section:

This can also be accomplished programmatically using our Terraform provider.

Using modules outside of Spacelift

Spacelift modules can be used outside of Spacelift by providing access credentials from generated from API keys. The credentials file generated upon the creation of each API key contains a section explaining how a key can be used to set up credentials in the Terraform configuration file (.terraformrc). To learn more about this please refer directly to Terraform documentation.