Renovating Rego

Summer is here, bringing thoughts of the beach, travel, and relaxation. However, summer isn’t without its chores; gardens need work, and summer houses require maintenance. Sometimes these chores can be enjoyable and even therapeutic, but most importantly, they keep our projects in good condition, allowing us to unwind and enjoy the summer months.

While I have neither a summer house nor a garden, my work on Open Policy Agent (OPA) for the past five years has produced a large pile of Rego policies scattered across my computer and various git repositories. A lot has happened in the world of OPA in that time and the policy language I learnt — and later, learnt to really enjoy —is something different from what it was when I got started.

All that code still works — OPA takes backwards compatibility very seriously — but new built-in functions, language constructs and syntax sugar makes modern Rego almost look like a different language at times. Importantly, a language that is easier to read, write and understand. Wouldn’t it be great to have all the Rego code I’ve written in the past look like that? Of course it would! As an added bonus, it’ll help me relax knowing it’ll continue to work after the coming OPA 1.0 release, where some of the currently optional features will be made mandatory. Looks like I’ve found my summer renovation project after all.

Renovating Rego!

While there are clear benefits of “renovating” Rego for one’s own sake, these benefits are multiplied many times over when those Rego policies are shared with others. Having a common and agreed upon baseline for what policies should look like — and tools to enforce that — helps make the policy authoring (and reading!) experience more productive, safe and enjoyable for everyone involved. In other words, the same unified experience OPA offers for policy across a wide and diverse tech stack applied to diverse teams and organizations working together on OPA and Rego.

Having renovated a large number of my own and other Rego projects recently made me realize that while there are some absolutely great tools for this around, there isn’t much written about the process of modernizing a Rego codebase.

With this article I’m hoping to change that by providing a simple step by step guide documenting  the process I follow for bringing old Rego code to modernity, maintainability and something to enjoy working with.

In order to get started renovating a Rego project, we’ll need the following tools installed:

Both should preferably be of the latest version. We’re trying to modernize things here after all!

NOTE: All the examples in this guide will assume that all Rego policies are stored in a directory called policy. Needless to say, you should change this to point to the location of your own policies.

NOTE: If you are following this guide with OPA 1.0, then you will need to use the --v0-v1 flag instead of --rego-v1 in both the opa check and fmt commands in order to check and reformat older Rego.

If the policy repo contains unit tests, take a look at them to get an understanding of what they’re testing. And of course, run them to see if they pass! Since the topic of unit testing is excellently covered by both  documentation and blogs already, we’ll leave the topic aside in this blog. Don’t take that to mean you should! Before you start renovating your Rego project, spend some time to ensure that you have working tests in place, and run them after every change you make.

The first step to proceed with is a preliminary check. Before we start digging into any code — let’s make sure it’s syntactically valid! OPA provides a tool to do just that, in the form of opa check.

Simply run the opa check command pointing at the directory where your Rego code resides:

opa check policy

If the command outputs nothing, you’ve passed, and may proceed to the next step!

The opa check command broadly reports two categories of errors — errors from parsing and errors from compilation.

Errors from parsing are easily spotted, as the error message will contain the text rego_parse_error. These errors are reported when the Rego code scanned is syntactically invalid. In other words, it’s not really Rego! Common causes for this — assuming the code is actually used somewhere — is that the scope of the check is too broad. Perhaps the directory checked contained a sub-directory with some old and experimental scratch files? Or perhaps someone thought it was a good idea to use a templating language to inject values into “Rego” files, like:

package acmecorp.{|{ .team }}.policies

I’d strongly advise you not to do that, but in case you need to, make sure to at least name those files using a different extension than .rego, like perhaps .rego.tpl.

If you’re seeing errors that don’t include the text rego_parse_error, your Rego is syntactically correct, and you can pat yourself on the back for that before figuring out why the Rego code fails to compile.

Compilation can fail for a number of reasons, but if parser errors sometimes happen because too many files (including some invalid ones) got included in the check, compilation errors commonly happen because too few files got included. Perhaps one of your policies is calling a function defined in another directory, and one which wasn’t included in the check? If so, make sure to include all the dependencies when calling the opa check command (which takes any number of files and directories as arguments). For example:

opa check policy helpers functions.rego

If that’s not the case, see below for how to identify and fix the error(s)!

The OPA Errors guide from Styra’s documentation portal is a great resource for learning more about common errors and how to fix them. If you encountered any error while running opa check and you don’t understand what it means, check to see if you can find it there! If not, let us know and we’ll add it.

Congratulations! You’ve made it past the preliminary check. If you had to make any changes to fix errors reported in the first step, now is a good time to commit them. We’re now going to up the game by running the same opa check command, but in strict mode. Strict mode adds a number of extra checks to the compilation step, identifying things that might be valid, but almost certainly not something you’ll want in your code, including:

Enabling strict mode for opa check is as simple as adding the --strict flag to the command:

opa check --strict policy

Important! If one of the strict mode checks fails, compilation stops at that check. This means that even when you’ve fixed all errors reported, there could be more errors found in subsequent steps. Re-run opa check --strict until no errors are reported.

Fixing problems reported at this stage should be pretty straightforward.

We’re making progress, and while we may not qualify as “modern” yet, we’re now at a stage where we know our Rego is valid and passes the strict mode checks. Well done! If you had to make any changes to fix issues reported by opa check --strict, make sure to commit them now so we’re at a clean stage when starting the next step.

Alright, so let’s start the modernization process! Modernizing Rego means embracing new language constructs like the use of the if keyword before rule bodies, or contains to define multi-value rules (i.e. a rule that generates a set of values). OPA v1.0 (to be released later this year) will make these constructs mandatory, but there’s no need to wait!

The opa fmt command is the built-in formatter for OPA, and should be used to ensure a uniform style is used across projects and teams. To help prepare for OPA 1.0, the opa fmt command provides a --rego-v1 flag that extends to normal formatting to also include rewriting rules for OPA 1.0 compatibility. That’s what we’ll be using!

Tip: If you haven’t used opa fmt on your Rego code in the past, I recommend running it first without the --rego-v1 flag set, and commit your code when done. That way, you’ll be able to see exactly what the “normal” formatter does first, and can then review the --rego-v1 changes separately.

Transforming our Rego code to OPA 1.0 modernity is simple:

opa fmt --write --rego-v1 policy

We’ve already explained the --rego-v1 flag. The --write flag ensures that the result of the formatting is written to the files rather than printed to the console.

Provided that the first two steps completed with no issues, or the issues reported have been fixed, you should have no problems to fix at this stage. In the unlikely case of opa fmt reporting any errors, please file an issue!

Before we move on, make sure to review the changes from the formatter and that you understand them. When done, commit the changes and move on to the next step.

At this stage, your Rego policies are known to be syntactically valid, pass all the strict mode checks and to be OPA 1.0 compliant. But even if we fixed a few mistakes along the way and had our policies rewritten to use more recent language constructs, our goal is more ambitious than mere compliance. We want our Rego policies to be modern and idiomatic.

For this we’ll need a more opinionated tool than OPA, and that tool is Regal. Regal is the linter for Rego, and with more than 80 rules to help identify bugs and common mistakes it’s become the tool of choice for policy authors to find and fix issues in their Rego code as early as possible.

Regal doesn’t just find bugs though. The idiomatic category of linter rules help identify code that can be made more — you guessed it — idiomatic, and the style category of rules provide opinionated advice for following modern conventions and coding style. Just what we need to accomplish our mission!

Using Regal to lint our code is simple. To lint our policy directory from the command line, we’ll simply point the regal lint command at it:

regal lint policy

The output of regal lint is a list of all issues Regal identified while linting. Depending on the size of your Rego codebase, the number of issues Regal reports can sometimes feel absolutely intimidating! Don’t worry though — it’s not uncommon at all for Regal to report hundreds of issues when linting code that hasn’t been touched in a while. While we could dive in right away to try and address all the issues reported, a more systematic approach is often more effective.

The approach I normally take is simple: address the most important issues first! If Regal finds potential bugs in our code, we’ll want to fix those before we worry about things like idiomatic coding style. Regal provides several means to ignore rules, including entire categories of rules from linting. To have Regal look only for bugs, we can exclude all rules before enabling only the bugs category:

regal lint --disable-all --enable-category=bugs policy

This normally brings the number of issues reported down to something more manageable, but in case it’s still overwhelming we can use an even stricter filter to narrow it down further. For example by using --enable to include only a single rule:

regal lint --disable-all --enable=rule-shadows-builtin policy

For every issue Regal reports, the output includes a pointer to the location in the code where the issue was found, and a link to the documentation for that linter rule. This is crucial! Before we start fixing the issues reported, we should first understand them. Not only does that help us avoid repeating them — we’ll learn something in the process.

With an understanding of why Regal considers something an issue, fix it following its documentation. For every rule fixed, commit your code and move on to the next one. If fixing all issues reported feels like a too big task to tackle right away, create a configuration file for Regal (.regal/config.yaml) where you can choose to ignore rules for the time being, and then have them enabled at a later time. This way you can start benefiting from having Regal integrated in your builds directly, even if there are things you’ll want to come back and fix later. See the Regal documentation on configuration for more information on the options available.

Besides the regal lint command, one tool that you can use to your advantage here is regal fix. As the name suggests, this command helps remediate issues as reported by the linter, and while only a limited subset of rules currently have a corresponding “fix”, this can save you a lot of time. Note that only “simple” rules in primarily the style category are covered by this tool, as fixing actual bugs is something you should do manually after having learnt about them.

Congratulations! If you’ve made it all the way through, and you’ve managed to fix at least most of the issues reported by Regal, you now have a policy project that is modern, idiomatic, and fun to work with! Commit your code, send it for review by your colleagues and enjoy your new status as a Rego renovator.

Just like any renovation project, renovating Rego is potentially both fun and rewarding. A more proactive approach to language updates, new best practices and linter rules is however almost certainly better. The best approach to avoid having to do large renovation projects in the future is to integrate as many of these processes as possible into your build and deployment pipeline. That way, you’ll be able to catch mistakes as soon as possible, and can fix them before they have a chance to accumulate into the next renovation project.

Utility projects like setup-opa and setup-regal make it trivial to install both OPA and Regal into your GitHub Actions pipeline. If you’re using another provider like GitLab, installation should be fairly straightforward too as both OPA and Regal are single file executables, or Docker images if you so prefer.

Since both OPA and Regal are updated continuously, make sure to watch each project for new updates, and have the versions you use both locally and in your build system updated accordingly.

If you’d like to discuss Rego renovation projects, OPA, Regal or anything related, please join me and many others in the Styra community on Slack!