Overview

One of my favorite parts of a new project is having the chance to address the sorts of things that always rose to the level of annoyance without ever quite making it over the treshold of problematic.

For me, one of those things was dealing with localization keys. They removed context around the usage of strings, they lead to unnecessary duplication, and they imposed and extra overhead & easily forgotten step of reconiciling the underlying file to the keys specified in the code.

I wanted to try to make the whole experience of dealign with localized strings a bit more pleasant in a new React Native project, inspired by the CSS-in-JS approach libraries like styled-components have introduced. This led to me exploring Javascript ASTs and writing a few tools that have had a major positive impact on my localization workflow.

Internationalization (I18n)

As W3C puts it

Internationalization is the design and development of a product, application or document content that enables easy localization for target audiences that vary in culture, region, or language.

There’s much more to localizing a product than simply translating it, but that’s going to be the focus here. How can you make it easier for developers to specify the abstract, pre-localization form of a string that can at runtime be replaced by what the user needs to see?

Locale-key based approaches

One of the more common ways I’ve seen to handle that abstract string specification is by introducing an intermediate value which can be used to look-up the actual string.

For example, in your user facing code you would have

<p><%= I18n.t('views.example.hello') %></p>

With corresponding strings files

# config/locales/en.yml
en:
  views:
    example:
      hello: Hello

# config/locales/es.yml
es:
  views:
    example:
      hello: Hola

When the user views the page, based on their language preferences the correct string can be retrieved and displayed.

My experience has been that this works with small projects, or large amounts of discipline but once the file grows past a certain size it can be very easy to wind up with the same string specified at multiple keys, or multiple UI elements pointed at the same key. In the latter case, if you ever want to change the copy of that element somebody could naively update the value of the given locale key (especially if they key doesn’t exactly match the English value to begin with) which has now had the unintended consequence of changing the content of other UI elements.

The locale key decouples the view from the localizations, which is much nicer than having to manage the localization in the view itself, but means it’s possible for somebody to forget to add a string or remove a string as the code changes. You also lose the nice ability to find a file by searching for the content of strings, since you now need to find the corresponding key which can be built in all sorts of dynamic ways.

Requirements

My main goal when I set out to define the localization strategy of this new app was to cut out the middleman of the locale key, and do so in a way that didn’t impose any extra work on whoever was adding/removing strings.

After discussing with a colleague who was responsible for working with the vendor that did the actual translation work, I also learned that a gap in our existing approach was in being able to convey context to the translators. While we were working with the vendor to better help the translators understand the usage of the strings they were translating, their current workflow had them just looking at a file of strings and trying to guess how they were being used. They had no way of knowing if a string was a statement or an instruction, which could vastly change the way a string could be translated.

We decided that being able to category strings into one of:

1. Title
2. Button
3. Label (essentially anything that wasn’t a title or button)

would convey what was needed.

We also needed to account for two specific cases of a string:

1. That the string contains a number and needs to pluralize its contents accordingly
2. That the string should be translated different per platform (this is relatively rare, but had come up in the past)

Proposed approach

As mentioned above, I took a lot of inspiration here from styled-components, in that components could be “decorated” with the metadata needed for them to be shown as desired, rather than having the intermediary of classes and identifiers mapping to a large external file.

We’re using Typescript in the project, so the resulting translation method has the following signature

type LocalizationCategory = 'TITLE' | 'BUTTON' | 'LABEL'

type LocalizationOptions = {
  platformSpecific?: Boolean
  identifier?: String
  pluralized?: Boolean
}

type StringContents = String | PluralizationContent | PlatformSpecificContent

type PlatformSpecificContent = {
  ios: String | PluralizationContent
  android: String | PluralizationContent
}

type PluralizationContent = {
  other: String
  zero: String
  one: String
}

const localize = (category: LocalizationCategory,
                  args1?: LocalizationOptions) =>
                 (string: StringContents,
                  args2?: TranslateOptions) => string

resulting in components that in the simplest case look like

const MyComponent = () => {
  const helloWorld = localize('TITLE')('Hello, world!')

  return (
    <View>
      <Text>{ helloWorld }</Text>
    </View>
  )
}

while the actual method call can get as elaborate as

localize('TITLE', { platformSpecific: true, pluralized: true })({
  ios: {
    other: 'Hello {{count}} times!',
    zero: 'Hello no times!',
    one: 'Hello one time!',
  },
  android: 'Hello from an Android phone!',
}, { count })

with all optionals supplied.

But how do we go from this method call to a localized string?

And now we get to the AST

And now we get to the heart of the matter/ the headline of the post — walking the Javascript abstract syntax tree (AST).

If you’ve made it this far, I’m going to assume you have at least a high level understanding of them, but the important concept to get is that an AST is a structure for representing the relationships between constructs (e.g. variables, methods, literals) in a code file. Traversing the AST of our codebase allows us to easily see where we’re invoking our localization methods and what arguments we’re giving them so that we can build our locale files and corresponding keys entirely behind the scenes.

Take the simple case call

localize('TITLE')('Hello, world!')

We can use the phenomenal AST Explorer to see that being the scenes Javascript understands this as

{
  "type": "Program",
  "start": 0,
  "end": 35,
  "body": [
    {
      "type": "ExpressionStatement",
      "start": 0,
      "end": 34,
      "expression": {
        "type": "CallExpression",
        "start": 0,
        "end": 34,
        "callee": {
          "type": "CallExpression",
          "start": 0,
          "end": 17,
          "callee": {
            "type": "Identifier",
            "start": 0,
            "end": 8,
            "name": "localize"
          },
          "arguments": [
            {
              "type": "Literal",
              "start": 9,
              "end": 16,
              "value": "TITLE",
              "raw": "'TITLE'"
            }
          ]
        },
        "arguments": [
          {
            "type": "Literal",
            "start": 18,
            "end": 33,
            "value": "Hello, world!",
            "raw": "'Hello, world!'"
          }
        ]
      }
    }
  ],
  "sourceType": "module"
}

This looks a bit byzantine, but the important thing to note is that by looking throughout our code for CallExpressions with callee name localize we can fairly robustly build a list of every invocation of localize throughout our codebase.

By inspecting then the arguments of both the callee and the parent node (remember that we’re using the function builder localize = () = () => { string } syntax) we can read out the string literals we’re passing in and construct the locale key on the fly as just the literal itself. For the more elaborate forms of the function supporting pluralization, or platform specific variants we just stringify the arguments and use them as keys. It looks a bit odd, but since we’re now capable of generating the key on the fly from both the writing & reading side, you don’t ever really need to look at it.

I’m not going to go too far into the actual code for parsing the AST because once you know the nodes that contain the localize calls, it’s mostly just a bunch of object access grabbing all of the necessary info, but it’s worth highlighting the tool that gives you those nodes to begin with.

To traverse the tree and allow me to operate on the necessary nodes, I used Acorn, a tremendously powerful JS parser capable of much more than what I needed for it here.

With the parser (I used acorn-loose because it’s enough in its syntax handling to allow me to use it on Typescript), after which the acorn-walk package provides a simple syntax for returning you any given type of node, e.g.

const fs = require('fs')
const Parser = require('acorn-loose')
const walk = require('acorn-walk')

const getStringsFromFile = (path, strings) => {
  const ast = Parser.parse(fs.readFileSync(path), { sourceType: 'module' })

  walk.simple(ast, {
    CallExpression(node) {
      if (
        node.callee.callee?.name == 'localize'
      ) {
        ...
      }
    }
  }

Once the right tooling is in place (feel free to email me if you have any questions about how this works in practice), you can write a localize call anywhere in your codebase, run the walker/builder script and have a fully up-to-date localizations file.

Using localize

So now we have a localize function, and a way to use invocations of that function to build a strings file, the last missing piece is what that localize function actually does at runtime.

This is pretty straightforward, since behind the scenes we actually are just using keys, we can just use any of the great i18n libraries out there for Javascript. We settled on i18n-js but it could be any number of the options out there. We stringify the arguments to the function using the same logic the AST walker does and presto, localizations.

Downsides

So having gone through all of the above, there are a few downsides to this approach that jump out immediately, the largest being that this won’t handle localizing non-literals. For example if you had some code like

let hello = 'Hello, world!'
localize(hello)

you’d have a problem because the AST has no way of knowing that hello evaluates to 'Hello, world!'. It looks easy enough in that example, but hello could be defined anywhere else in your codebase in any manner of ways. To really know what the value of hello is, you’d need to effectively interpret the codebase and that becomes a much longer blogpost.

In practice, we’ve found that avoiding these cases is pretty easy — there might be some times where you want to inject in strings into some code running elsewhere, but we haven’t had any pressing need to do so since we started using this approach some months ago.

Another potential downside is that there are myriad ways to call a function that looks different from the AST to the idiomatic usage (e.g. evaling localize). Again, this mostly just comes down to only being a problem if you do it — we haven’t had any issues avoiding this problem either.

Finally, we write these localizations out to a yaml file, and it doesn’t like keys that have periods in them. I just .replace(/\./g, '##PERIOD##'). This one I readily admit probably has a much better solution.

Upsides

With keys being built automatically behind the scenes, a whole class of issues is removed. Developers no longer need to manually update the strings file, one command will walk the entire project and update it as needed (the script runs fast enough that it could probably configured to auto-run but we haven’t gotten around to setting that up). We regenerate the file on each run, so if strings have been removed those changes happen automatically as well.

We have greater locality between our strings and our code, so it’s easier to find where strings are used, and we’ve added additional context to them to assit the translators. There’s also no longer any risk of changes in one place effecting changes in another. Keys are based on the contents of the localization call so if the same string is used in two places it can be represented with only one line in the localizations file, but as soon as one of them is changed it becomes its own key without altering the other.

Conclusion

All told, I’m very happy with how this approach turned out. At the expense of some strictness in invocation method we’ve now obviated the need the need to manage a giant file of strings and every time I need to add a new localization that little missing bit of friction makes me happy. If you’re interested in more details of how this was implemented, or have solved this problem in a different way I’d love to hear about it — get in touch!