Continuous Integration and Localisation

Hi, I’m Thomas, a graduate engineer at pixiv, and this is the fourth entry to pixiv’s advent calendar.

Continuous Integration and Localisation

Continuous integration is great. Developers are happy because bugs are caught early and are easy to track down. Managers are happy because project risk is reduced and features are delivered often. Here at pixiv we deploy continuously to production many times per day.

When it comes to localisation, however, continuous integration leaves translators forever chasing a moving target. Source strings are added or changed often and the bottleneck to continuous deployment often ends up being localisation. Translators have to be contacted frequently for trivial amounts of work. This results in a lot of management time compared to a large pre-release string freeze for translation in conventional projects, and the cost of managing translators can often be greater than the cost of translation itself.

However, these problems can be minimised by moving towards an automated and continuous approach to translation management.

Translation Management

Using a web-based approach to translation management allows for translations to be completed and applied incrementally. This could be a tool created in-house, using such as Translate Toolkit to manage different formats, or one of many existing platforms.

At pixiv we’ve chosen Weblate to manage translations. Translators work collaboratively and make suggestions which are then reviewed and accepted. We can even see the diffs for translation edits, provide screen captures for source strings and keep a history of translation changes. Neat!

With it we use a workflow tightly integrated with git. Each time the source strings are updated Weblate fetches them to its own local repository via a hook and rebuilds the list of strings for translators. When a translation is completed Weblate then creates a new commit, crediting the author. This means translations can be pushed as they are completed instead of in batches. Translators can also be automatically notified of new strings, reducing the management overhead continuous integration brings with it.

Automated Checks and Previous Translations

Managing a large number of rapidly changing source strings has its own difficulties. There are over six thousand Japanese source strings in pixiv. With this number it becomes difficult to track new translations and their changes. If translators accidentally introduce newlines or mistype HTML these mistakes might go unnoticed by developers and cause problems. Automating these checks stops translator errors slipping through to release.

At the same time, the larger a project gets, the harder it becomes to keep translations consistent. Translation memory is a must, especially when there are multiple contributors for a language. Many different translation memory tools can be connected directly to Weblate. By suggesting previous similar translations both time and money is saved.

For agile teams, taking a continuous localisation approach by integrating software projects into a web-based translation management server instead of going back and forth between translators allows localisation to be completed incrementally and management overhead reduced.