Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: Liam-Williams/go-i18n
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: master
Choose a base ref
...
head repository: EverlongProject/go-i18n
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref
Able to merge. These branches can be automatically merged.
  • 2 commits
  • 20 files changed
  • 1 contributor

Commits on Mar 29, 2021

  1. CLOUD-487: support string:string maps in go-i18n 

    This flat file format assumes everything is a map of strings to objects. We want to be able to support string:string, since that's the default that Localise uses if there's only one translation. We also want to be able to support string:map, to allow specifying plural rules.

Additionally, update our fork to include proper module information, and fix a couple of tests that had go vet failures.
    @reillywatson
    reillywatson committed Mar 29, 2021
    Copy the full SHA
    f5728ab View commit details

Commits on Mar 31, 2021

  1. BACK-750: fix encoding of fallback translations (nicksnyder#2) 

    We want to use the fallback behaviour instead of storing thousands of identical strings (and instead of adding English strings to the French locale files). However, the fallback behaviour of go-i18n is buggy: it uses text/template for found translations, but html/template for fallbacks. This causes different encoding behaviour!

Use text/template instead so fallback behaviour matches the normal translation behaviour.
    @reillywatson
    reillywatson authored Mar 31, 2021

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    GPG key ID: 4AEE18F83AFDEB23
    Expired
    Verified
    Learn about vigilant mode
    Copy the full SHA
    d79109e View commit details
18 changes: 9 additions & 9 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
go-i18n [![Build Status](https://travis-ci.org/Liam-Williams/go-i18n.svg?branch=master)](http://travis-ci.org/Liam-Williams/go-i18n) [![Sourcegraph](https://sourcegraph.com/github.com/Liam-Williams/go-i18n/-/badge.svg)](https://sourcegraph.com/github.com/Liam-Williams/go-i18n?badge)
go-i18n [![Build Status](https://travis-ci.org/EverlongProject/go-i18n.svg?branch=master)](http://travis-ci.org/EverlongProject/go-i18n) [![Sourcegraph](https://sourcegraph.com/github.com/EverlongProject/go-i18n/-/badge.svg)](https://sourcegraph.com/github.com/EverlongProject/go-i18n?badge)
=======

go-i18n is a Go [package](#i18n-package) and a [command](#goi18n-command) that helps you translate Go programs into multiple languages.
* Supports [pluralized strings](http://cldr.unicode.org/index/cldr-spec/plural-rules) for all 200+ languages in the [Unicode Common Locale Data Repository (CLDR)](http://www.unicode.org/cldr/charts/28/supplemental/language_plural_rules.html).
* Code and tests are [automatically generated](https://github.com/Liam-Williams/go-i18n/tree/master/i18n/language/codegen) from [CLDR data](http://cldr.unicode.org/index/downloads)
* Code and tests are [automatically generated](https://github.com/EverlongProject/go-i18n/tree/master/i18n/language/codegen) from [CLDR data](http://cldr.unicode.org/index/downloads)
* Supports strings with named variables using [text/template](http://golang.org/pkg/text/template/) syntax.
* Translation files are simple JSON, TOML or YAML.
* [Documented](http://godoc.org/github.com/Liam-Williams/go-i18n) and [tested](https://travis-ci.org/Liam-Williams/go-i18n)!
* [Documented](http://godoc.org/github.com/EverlongProject/go-i18n) and [tested](https://travis-ci.org/EverlongProject/go-i18n)!

Package i18n [![GoDoc](http://godoc.org/github.com/Liam-Williams/go-i18n?status.svg)](http://godoc.org/github.com/Liam-Williams/go-i18n/i18n)
Package i18n [![GoDoc](http://godoc.org/github.com/EverlongProject/go-i18n?status.svg)](http://godoc.org/github.com/EverlongProject/go-i18n/i18n)
------------

The i18n package provides runtime APIs for fetching translated strings.

Command goi18n [![GoDoc](http://godoc.org/github.com/Liam-Williams/go-i18n?status.svg)](http://godoc.org/github.com/Liam-Williams/go-i18n/goi18n)
Command goi18n [![GoDoc](http://godoc.org/github.com/EverlongProject/go-i18n?status.svg)](http://godoc.org/github.com/EverlongProject/go-i18n/goi18n)
--------------

The goi18n command provides functionality for managing the translation process.
@@ -23,7 +23,7 @@ Installation

Make sure you have [setup GOPATH](http://golang.org/doc/code.html#GOPATH).

go get -u github.com/Liam-Williams/go-i18n/goi18n
go get -u github.com/EverlongProject/go-i18n/goi18n
goi18n -help

Workflow
@@ -116,11 +116,11 @@ Here is an example of the default file format that go-i18n supports:
]
```
To use a different file format, write a parser for the format and add the parsed translations using [AddTranslation](https://godoc.org/github.com/Liam-Williams/go-i18n/i18n#AddTranslation).
To use a different file format, write a parser for the format and add the parsed translations using [AddTranslation](https://godoc.org/github.com/EverlongProject/go-i18n/i18n#AddTranslation).
Note that TOML only supports the flat format, which is described below.
More examples of translation files: [JSON](https://github.com/Liam-Williams/go-i18n/tree/master/goi18n/testdata/input), [TOML](https://github.com/Liam-Williams/go-i18n/blob/master/goi18n/testdata/input/flat/ar-ar.one.toml), [YAML](https://github.com/Liam-Williams/go-i18n/blob/master/goi18n/testdata/input/yaml/en-us.one.yaml).
More examples of translation files: [JSON](https://github.com/EverlongProject/go-i18n/tree/master/goi18n/testdata/input), [TOML](https://github.com/EverlongProject/go-i18n/blob/master/goi18n/testdata/input/flat/ar-ar.one.toml), [YAML](https://github.com/EverlongProject/go-i18n/blob/master/goi18n/testdata/input/yaml/en-us.one.yaml).
Flat Format
-------------
@@ -170,7 +170,7 @@ and name of substructures (ids) should be always a string.
If there is only one key in substructure and it is "other", then it's non-plural
translation, else plural.
More examples of flat format translation files can be found in [goi18n/testdata/input/flat](https://github.com/Liam-Williams/go-i18n/tree/master/goi18n/testdata/input/flat).
More examples of flat format translation files can be found in [goi18n/testdata/input/flat](https://github.com/EverlongProject/go-i18n/tree/master/goi18n/testdata/input/flat).
Contributions
-------------
8 changes: 8 additions & 0 deletions go.mod
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
module github.com/EverlongProject/go-i18n

go 1.16

require (
github.com/pelletier/go-toml v1.8.1
gopkg.in/yaml.v2 v2.4.0
)
8 changes: 8 additions & 0 deletions go.sum
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/pelletier/go-toml v1.8.1 h1:1Nf83orprkJyknT6h7zbuEGUEjcyVlCxSUGTENmNCRM=
github.com/pelletier/go-toml v1.8.1/go.mod h1:T2/BmBdy8dvIRq1a/8aqjN41wvWlN4lrapLU/GW4pbc=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
6 changes: 3 additions & 3 deletions goi18n/constants_command.go
View file
Original file line number Diff line number Diff line change
@@ -13,9 +13,9 @@ import (
"text/template"
"unicode"

"github.com/Liam-Williams/go-i18n/i18n/bundle"
"github.com/Liam-Williams/go-i18n/i18n/language"
"github.com/Liam-Williams/go-i18n/i18n/translation"
"github.com/EverlongProject/go-i18n/i18n/bundle"
"github.com/EverlongProject/go-i18n/i18n/language"
"github.com/EverlongProject/go-i18n/i18n/translation"
)

type constantsCommand struct {
2 changes: 1 addition & 1 deletion goi18n/doc.go
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// The goi18n command formats and merges translation files.
//
// go get -u github.com/Liam-Williams/go-i18n/goi18n
// go get -u github.com/EverlongProject/go-i18n/goi18n
// goi18n -help
//
// Help documentation:
2 changes: 1 addition & 1 deletion goi18n/gendoc.sh
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
go install
echo "// The goi18n command formats and merges translation files." > doc.go
echo "//" >> doc.go
echo "// go get -u github.com/Liam-Williams/go-i18n/goi18n" >> doc.go
echo "// go get -u github.com/EverlongProject/go-i18n/goi18n" >> doc.go
echo "// goi18n -help" >> doc.go
echo "//" >> doc.go
echo "// Help documentation:" >> doc.go
6 changes: 3 additions & 3 deletions goi18n/merge_command.go
View file
Original file line number Diff line number Diff line change
@@ -11,9 +11,9 @@ import (

"gopkg.in/yaml.v2"

"github.com/Liam-Williams/go-i18n/i18n/bundle"
"github.com/Liam-Williams/go-i18n/i18n/language"
"github.com/Liam-Williams/go-i18n/i18n/translation"
"github.com/EverlongProject/go-i18n/i18n/bundle"
"github.com/EverlongProject/go-i18n/i18n/language"
"github.com/EverlongProject/go-i18n/i18n/translation"
toml "github.com/pelletier/go-toml"
)

4 changes: 1 addition & 3 deletions goi18n/testdata/en-us.flat.json
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
{
"program_greeting": {
"other": "Hello world"
},
"program_greeting": "Hello world",

"person_greeting": {
"other": "Hello {{.Person}}"
42 changes: 29 additions & 13 deletions i18n/bundle/bundle.go
View file
Original file line number Diff line number Diff line change
@@ -5,15 +5,15 @@ import (
"bytes"
"encoding/json"
"fmt"
"html/template"
"io/ioutil"
"path/filepath"
"reflect"
"sync"
"text/template"
"unicode"

"github.com/Liam-Williams/go-i18n/i18n/language"
"github.com/Liam-Williams/go-i18n/i18n/translation"
"github.com/EverlongProject/go-i18n/i18n/language"
"github.com/EverlongProject/go-i18n/i18n/translation"
toml "github.com/pelletier/go-toml"
"gopkg.in/yaml.v2"
)
@@ -96,7 +96,7 @@ func parseTranslations(filename string, buf []byte) ([]translation.Translation,
return nil, err
}

m := make(map[string]map[string]interface{})
m := make(map[string]interface{})
for k, v := range tree.ToMap() {
m[k] = v.(map[string]interface{})
}
@@ -112,7 +112,7 @@ func parseTranslations(filename string, buf []byte) ([]translation.Translation,
}
return parseStandardFormat(standardFormat)
} else {
var flatFormat map[string]map[string]interface{}
var flatFormat map[string]interface{}
if err := unmarshal(ext, buf, &flatFormat); err != nil {
return nil, fmt.Errorf("failed to unmarshal %v: %v", filename, err)
}
@@ -185,20 +185,36 @@ func parseStandardFormat(data []map[string]interface{}) ([]translation.Translati
// and passes it to parseStandardFormat.
//
// Flat format logic:
// key of data must be a string and data[key] must be always map[string]interface{},
// key of data must be a string and data[key] must be either a string or a map[string]interface{},
// but if there is only "other" key in it then it is non-plural, else plural.
func parseFlatFormat(data map[string]map[string]interface{}) ([]translation.Translation, error) {
func parseFlatFormat(data map[string]interface{}) ([]translation.Translation, error) {
var standardFormatData []map[string]interface{}
for id, translationData := range data {
dataObject := make(map[string]interface{})
dataObject["id"] = id
if len(translationData) == 1 { // non-plural form
_, otherExists := translationData["other"]
if otherExists {
dataObject["translation"] = translationData["other"]
switch t := translationData.(type) {
case map[string]interface{}:
if len(t) == 1 { // non-plural form
_, otherExists := t["other"]
if otherExists {
dataObject["translation"] = t["other"]
}
} else { // plural form
dataObject["translation"] = t
}
} else { // plural form
dataObject["translation"] = translationData
case map[interface{}]interface{}: // YAML unmarshals in this format, since keys might not be strings
if len(t) == 1 { // non-plural form
_, otherExists := t["other"]
if otherExists {
dataObject["translation"] = t["other"]
}
} else { // plural form
dataObject["translation"] = t
}
case string:
dataObject["translation"] = t
default:
panic(fmt.Sprintf("HMM? %v", reflect.TypeOf(translationData)))
}

standardFormatData = append(standardFormatData, dataObject)
20 changes: 18 additions & 2 deletions i18n/bundle/bundle_test.go
View file
Original file line number Diff line number Diff line change
@@ -9,8 +9,8 @@ import (
"reflect"
"sort"

"github.com/Liam-Williams/go-i18n/i18n/language"
"github.com/Liam-Williams/go-i18n/i18n/translation"
"github.com/EverlongProject/go-i18n/i18n/language"
"github.com/EverlongProject/go-i18n/i18n/translation"
)

func TestMustLoadTranslationFile(t *testing.T) {
@@ -162,6 +162,22 @@ func TestTfuncAndLanguage(t *testing.T) {
}
}

func TestFallbackTemplateEncoding(t *testing.T) {
b := New()
translationID := "id with param: {{.param}}"
frenchLanguage := languageWithTag("fr-FR")
addFakeTranslation(t, b, frenchLanguage, "unused")
frenchT, _, err := b.TfuncAndLanguage("fr-FR")
if err != nil {
t.Fatal(err)
}
expected := "id with param: a & b"
got := frenchT(translationID, map[string]interface{}{"param": "a & b"}) // we expect the '&' to be preserved
if got != expected {
t.Errorf("translation was %s; expected %s", got, expected)
}
}

func TestConcurrent(t *testing.T) {
b := New()
// bootstrap bundle
2 changes: 1 addition & 1 deletion i18n/example_test.go
View file
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@ package i18n_test
import (
"fmt"

"github.com/Liam-Williams/go-i18n/i18n"
"github.com/EverlongProject/go-i18n/i18n"
)

func Example() {
2 changes: 1 addition & 1 deletion i18n/exampletemplate_test.go
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
package i18n_test

import (
"github.com/Liam-Williams/go-i18n/i18n"
"github.com/EverlongProject/go-i18n/i18n"
"os"
"text/template"
)
2 changes: 1 addition & 1 deletion i18n/exampleyaml_test.go
View file
Original file line number Diff line number Diff line change
@@ -2,7 +2,7 @@ package i18n_test

import (
"fmt"
"github.com/Liam-Williams/go-i18n/i18n"
"github.com/EverlongProject/go-i18n/i18n"
)

func ExampleYAML() {
6 changes: 3 additions & 3 deletions i18n/i18n.go
View file
Original file line number Diff line number Diff line change
@@ -56,9 +56,9 @@
package i18n

import (
"github.com/Liam-Williams/go-i18n/i18n/bundle"
"github.com/Liam-Williams/go-i18n/i18n/language"
"github.com/Liam-Williams/go-i18n/i18n/translation"
"github.com/EverlongProject/go-i18n/i18n/bundle"
"github.com/EverlongProject/go-i18n/i18n/language"
"github.com/EverlongProject/go-i18n/i18n/translation"
)

// TranslateFunc returns the translation of the string identified by translationID.
2 changes: 1 addition & 1 deletion i18n/language/pluralspec_test.go
View file
Original file line number Diff line number Diff line change
@@ -70,7 +70,7 @@ func TestGetPluralSpec(t *testing.T) {
for _, test := range tests {
spec := getPluralSpec(test.src)
if spec != test.spec {
t.Errorf("getPluralSpec(%q) = %q expected %q", test.src, spec, test.spec)
t.Errorf("getPluralSpec(%v) = %v expected %v", test.src, spec, test.spec)
}
}
}
2 changes: 1 addition & 1 deletion i18n/translation/plural_translation.go
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
package translation

import (
"github.com/Liam-Williams/go-i18n/i18n/language"
"github.com/EverlongProject/go-i18n/i18n/language"
)

type pluralTranslation struct {
2 changes: 1 addition & 1 deletion i18n/translation/plural_translation_test.go
View file
Original file line number Diff line number Diff line change
@@ -4,7 +4,7 @@ import (
"reflect"
"testing"

"github.com/Liam-Williams/go-i18n/i18n/language"
"github.com/EverlongProject/go-i18n/i18n/language"
)

func mustTemplate(t *testing.T, src string) *template {
2 changes: 1 addition & 1 deletion i18n/translation/single_translation.go
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
package translation

import (
"github.com/Liam-Williams/go-i18n/i18n/language"
"github.com/EverlongProject/go-i18n/i18n/language"
)

type singleTranslation struct {
2 changes: 1 addition & 1 deletion i18n/translation/translation.go
View file
Original file line number Diff line number Diff line change
@@ -4,7 +4,7 @@ package translation
import (
"fmt"

"github.com/Liam-Williams/go-i18n/i18n/language"
"github.com/EverlongProject/go-i18n/i18n/language"
)

// Translation is the interface that represents a translated string.
4 changes: 2 additions & 2 deletions i18n/translations_test.go
View file
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@ package i18n
import (
"testing"

"github.com/Liam-Williams/go-i18n/i18n/bundle"
"github.com/EverlongProject/go-i18n/i18n/bundle"
)

var bobMap = map[string]interface{}{"Person": "Bob"}
@@ -63,7 +63,7 @@ func testFile(t *testing.T, path string) {

got := T(tc.id, args...)
if got != tc.want {
t.Error("got: %v; want: %v", got, tc.want)
t.Errorf("got: %v; want: %v", got, tc.want)
}
}
}