After upgrade to Rails 7.1, translations that end in a colon return a Hash, not a String #147

floydj · 2024-01-22T15:46:22Z

I'm trying to upgrade a Rails 7 app to 7.1, but running into issues with I18n translations.

Given the following translation:

[12] pry(main)> Translation.last
=> #<Translation:0x00000001183f0b60
 id: 118058,
 locale: "en",
 key: "project_customer_name",
 value: "Customer's Company Name:",
 interpolations: nil,
 is_proc: false,
 created_at: Mon, 22 Jan 2024 14:37:05.000000000 UTC +00:00,
 updated_at: Mon, 22 Jan 2024 15:00:36.000000000 UTC +00:00>

And trying to use that in my view:

<p><%= t(:project_customer_name) %></p>

I'll always get a hash in the resulting HTML, rather than the String I expect:

{"Customer's Company Name"=>nil}

Translation occurs properly in any of the following conditions:

Adding the translation to the en.yml file AND removing it from the translations table
Downgrading back to Rails 7.0
Removing i18n.-active_record (not a long-term option for this project, but tried it for testing)
Removing the colon from the end of the translated value. e.g. "Customer's Company Name"
Moving the colon anywhere else in the value e.g. "Customer's Company: Name"

About the app:

Rails 7.1.3
Ruby 3.2.2

I have tried creating a brand-new Rails 7.1 app with i18n-active_record and...the translation works fine, even with the colon at the end of a value. So it seems there is something in my app causing this, but I'm stuck, unsure of how or what to debug further.

Any idea what's going on or how I can troubleshoot further? Appreciate the help!

The text was updated successfully, but these errors were encountered:

timfjord · 2024-01-22T18:37:26Z

Since a brand new app with the gem works fine, then it looks like there is some extra processing happening on top of the original method in your application.
So I wonder what happens if you use <p><%= I18n.t(:project_customer_name) %></p>, to see whether it is something in the i18n gem of rails itself.

Also, what happens if you call t(:project_customer_name) directly in the controller? You could just use

def my_action
  raise t(:project_customer_name)
end

and check the console

floydj · 2024-01-22T19:18:13Z

@timfjord - Using I18n.t(:project_customer_name) gives the same result: showing the hash in the HTML.

I'm not getting anything using that example method in a controller:

TypeError - exception class/object expected:

I've found as I've kept digging on this that it appears any translated value that contains an asterisk (*) will bomb out too, which is an issue for me, since I'm translating a lot of Markdown. Here's an example inside of app/views/translations/_translation:

<td><%= t(translation.key) %></td>

If the translation value contains an asterisk, I'll get:

(<unknown>): did not find expected alphabetic or numeric character while scanning an alias at line 4 column 1

timfjord · 2024-01-22T19:23:17Z

And where is this error coming from? Is there a stack trace, as it might hint what tool processes the translation

floydj · 2024-01-22T19:31:04Z

I'm not getting much of a stack trace, actually. It just takes me to the line in app/views/translations/index where the _translation partial is rendered.

Psych::SyntaxError - (<unknown>): did not find expected alphabetic or numeric character while scanning an alias at line 4 column 1:
  app/views/translations/_translation.html.slim:3
  app/views/translations/index.html.slim:36

Maybe Psych is a clue here? You'll notice I'm using slim, but I've tried in ERB with the same results.

timfjord · 2024-01-23T09:50:43Z

From quick googling this did not find expected alphabetic or numeric ... error is a YAML error.
So maybe try upgrading psych.

Alternatively, you could use your plain Rails 7.1 app with the i18n-active_record and try adding gems from your app's Gemfile one by one to find the one that causing the issue.

Since you have this issue when you use raise t(:project_customer_name).inspect too, maybe there is some gem that has overridden the I18n.translate method. You could debug this but running this I18n.method(:translate).source_location either in the console, or, even better in the controller

def my_action
  raise I18n.method(:translate).source_location.inspect
end

floydj · 2024-01-23T14:22:21Z

Ok, good advice. I'll try adding gems to the new app, but at this point, I think it has more gems than my old one.

Will report back - Thank you for your help.

floydj · 2024-01-23T14:52:36Z

Well, I have stripped my app down to the bone.

My Gemfile:

source "https://rubygems.org"
git_source(:github) { |repo| "https://github.com/#{repo}.git" }
ruby "3.2.2"
gem "i18n-active_record", require: "i18n/active_record"
gem "jsbundling-rails"
gem "mysql2"
gem "puma" # , '~> 3.7'
gem "rails", "7.1.3"

I've also gone through and commented out every single initializer.

And blanked out the ApplicationController just to ensure I wasn't massaging the text in some way I forgot.

I also reinstalled Ruby 3.2.2, removed and regenerated the Gemfile.lock.

And...it is still showing that hash instead of text when it's pulling the translation out of ActiveRecord.

Any ideas on what I could try next?

floydj · 2024-01-23T15:08:52Z

Actually, I pasted/copied Gemfile from the working new app to the old one, ran rails app:update and let it rewrote all configuration files, and it still isn't translating properly.

floydj · 2024-01-23T15:13:58Z

Adding raise I18n.method(:translate).source_location.inspect to the controller gives this output:

RuntimeError (["/Users/jasonfloyd/.gem/ruby/3.2.1/gems/i18n-1.14.1/lib/i18n.rb", 210]):

app/controllers/checker_controller.rb:4:in `index'

Reverting back to Rails 7.0, I get the same RuntimeError, referencing the same line.

floydj · 2024-01-23T21:18:38Z

Another possible clue. It seems Rails 7.1 might be storing translation values a little differently? Notice how the value field is displayed differently:

±  bin/rails console
Loading development environment (Rails 7.0.8)
[1] pry(main)> Translation.find(117998)
=> #<Translation:0x000000010ecd3700
 id: 117998,
 locale: "en",
 key: "europe_quote_requirements_html",
 value: "<span class=\"font-weight-bold\">Please check the following requirements to view the quote sheet:</span>\n\n* Fill in both the *Payment Terms* and *Delivery (in # of weeks)* fields\n* Select either *Ex-Works* or *Delivered To* selected\n* Fill in the *Shipping Location* if the *Delivered To* option is checked",
 interpolations: nil,
 is_proc: false,
 created_at: Tue, 05 May 2020 20:10:32.000000000 UTC +00:00,
 updated_at: Tue, 05 May 2020 20:15:24.000000000 UTC +00:00>
[2] pry(main)> exit

±  bin/rails console
Loading development environment (Rails 7.1.3)
[1] pry(main)> Translation.find(117998)
=> #<I18n::Backend::ActiveRecord::Translation:0x0000000110dbe848 id: 117998, locale: "en", key: "europe_quote_requirements_html", value: #<I18n::Backend::ActiveRecord::Translation:0x90d8>

timfjord · 2024-01-24T13:47:36Z

Adding raise I18n.method(:translate).source_location.inspect to the controller gives this output:
RuntimeError (["/Users/jasonfloyd/.gem/ruby/3.2.1/gems/i18n-1.14.1/lib/i18n.rb", 210]):

app/controllers/checker_controller.rb:4:in `index'
Reverting back to Rails 7.0, I get the same RuntimeError, referencing the same line.

So, the method hasn't been overridden. That's good.

Well, I have stripped my app down to the bone.

My Gemfile:

source "https://rubygems.org"
git_source(:github) { |repo| "https://github.com/#{repo}.git" }
ruby "3.2.2"
gem "i18n-active_record", require: "i18n/active_record"
gem "jsbundling-rails"
gem "mysql2"
gem "puma" # , '~> 3.7'
gem "rails", "7.1.3"

Ok, and what happens if you use this Gemfile with the plain app you mentioned earlier(the one that worked properly)?

floydj · 2024-01-24T15:08:39Z

If I use that gemfile with the plain app, it still translates everything properly.

timfjord · 2024-01-25T12:06:16Z

Well, then it is clear that it is not gems that causing this issue.

And how do you initialize the i18n-active_record gem? I assume you use the fallback module, maybe try to disable it in the real app.

Regarding this one

Another possible clue. It seems Rails 7.1 might be storing translation values a little differently? Notice how the value field is displayed differently:

±  bin/rails console
Loading development environment (Rails 7.0.8)
[1] pry(main)> Translation.find(117998)
=> #<Translation:0x000000010ecd3700
 id: 117998,
 locale: "en",
 key: "europe_quote_requirements_html",
 value: "<span class=\"font-weight-bold\">Please check the following requirements to view the quote sheet:</span>\n\n* Fill in both the *Payment Terms* and *Delivery (in # of weeks)* fields\n* Select either *Ex-Works* or *Delivered To* selected\n* Fill in the *Shipping Location* if the *Delivered To* option is checked",
 interpolations: nil,
 is_proc: false,
 created_at: Tue, 05 May 2020 20:10:32.000000000 UTC +00:00,
 updated_at: Tue, 05 May 2020 20:15:24.000000000 UTC +00:00>
[2] pry(main)> exit

±  bin/rails console
Loading development environment (Rails 7.1.3)
[1] pry(main)> Translation.find(117998)
=> #<I18n::Backend::ActiveRecord::Translation:0x0000000110dbe848 id: 117998, locale: "en", key: "europe_quote_requirements_html", value: #<I18n::Backend::ActiveRecord::Translation:0x90d8>

This output is very unusual, because the value of the Translation instance is a Translation instance, which is definitely not how it is supposed to be.
So what if you set the value directly, in the console

Translation.find(117998).update(value: "Customer's Company Name:")

floydj · 2024-01-29T14:08:52Z

If I save directly in the console per your example, it seems to work fine.

But for many records, the value field changes from a simple string to an instance of I18n::Backend::ActiveRecord::Translation just from switching from Rails 7 to Rails 7.1. Not sure why? I have thousands of records in this project, and I can't even read what the translated value is for many of them, because when I try to access it, I get an error: Psych::SyntaxError - (<unknown>): did not find expected alphabetic or numeric character while scanning an alias

Regarding the initializing of i18n-active_record, I have the standard initializer from this project's readme, plus the following in my config/application.rb file:

 config.i18n.default_locale = :en
 config.i18n.fallbacks = [:en]

floydj · 2024-01-29T14:52:39Z

Ok, I think we're getting closer. I have a app/models/translation.rb file present with various "helper" methods like listing out locales, copying the English values to other languages as starting points, etc. I'm wondering if that's now a problem with upgrading to Rails 7.1?

In Rails 7, if I run the following in my browser-based debugger, you can see this class recognized:

>> Translation
=> Translation(id: integer, locale: string, key: string, value: text, interpolations: text, is_proc: boolean, created_at: datetime, updated_at: datetime)

But if I do the same thing in 7.1, I wind up with something different:

>> Translation
=> I18n::Backend::ActiveRecord::Translation(id: integer, locale: string, key: string, value: text, interpolations: text, is_proc: boolean, created_at: datetime, updated_at: datetime)

This has not been an issue so far in my years of using this gem, but I'm wonder if it's now required for me to setup a custom translation model as described in the readme?

If so, where do I put the following block (from the readme)?

I18n::Backend::ActiveRecord.configure do |config|
  config.translation_model = MyTranslation
end

I tried in an initializer, the config/application.rb, and config/environments/development.rb, but rails wouldn't load that way. Does it go after the end of the custom class? I put it there, without any errors, but unsure if it's working.

In any case, using a custom model still gives a Psych::Syntax error as described in a previous comment. I get the following results in the console. Notice the value for id 111141.

Loading development environment (Rails 7.1.3)
[1] pry(main)> Translation.where(key: :project_customer_name)
  I18n::Backend::ActiveRecord::Translation Load (2.7ms)  SELECT `translations`.* FROM `translations` WHERE `translations`.`key` = 'project_customer_name' /* loading for pp */ LIMIT 11
=> [#<I18n::Backend::ActiveRecord::Translation:0x000000010e688d28
  id: 49,
  locale: "en",
  key: "project_customer_name",
  value: "Customer's Company Name:",
  interpolations: [],
  is_proc: false,
  created_at: Wed, 21 May 2014 18:07:36.000000000 UTC +00:00,
  updated_at: Thu, 25 Jan 2024 15:47:20.000000000 UTC +00:00>,
 #<I18n::Backend::ActiveRecord::Translation:0x000000010cc64398
  id: 111140,
  locale: "de",
  key: "project_customer_name",
  value: {"Firmenname des Kundes"=>nil},
  interpolations: [],
  is_proc: false,
  created_at: Wed, 29 Jul 2015 12:21:56.000000000 UTC +00:00,
  updated_at: Mon, 10 Aug 2015 06:41:38.000000000 UTC +00:00>,
 #<I18n::Backend::ActiveRecord::Translation:0x000000010cc64258
  id: 111141,
  locale: "cn",
  key: "project_customer_name",
  value: #<I18n::Backend::ActiveRecord::Translation:0x90d8>]

Here's the same results from Rails 7.0.8:

[2] pry(main)> Translation.where(key: :project_customer_name)
=> [#<Translation:0x000000011919bd78
  id: 49,
  locale: "en",
  key: "project_customer_name",
  value: "--- 'Customer''s Company Name:'\n",
  interpolations: nil,
  is_proc: false,
  created_at: Wed, 21 May 2014 18:07:36.000000000 UTC +00:00,
  updated_at: Thu, 25 Jan 2024 15:47:20.000000000 UTC +00:00>,
 #<Translation:0x000000011919bcd8
  id: 111140,
  locale: "de",
  key: "project_customer_name",
  value: "Firmenname des Kundes:",
  interpolations: nil,
  is_proc: false,
  created_at: Wed, 29 Jul 2015 12:21:56.000000000 UTC +00:00,
  updated_at: Mon, 10 Aug 2015 06:41:38.000000000 UTC +00:00>,
 #<Translation:0x000000011919bc38
  id: 111141,
  locale: "cn",
  key: "project_customer_name",
  value: "Customer's Company Name:\r\n客户的公司名称",
  interpolations: nil,
  is_proc: false,
  created_at: Wed, 29 Jul 2015 12:21:56.000000000 UTC +00:00,
  updated_at: Mon, 05 Sep 2016 05:19:29.000000000 UTC +00:00>]

timfjord · 2024-01-30T17:44:40Z

First of all, It looks like there are issues with auto-loading models from initialisers, but this is not the case here.

I would do the following.
First, comment out the custom model altogether and see if the value of the value of 111140 and 111141 changes.
Then, I would keep the content(all additional methods, etc) of the custom model commented out but try to make that translation_model thing work.
I think this can be achieved by adding the init code in the to_prepare callback in the config/application.rb

module MyApp
  class Application < Rails::Application
    # ...

    config.to_prepare do
      I18n::Backend::ActiveRecord.configure do |config|
        config.translation_model = Translation
      end
    end
  end
end

The idea here is to make the value of the value field of those two items(111140 and 111141) look "normal" (similar to other items in the table) and figure out at which state it is getting broken.

floydj · 2024-01-30T19:37:22Z

Thanks, @timfjord - I'll give that a shot and see what I come up with. In the meantime, here's some more of what I've found.

After doing a mysqlimport from my production database translations table:

mysql> select * from translations where id = 117502;
+--------+--------+-----------------------+------------+----------------+---------+---------------------+---------------------+
| id     | locale | key                   | value      | interpolations | is_proc | created_at          | updated_at          |
+--------+--------+-----------------------+------------+----------------+---------+---------------------+---------------------+
| 117502 | en     | percent_neat_oil_html | % Neat Oil | NULL           |       0 | 2019-01-14 18:42:09 | 2019-01-14 18:42:09 |
+--------+--------+-----------------------+------------+----------------+---------+---------------------+---------------------+

This cannot be updated via Rails console, I get the Psych::SyntaxError every time. So...

mysql> update translations set value = '<span>% Neat Oil</span>' where id = 117502;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> select * from translations where id = 117502;
+--------+--------+-----------------------+-------------------------+----------------+---------+---------------------+---------------------+
| id     | locale | key                   | value                   | interpolations | is_proc | created_at          | updated_at          |
+--------+--------+-----------------------+-------------------------+----------------+---------+---------------------+---------------------+
| 117502 | en     | percent_neat_oil_html | <span>% Neat Oil</span> | NULL           |       0 | 2019-01-14 18:42:09 | 2019-01-14 18:42:09 |
+--------+--------+-----------------------+-------------------------+----------------+---------+---------------------+---------------------+

Now, because the % is no longer the first character, I can edit the translation record via my Rails UI, and remove the <span> tag, which works properly without any Psych error. Which gives me:

mysql> select * from translations where id = 117502;
+--------+--------+-----------------------+-------------------+----------------+---------+---------------------+---------------------+
| id     | locale | key                   | value             | interpolations | is_proc | created_at          | updated_at          |
+--------+--------+-----------------------+-------------------+----------------+---------+---------------------+---------------------+
| 117502 | en     | percent_neat_oil_html | --- "% Neat Oil"
 | NULL           |       0 | 2019-01-14 18:42:09 | 2024-01-30 19:28:58 |
+--------+--------+-----------------------+-------------------+----------------+---------+---------------------+---------------------+

You can see the way the value is presented has changed. Not sure why - using same version of MySQL in dev and production. But from this point forward, everything with this record will work as expected before the upgrade to Rails 7.1.

floydj · 2024-01-30T20:38:37Z

@timfjord - I have a Rails 7.1 test app that throws the errors I'm seeing at https://github.com/floydj/translated.

You can see in that project that when seeding the database, or running integration tests, I'm using direct mysql statements to create the translations and then trying to use them. If I use ActiveRecord to create the translations, it works as it should. But the app is more setup to replicate what I'm dealing with, which is existing translations from a Rails 7 app.

timfjord · 2024-01-31T09:49:15Z

Thanks for sharing the app, it was very useful.

So the good thing is that I've figured out what the problem was and sent a PR to your test app with a fix - floydj/translated#1

Basically, the problem here was that you were trying to insert values as they are, but, apparently, they weren't compatible with Rails 7.1
Here is how the value is defined in the Translation model - https://github.com/svenfuchs/i18n-active_record/blob/master/lib/i18n/backend/active_record/translation.rb#L56-L62
As you can see, starting from Rails 7.1 an explicit coder is required.
And since the value you manually entered wasn't compatible with the YAML coder, it couldn't parse it - thus all those Psych errors.

In terms of how to deal with this. If you really need to insert the values directly, you should probably just adapt them so they are compatible with the YAML coder.

I am closing the issue for now, as I don't think there is something we could do with gem itself to prevent this.
I might add a note to README later on.

floydj · 2024-01-31T14:17:29Z

Awesome. Glad you found the point where it gave trouble with Rails 7.1.

But it doesn't solve my issue. That example app I shared is not how I normally add translations, it was just to illustrate the error I saw. I normally add the translations using standard Active Record methods, not direct SQL statements.

I still have a table full of translations that throw Psych errors, since they were inserted using previous versions of Rails. Is there no way to "upgrade" them without going through all of them manually?

timfjord · 2024-01-31T14:23:41Z

You could write a small script that processes all items and wraps them in --- \"my_value\"\n (which is the result of calling YAML.dump("my_value"))
You could even do that in the Rails 7.1 app, you just need to avoid using AR, as it will always complain regarding the value field

timfjord · 2024-01-31T14:32:20Z

Another option to discover could be changing the default coder.
Since you use the custom model, you could try adding something like this in the Translation model, to override the defaults

serialize :value, coder: ActiveRecord::Coders::YAMLColumn

If that doesn't work, you could try to override this directly in the gem just to see if it is working or nor.
bundle open i18n-active_record, go to lib/i18n/backend/active_record/translation.rb and change the line 57 accordingly. It will work only locally, but at least you could test it.

floydj · 2024-01-31T14:46:01Z

Ok, thanks for possible solutions. My vote would be for some kind of note in the README, I can't be the only one that'll run into this use case.

floydj · 2024-02-01T17:25:14Z

In case it helps someone else, here are a few bits that helped me convert things over to be compatible with Rails 7.1. Using MySQL here.

# find Psych problems
Translation.all.each do |trans|
  trans.value
rescue Psych::SyntaxError
  puts "Serialization problem with #{trans.id} - #{trans.key}"
end

# convert everything over to newer serialization format
Translation.all.each do |translation|
  sql = %{update translations set value = CONCAT('--- "', value, '"\n') where id = #{translation.id}}
  ActiveRecord::Base.connection.execute(sql)
end

After doing that, there will still be some odds and ends to look at. I had some HTML translations with links that I had to manually fix in the MySQL client:

update translations set value = replace(value, '\"%{link}\"', '\'%{link}\'') where id = 112030;

timfjord · 2024-02-01T17:59:20Z

You could have also used YAML.dump.
Something like that (didn't tested)

Translation.all.each do |translation|
  sql = %{update translations set value = '#{YAML.dump(translation.value_before_type_cast)}' where id = #{translation.id}}
  ActiveRecord::Base.connection.execute(sql)
end

But in general, yeah, this is the right way to go.

I will mention this issue in the README

timfjord closed this as completed Jan 31, 2024

matko9 mentioned this issue Dec 11, 2024

Change using of class object for custom translation model #149

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

After upgrade to Rails 7.1, translations that end in a colon return a Hash, not a String #147

After upgrade to Rails 7.1, translations that end in a colon return a Hash, not a String #147

floydj commented Jan 22, 2024

timfjord commented Jan 22, 2024 •

edited

Loading

floydj commented Jan 22, 2024 •

edited

Loading

timfjord commented Jan 22, 2024

floydj commented Jan 22, 2024

timfjord commented Jan 23, 2024 •

edited

Loading

floydj commented Jan 23, 2024

floydj commented Jan 23, 2024

floydj commented Jan 23, 2024

floydj commented Jan 23, 2024 •

edited

Loading

floydj commented Jan 23, 2024

timfjord commented Jan 24, 2024

floydj commented Jan 24, 2024

timfjord commented Jan 25, 2024 •

edited

Loading

floydj commented Jan 29, 2024

floydj commented Jan 29, 2024 •

edited

Loading

timfjord commented Jan 30, 2024 •

edited

Loading

floydj commented Jan 30, 2024

floydj commented Jan 30, 2024

timfjord commented Jan 31, 2024

floydj commented Jan 31, 2024 •

edited

Loading

timfjord commented Jan 31, 2024 •

edited

Loading

timfjord commented Jan 31, 2024

floydj commented Jan 31, 2024

floydj commented Feb 1, 2024

timfjord commented Feb 1, 2024

After upgrade to Rails 7.1, translations that end in a colon return a Hash, not a String #147

After upgrade to Rails 7.1, translations that end in a colon return a Hash, not a String #147

Comments

floydj commented Jan 22, 2024

timfjord commented Jan 22, 2024 • edited Loading

floydj commented Jan 22, 2024 • edited Loading

timfjord commented Jan 22, 2024

floydj commented Jan 22, 2024

timfjord commented Jan 23, 2024 • edited Loading

floydj commented Jan 23, 2024

floydj commented Jan 23, 2024

floydj commented Jan 23, 2024

floydj commented Jan 23, 2024 • edited Loading

floydj commented Jan 23, 2024

timfjord commented Jan 24, 2024

floydj commented Jan 24, 2024

timfjord commented Jan 25, 2024 • edited Loading

floydj commented Jan 29, 2024

floydj commented Jan 29, 2024 • edited Loading

timfjord commented Jan 30, 2024 • edited Loading

floydj commented Jan 30, 2024

floydj commented Jan 30, 2024

timfjord commented Jan 31, 2024

floydj commented Jan 31, 2024 • edited Loading

timfjord commented Jan 31, 2024 • edited Loading

timfjord commented Jan 31, 2024

floydj commented Jan 31, 2024

floydj commented Feb 1, 2024

timfjord commented Feb 1, 2024

timfjord commented Jan 22, 2024 •

edited

Loading

floydj commented Jan 22, 2024 •

edited

Loading

timfjord commented Jan 23, 2024 •

edited

Loading

floydj commented Jan 23, 2024 •

edited

Loading

timfjord commented Jan 25, 2024 •

edited

Loading

floydj commented Jan 29, 2024 •

edited

Loading

timfjord commented Jan 30, 2024 •

edited

Loading

floydj commented Jan 31, 2024 •

edited

Loading

timfjord commented Jan 31, 2024 •

edited

Loading