sequel_pg is incompatible to pg-1.2.0

[larskanis]

This was originally raised at ruby-pg: ged/ruby-pg#318

sequel_pg retrieves the connection encoding from the result T_DATA object here.
However the storage of the encoding information has changed in ruby-pg-1.2.0 due to ged/ruby-pg#280 . As a result sequel_pg marks all strings as ASCII-8BIT.

I can imagine the following ways to solve this:

1. Redefine struct t_pg_result in sequel_pg to get access to enc_idx
2. Add an export function to pg-1.2.1 kind of pg_get_connection_enc_idx()
3. Use PG::Connection.internal_encoding to retrieve the encoding
4. Set string encoding fixed to UTF-8
5. Make use of pg's builtin streaming and typecast work making sequel_pg obsolete

[jeremyevans]

Thanks for the report. I think option 3 sounds easiest in the short term, and should be less brittle. Option 5 may be possible in the long term (maybe in Sequel 6), but probably not until then. I'll try to work on this soon, and hopefully can release a fixed version in a few days.

[jeremyevans]

Unfortunately, option 3 looks not possible. The places where enc_get_index is called do not have access to the PG::Connection, only PG::Result, and there doesn't appear to be a way to get the PG::Connection using the PG::Result without using option 1.

Option 1 is very brittle. It would be very difficult to support multiple versions of ruby-pg. I suppose if that is the only option, I could update the gemspec to require a specific version of ruby-pg. I really don't like limiting user's flexibility like that, though.

One thing that could reduce the brittleness of option 1 is to use a partial struct definition, with only the first two members (pgresult and connection), then retrieve the encoding index from the connection.

Option 4 is not possible, as we cannot assume the database is in UTF-8. I access data in non-UTF-8 databases on a regular basis, for example.

Option 2 would not work for the same reason that option 3 does not work.

Additional options that I think would make this easier:

6. Give PG::Result access to the PG::Connection that created it via a Ruby method.
7. Add PG::Result#internal_encoding
8. Add pg_get_result_enc_idx()

I think option 8 would result in the highest performance, though I doubt this is a performance-sensitive function. Are you OK with any of options 6-8?

[gencer]

Hey @jeremyevans, Have you consider any option (or 8th option as you suggested) until now? I've downgraded pg gem to 1.1.4 until this fixed.

[jeremyevans]

All options 6-8 require changes to pg. I'm waiting on feedback from @larskanis before moving forward. If none of options 6-8 are acceptable, I'll probably use option 1, maybe in connection with option 3.

[larskanis]

I'm OK with option 8. I'm just too busy to implement it right now.

Regarding option 4: Do you really access with non UTF8 client encoding (in contrast to server/database encoding)? My experience is that non UTF8 client encodings fail in so many places (validations, external libraries, etc.) that it's pretty hard to use. IMHO everyone uses PostgreSQL's on the fly encoding conversion to UTF8 to access non UTF8 databases.

[jeremyevans]

@larskanis I'll try to implement option 8 and submit it as a pull request.

For option 4, yes, I routinely deal with non-UTF8 client encodings (as well as non-UTF8 database encodings).

[ged]

I've merged the fix, and will be pushing pg 1.2.1 in a few minutes.

[ged]

Source gem is pushed, Windows binary gems are building now...

[jeremyevans]

Thanks! I'm doing some final testing for sequel_pg and will release a fixed gem within an hour hopefully.

[jeremyevans]

OK, sequel_pg 1.12.4 released, which should work with pg 1.2.1. If you have any issues with it, please let me know.

[kenaniah]

@jeremyevans could you please also add a minimum version dependency on pg >= 1.2.1 to sequel_pg?

When version locked against pg 1.1.4 and one tries to update sequel_pg to latest, they may run into something like the following (due to the version lock):

# => require 'sequel_pg'
Traceback (most recent call last):
        1: from (irb):3
LoadError (Error relocating /srv/vendor/bundle/ruby/2.7.0/gems/sequel_pg-1.14.0/lib/sequel_pg.so: pg_get_result_enc_idx: symbol not found - /srv/vendor/bundle/ruby/2.7.0/gems/sequel_pg-1.14.0/lib/sequel_pg.so)

Removing the version lock of course avoids this issue, but it broke one of my builds and it took me a minute to figure out why. Adjusting the minimum supported version should help avoid the above symbol mismatch altogether.

[jeremyevans]

@kenaniah sequel_pg should be compatible with pg 0.18.0+ (other than pg 1.2.0), as specified in the gemspec. I just ran a test with sequel_pg 1.14.0 and pg 1.1.4 and it ran fine.

I think what you are running into is the last bullet in https://github.com/jeremyevans/sequel_pg#label-Known+Issues, where you are erroring on undefined functions even if they are never called. sequel_pg only calls that function on pg 1.2+. Maybe try the fix mentioned in that bullet?

[kenaniah]

@jeremyevans Even with that known issue workaround in place, it's still an issue for alpine (or more generally MUSL) builds specifically (otherwise works on debian, OSX, etc.). If interested, I'd be happy to publish a docker container demonstrating the specific failure against the following Gemfile:

gem 'pg', "1.1.4"
gem 'sequel',
gem 'sequel_pg', require: 'sequel'

[jeremyevans]

@kenaniah Not interested in a docker container. If you have any ideas how to get it to work with musl/alpine, I'm willing to consider them. However, I'm not adding an unnecessary version restriction to work around musl/alpine issues.

[kenaniah]

@jeremyevans no worries then. Since the issue is specific to a version lock on pg 1.1.4 on musl and is fixed by removing that version lock (thus allowing pg to update to 1.2.1 or later), it's possible that this conversation alone may be enough to help the next person who finds themself in a similar situation. Thanks!