BSim: Update PostgreSQL (15.3->17.0) to fix SSL errors #7085

gemesa · 2024-10-23T19:16:39Z

As mentioned in the linked issues, on distros (Fedora, Arch etc.) with newer packages (especially OpenSSL), the currently used PostgreSQL (15.3) is not working properly resulting in SSL error(s) (detailed in the issues). In #7084 I fixed the SSL error by downgrading OpenSSL but this was a painful solution. In this PR I have upgraded the PostgreSQL version from 15.3 to 17.0 and now bsim_ctl start works on Fedora 40 with OpenSSL 3.2.2 and also on Ubuntu 22.04 with OpenSSL 3.0.2. Edit: I also added the dumpall and restore commands to bsim_ctl which can be used to migrate DBs from 15.3 to 17.0 (see #7085 (comment)).

$ uname -a
Linux fedora 6.10.12-200.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Sep 30 21:38:25 UTC 2024 x86_64 GNU/Linux
$ openssl version
OpenSSL 3.2.2 4 Jun 2024 (Library: OpenSSL 3.2.2 4 Jun 2024)
$ ./support/bsim_ctl start ~/git-repos/bsim-db       
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
No client authentication
Initializing data directory
Generating servers SSL certificate
Server started
BSim extension enabled

$ uname -a
Linux ubuntu-22 6.8.0-47-generic #47~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Oct  2 16:16:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
$ openssl version
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
$ ./support/bsim_ctl start ~/git-repos/bsim-db/
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
No client authentication
Initializing data directory
Generating servers SSL certificate
Server started
BSim extension enabled

gemesa · 2024-10-24T18:30:06Z

I also compared pga_hba.conf from version 15.3 to 17.0 and found no differences, except in the comments.

Similarly I compared postgresql.conf from version 15.3 to 17.0. Some options were added, removed or had their default values changed but the customized options (listen_addresses, password_encryption, ssl, shared_buffers, work_mem, checkpoint_timeout) are not affected.

I have attached the 2 compare reports created by Beyond Compare. Remove the .txt extension and then they can be viewed in a browser.
postgresql.conf.diff.html.txt
pg_hba.conf.diff.html.txt

byt3exec

*looking at the

gemesa · 2024-10-25T19:25:50Z

I tested most of the main functionalities and everything seems to be working fine:

$ ./support/bsim_ctl start ~/git-repos/bsim-db
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
No client authentication
Initializing data directory
Generating servers SSL certificate
Server started
BSim extension enabled

$ ./support/bsim_ctl adduser ~/git-repos/bsim-db test                   
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Added user: test

$ ./support/bsim_ctl adduser ~/git-repos/bsim-db test2 --user test
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Error connecting to the database
ERROR: permission denied to create role
  Detail: Only roles with the CREATEROLE attribute may create roles.

$ ./support/bsim_ctl changeprivilege test admin --user test      
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Granting admin privileges to test
Error connecting to the database
ERROR: permission denied to alter role
  Detail: Only roles with the SUPERUSER attribute may change the SUPERUSER attribute.

$ ./support/bsim_ctl changeprivilege test admin            
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Granting admin privileges to test

$ ./support/bsim_ctl dropuser ~/git-repos/bsim-db test --user test 
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Error connecting to the database
ERROR: current user cannot be dropped

$ ./support/bsim_ctl dropuser ~/git-repos/bsim-db test            
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Removed user: test

$ ./support/bsim createdatabase postgresql://localhost:5432/repo medium_64 -n repo
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Max-Memory: 3.4 GBytes
Created database: repo
   owner       = Example Owner
   description = A medium sized (~10 million functions) database tuned for 64-bit executables
   template    = medium_64
 
$ ./support/bsim generatesigs ghidra://localhost/repo ~/git-repos/bsim-xml --bsim postgresql://localhost/repo --commit 
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Max-Memory: 3.4 GBytes
Using configuration for:
 Database: repo
 Owner:    Example Owner
 
Repository Server: localhost
Password: 
Generating signatures for: hello 
Writing signatures for sigs_585c70eef0bbd516c846dcd3103dced7

$ ./support/bsim delete postgresql://localhost/repo --name hello
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Max-Memory: 3.4 GBytes
Successfully deleted hello(10 functions)585c70eef0bbd516c846dcd3103dced7

$ ./support/bsim generatesigs ghidra://localhost/repo ~/git-repos/bsim-xml --bsim postgresql://localhost/repo --commit
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Max-Memory: 3.4 GBytes
Using configuration for:
 Database: repo
 Owner:    Example Owner
 
Repository Server: localhost
Password: 
WARN  Signature file already exists for: hello (SignatureRepository)  
Writing signatures for sigs_585c70eef0bbd516c846dcd3103dced7

$ ./support/bsim listexes postgresql://localhost/repo    
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Max-Memory: 3.4 GBytes
585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc 
1 executables found

$ ./support/bsim getexecount postgresql://localhost/repo
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Max-Memory: 3.4 GBytes
Matching executable count: 1

$ ./support/bsim listfuncs postgresql://localhost/repo --name hello
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Max-Memory: 3.4 GBytes
585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  FUN_00401020 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  __do_global_dtors_aux 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  _dl_relocate_static_pie 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  _fini 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  _init 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  _start 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  deregister_tm_clones 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  main 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  print_hello 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  register_tm_clones 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc

$ ./support/bsim dropindex postgresql://localhost/repo    
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Max-Memory: 3.4 GBytes
Successfully dropped index for database repo (repo  (postgres: localhost))

$ ./support/bsim rebuildindex postgresql://localhost/repo
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Max-Memory: 3.4 GBytes
Starting rebuild ...
Successfully rebuilt index for database repo (repo  (postgres: localhost))

$ ./support/bsim dumpsigs postgresql://localhost/repo ~/git-repos/bsim-xml --name hello 
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Max-Memory: 3.4 GBytes
$ head -n 10 sigs_585c70eef0bbd516c846dcd3103dced7 
<description layout_version="5" major="6" minor="1" settings="0x49">
<execlist>
<exe>
  <md5>585c70eef0bbd516c846dcd3103dced7</md5>
  <name>hello</name>
  <arch>x86:LE:64:default</arch>
  <compiler>gcc</compiler>
  <date millis="0x2b5">0x671be47b</date>
  <repository>ghidra://localhost/repo</repository>
</exe>

$ rm -rf ~/git-repos/bsim-db
$ mkdir ~/git-repos/bsim-db
$ ./support/bsim_ctl start ~/git-repos/bsim-db --auth pki --dn=CN=gemesa --cafile=/home/gemesa/git-repos/bsim-pki/ca.crt --cert=/home/gemesa/git-repos/bsim-pki/client.p12
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Remote client authentication with PKI certificates
Initializing data directory
Generating servers SSL certificate
Server started
BSim extension enabled

$ ./support/bsim_ctl stop ~/git-repos/bsim-db 
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Server shutdown complete

gemesa · 2024-10-27T19:12:16Z

I also added the dumpall and restore commands to bsim_ctl which can be used to migrate DBs from 15.3 to 17.0.

Dump DBs with PostgreSQL version 15.3

Prerequisites:

run ./make-postgres.sh with POSTGRES=postgresql-15.3
init a db with ./support/bsim_ctl start
add some signatures with ./support/bsim commitsigs

I added this one:

$ head -n 10 ~/git-repos/bsim-xml/sigs_585c70eef0bbd516c846dcd3103dced7 
<description layout_version="5" major="6" minor="1" settings="0x49">
<execlist>
<exe>
  <md5>585c70eef0bbd516c846dcd3103dced7</md5>
  <name>hello</name>
  <arch>x86:LE:64:default</arch>
  <compiler>gcc</compiler>
  <date millis="0x2b5">0x671be47b</date>
  <repository>ghidra://localhost/repo</repository>
</exe>

Dump the databases (I am prompted for the password 4 times because there are 4 databases so 4 connections made):

$ ./support/bsim_ctl dumpall mydump-15.3.sql
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Password: 
Password: 
Password: 
Password: 
Databases dumped to /home/gemesa/git-repos/ghidra/build/dist/ghidra_11.3_DEV/mydump-15.3.sql

Inspect the content:

$ grep version mydump-15.3.sql 
-- Dumped from database version 15.3
-- Dumped by pg_dump version 15.3
-- Dumped from database version 15.3
-- Dumped by pg_dump version 15.3
-- Dumped from database version 15.3
-- Dumped by pg_dump version 15.3

$ grep hello mydump-15.3.sql                           
9	print_hello	1	-1810480557466029468	0	4198694
1	585c70eef0bbd516c846dcd3103dced7	hello	1	1	2024-10-25 20:33:31.693+02	1	0

Restore DBs with PostgreSQL version 17.0

Prerequisites:

run ./make-postgres.sh with POSTGRES=postgresql-17.0
init a db with ./support/bsim_ctl start

Restore the databases:

$ ./support/bsim_ctl restore mydump-15.3.sql 
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Password for user gemesa: 
Databases restored from /home/gemesa/git-repos/ghidra/build/dist/ghidra_11.3_DEV/mydump-15.3.sql

Inspect the content:

$ ./support/bsim listexes postgresql://localhost/repo
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Max-Memory: 3.4 GBytes
BSim Database Server: repo  (postgres: localhost)
Password: 
585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc 
1 executables found

$ ./support/bsim listfuncs postgresql://localhost/repo -n hello
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Max-Memory: 3.4 GBytes
BSim Database Server: repo  (postgres: localhost)
Password: 
585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  FUN_00401020 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  __do_global_dtors_aux 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  _dl_relocate_static_pie 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  _fini 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  _init 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  _start 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  deregister_tm_clones 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  main 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  print_hello 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc
  register_tm_clones 585c70eef0bbd516c846dcd3103dced7 hello x86:LE:64:default gcc

ghidra1 · 2024-12-20T18:46:28Z

For the time being we will be updating to postgresql 15.10 which works with openssl 3.2.2 and was updated on 18-Nov-2024. Support for postgresql 15.x continues until 11-Nov-2027 (see Postgrsql Releases).

gemesa · 2024-12-20T19:07:14Z

This is awesome news. Let me know if you want to include any changes from this PR in the future.

ryanmkurtz assigned ghidracadabra Oct 24, 2024

ryanmkurtz added Status: Triage Information is being gathered Feature: BSim labels Oct 24, 2024

byt3exec reviewed Oct 25, 2024

View reviewed changes

gemesa force-pushed the bsim-pg-17.0 branch from 2d6ec1f to bfd51e7 Compare October 25, 2024 08:01

gemesa force-pushed the bsim-pg-17.0 branch from bfd51e7 to 8bda2e3 Compare October 27, 2024 18:51

ghidracadabra added Status: Prioritize This is currently being prioritized and removed Status: Triage Information is being gathered labels Nov 7, 2024

gemesa force-pushed the bsim-pg-17.0 branch from 8bda2e3 to 284d09e Compare November 8, 2024 10:20

BSim: Update PostgreSQL (15.3->17.0) to fix SSL errors

ba096c4

gemesa force-pushed the bsim-pg-17.0 branch from 284d09e to ba096c4 Compare November 13, 2024 18:45

gemesa mentioned this pull request Nov 18, 2024

BSim Elasticsearch setup - ERROR Invalid URL specified: BSim URL missing DB name/path (BSimLaunchable) #7204

Closed

ghidracadabra added Status: Internal This is being tracked internally by the Ghidra team and removed Status: Prioritize This is currently being prioritized labels Dec 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BSim: Update PostgreSQL (15.3->17.0) to fix SSL errors #7085

BSim: Update PostgreSQL (15.3->17.0) to fix SSL errors #7085

gemesa commented Oct 23, 2024 •

edited

Loading

gemesa commented Oct 24, 2024 •

edited

Loading

byt3exec left a comment

gemesa commented Oct 25, 2024 •

edited

Loading

gemesa commented Oct 27, 2024 •

edited

Loading

ghidra1 commented Dec 20, 2024

gemesa commented Dec 20, 2024

BSim: Update PostgreSQL (15.3->17.0) to fix SSL errors #7085

Are you sure you want to change the base?

BSim: Update PostgreSQL (15.3->17.0) to fix SSL errors #7085

Conversation

gemesa commented Oct 23, 2024 • edited Loading

gemesa commented Oct 24, 2024 • edited Loading

byt3exec left a comment

Choose a reason for hiding this comment

gemesa commented Oct 25, 2024 • edited Loading

gemesa commented Oct 27, 2024 • edited Loading

Dump DBs with PostgreSQL version 15.3

Restore DBs with PostgreSQL version 17.0

ghidra1 commented Dec 20, 2024

gemesa commented Dec 20, 2024

gemesa commented Oct 23, 2024 •

edited

Loading

gemesa commented Oct 24, 2024 •

edited

Loading

gemesa commented Oct 25, 2024 •

edited

Loading

gemesa commented Oct 27, 2024 •

edited

Loading