Do comparison in ocaml instead of dump the result and then run diff on it

[ghost-not-in-the-shell]

Explain your changes:
This PR modifies the verifier so that it directly does the comparison in ocaml instead of dump the result and then run diff on it. This change would make the verifier a little bit slower (not too bad). For mainnet, it increased about 30 secs.

Explain how you tested your changes:
I've run the verifier against the devnet data using this dump: gs://mina-archive-dumps/devnet-archive-migrated-2024-03-22_2050.sql.tar.gz and everything is good
I've also run the verifier against the mainnet using the following 2 dumps:

1. gs://mina-archive-dumps/mainnet-migrated-archive-dump-2024-04-04_0200.sql.tar.gz This is failing for the 3rd check

jiawei@bench-30vcpu-120gb-c2-cl:~/mina$ time _build/default/src/app/berkeley_migration_verifier/berkeley_migration_verifier.exe pre-fork --mainnet-archive-uri postgres://jiawei:postgres@127.0.0.1:5432/archive_balances_migrated --migrated-archive-uri postgres://jiawei:postgres@127.0.0.1:5432/migrated3
Running verifications for incremental migration between 'postgres://jiawei:postgres@127.0.0.1:5432/archive_balances_migrated' and 'postgres://jiawei:postgres@127.0.0.1:5432/migrated3' schemas. It may take a couple of minutes... 
[1/6] D3.1) Migrated blocks are connected  ... ✅ 
[2/6] D3.2) No orphaned nor pending blocks in migrated db  ... ✅ 
[3/6] D3.3) All accounts referred in internal commands or transactions are recorded in the accounts_accessed table.  ... ❌ 
	failed
[4/6] D3.4) All block hashes (state_hash, ledger_hashes) are equal  ... ✅ 
[5/6] D3.5) Verify user commands  ... ✅ 
[6/6] D3.6) Verify internal commands  ... ✅ 
Some tests failed. Please refer to above output for details
 - Details : 

real	1m27.514s
user	0m29.656s
sys	0m5.465s

And I got the same failing result with the latest berkeley verifier.

2. gs://mina-archive-dumps/mainnet-migrated-archive-dump-2024-02-10_0000.sql.tar.gz, everything is good for this dump.

jiawei@bench-30vcpu-120gb-c2-cl:~/mina$ time _build/default/src/app/berkeley_migration_verifier/berkeley_migration_verifier.exe pre-fork --mainnet-archive-uri postgres://jiawei:postgres@127.0.0.1:5432/archive_balances_migrated --migrated-archive-uri postgres://jiawei:postgres@127.0.0.1:5432/migrated4
Running verifications for incremental migration between 'postgres://jiawei:postgres@127.0.0.1:5432/archive_balances_migrated' and 'postgres://jiawei:postgres@127.0.0.1:5432/migrated4' schemas. It may take a couple of minutes... 
[1/6] D3.1) Migrated blocks are connected  ... ✅ 
[2/6] D3.2) No orphaned nor pending blocks in migrated db  ... ✅ 
[3/6] D3.3) All accounts referred in internal commands or transactions are recorded in the accounts_accessed table.  ... ✅ 
[4/6] D3.4) All block hashes (state_hash, ledger_hashes) are equal  ... ✅ 
[5/6] D3.5) Verify user commands  ... ✅ 
[6/6] D3.6) Verify internal commands  ... ✅ 

real	1m19.475s
user	0m26.957s
sys	0m5.131s

Checklist:

• Dependency versions are unchanged
  • Notify Velocity team if dependencies must change in CI
• Modified the current draft of release notes with details on what is completed or incomplete within this project
• Document code purpose, how to use it
  • Mention expected invariants, implicit constraints
• Tests were added for the new behavior
  • Document test purpose, significance of failures
  • Test names should reflect their purpose
• All tests pass (CI will check this if you didn't)
• Serialized types are in stable-versioned modules
• Does this close issues? List them

• Closes Devnet Migration Error - validating initial migration fails - must be superuser or member of pg_write_server_files #15476

[deepthiskumar]

!ci-build-me

[deepthiskumar]

!ci-build-me

[ghost-not-in-the-shell]

!ci-build-me

[ghost-not-in-the-shell]

!ci-build-me

[mrmr1993]

LGTM except: if the diff does fail, haven't we completely lost the ability to see why it failed?

[emberian]

Writing out something to a file is very helpful for this verifier for when it goes wrong, JSON for the lists would be just as good

[ghost-not-in-the-shell]

!ci-build-me

[ghost-not-in-the-shell]

!ci-build-me

[deepthiskumar]

!approved-for-mainnet

[jrwashburn]

All accounts referred in internal commands or transactions are recorded in the accounts_accessed table. ... ❌
failed

I'm curious why this is okay? And is this just because devnet, or would this be okay on mainnet too?

[deepthiskumar]

@jrwashburn the first test run seems to be using a DB instance that was incorrectly migrated. @ghost-not-in-the-shell can you confirm?
All the validations must pass for both devnet and mainnet.

[jrwashburn]

@jrwashburn the first test run seems to be using a DB instance that was incorrectly migrated. @ghost-not-in-the-shell can you confirm? All the validations must pass for both devnet and mainnet.

Okay so is there a build that I can run now? I'd like to get at least an incremental that checks out okay if possible. Sorry, I'm not sure where to find the artifacts.

[jrwashburn]

I tried an incremental run with the unstable build, but I'm getting the same discrepancy reported:

2024-04-09 00:56:41 UTC [Info] Done populating original genesis ledger balances!
(monitor.ml.Error
 (Sys_error "migration-checkpoint-443967: No such file or directory")
 ("Raised by primitive operation at Stdlib.open_in_gen in file \"stdlib.ml\", line 405, characters 28-54"
  "Called from Stdlib.open_in in file \"stdlib.ml\" (inlined), line 410, characters 2-45"
  "Called from Yojson.Safe.from_file in file \"lib/read.mll\", line 1135, characters 13-25"
  "Called from Dune__exe__Replayer.main in file \"src/app/replayer/replayer.ml\", line 653, characters 13-45"
  "Called from Async_kernel__Monitor.Exported_for_scheduler.schedule'.upon_work_fill_i in file \"src/monitor.ml\", line 295, characters 42-51"
  "Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5"
  "Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47"))
Running verifications for incremental migration between 'postgres://.../devnet-archive' and 'postgres://.../devnet-archive-berkeley' schemas. It may take a couple of minutes... 
[1/6] D3.1) Migrated blocks are connected  ... ✅ 
[2/6] D3.2) No orphaned nor pending blocks in migrated db  ... ✅ 
[3/6] D3.3) All accounts referred in internal commands or transactions are recorded in the accounts_accessed table.  ... ❌ 
        Discrepancies found between Berkeley.user_and_internal_command and Berkeley.accounts_accessed. To reproduce please run `diff /tmp/Berkeley.user_and_internal_command.json /tmp/Berkeley.accounts_accessed.json`.
[4/6] D3.4) All block hashes (state_hash, ledger_hashes) are equal  ... ✅ 
[5/6] D3.5) Verify user commands  ... ✅ 
[6/6] D3.6) Verify internal commands  ... ✅ 
Some tests failed. Please refer to above output for details
 - Details : ```

[jrwashburn]

When I read the error message "migration-checkpoint-443967: No such file or directory") and pass in the correct checkpoint file name, it works. Sorry for the confusion.