Change `printf` to PDC logger #232

TheAssembler1 · 2025-03-20T00:15:04Z

This is a large commit but I think it is very beneficial for all logging to go through the PDC logger. This way we get the filename, function name, and line numbers for all logging.

Also removed large blocks of comments and changed the PDC logger to print the file name, function, and line number.

We can now remove a lot of the code that manually printed the line numbers for debugging and error identification reasons.

TheAssembler1 · 2025-03-20T00:33:32Z

Still working on the clang-format I ran it on the codebase I'll look at what the cd/ci scripts do...

Also removed large blocks of comments and chanegd the pdc logger to print the file name, function, and line number.

examples/haccio.c

examples/haccio_v2.c

examples/llsm/llsm_aux/parallelReadTiff.c

examples/obj_get_data.c

examples/pdc_init.c

examples/obj_get_data.c

examples/read_write_perf.c

houjun

@TheAssembler1 can you also add to the documentation about these new macros and how to use them? You can add a subsection in Developer Notes

TheAssembler1 · 2025-03-20T16:01:51Z

@TheAssembler1 can you also add to the documentation about these new macros and how to use them? You can add a subsection in Developer Notes

I didn't create the logging macros they were already there just no one was using them. Yes I will add documentation for them.

TheAssembler1 · 2025-03-22T13:21:47Z

@TheAssembler1 can you also add to the documentation about these new macros and how to use them? You can add a subsection in Developer Notes

Ok I added docs for each of the macros in the developer notes.

jeanbez · 2025-03-24T20:07:14Z

Do we need the macro in the example codes? Could that confuse folks between logs in the server and regular print messages of tests/examples?

* Fix cache flush (#226) * Fix a thread race issue that may cause memory error when larger than cache max size data is transferred * Add a test that writes more data than server cache size * Fix CI run command * Update nersc.yml (#238) * Since PDCinit returns a uint64_t, 0 should indicate failure (#233) Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Check the return value of `PDC_Client_init` in `PDC_init` (#230) * Check that return value of PDC_Client_init in PDC_init * Change return to 0 This will make is simpler when merging #233 (comment) --------- Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Change `printf` to PDC logger (#232) * Changed all printf to use pdc logger Also removed large blocks of comments and chanegd the pdc logger to print the file name, function, and line number. * Change typo of LOG_INFO to LOG_ERROR * Correct grammar from fail -> failed * update grammer succesfully close -> successfully closed * switch type of LOG_INFO to LOG_ERROR * Add logging docs and fix some LOG_INFO->LOG_JUST_PRINT * update clang formatting --------- Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Malloc correct size for pdc_obj_metadata_pkg (#237) Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * PDCregion_transfer_create validate client buf, local region, and remote regions (#236) Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> --------- Co-authored-by: Houjun Tang <htang4@lbl.gov> Co-authored-by: Noah Lewis <47840925+TheAssembler1@users.noreply.github.com>

* Fix cache flush (#226) * Fix a thread race issue that may cause memory error when larger than cache max size data is transferred * Add a test that writes more data than server cache size * Fix CI run command * Fix restart issue * Update nersc.yml (#238) * Since PDCinit returns a uint64_t, 0 should indicate failure (#233) Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Check the return value of `PDC_Client_init` in `PDC_init` (#230) * Check that return value of PDC_Client_init in PDC_init * Change return to 0 This will make is simpler when merging #233 (comment) --------- Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Change `printf` to PDC logger (#232) * Changed all printf to use pdc logger Also removed large blocks of comments and chanegd the pdc logger to print the file name, function, and line number. * Change typo of LOG_INFO to LOG_ERROR * Correct grammar from fail -> failed * update grammer succesfully close -> successfully closed * switch type of LOG_INFO to LOG_ERROR * Add logging docs and fix some LOG_INFO->LOG_JUST_PRINT * update clang formatting --------- Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Malloc correct size for pdc_obj_metadata_pkg (#237) Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * PDCregion_transfer_create validate client buf, local region, and remote regions (#236) Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> --------- Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> Co-authored-by: Noah Lewis <47840925+TheAssembler1@users.noreply.github.com>

* Add pdc_logger.h to installation (#245) * sync with gitlab (#248) * Fix restart issue (#228) * Fix cache flush (#226) * Fix a thread race issue that may cause memory error when larger than cache max size data is transferred * Add a test that writes more data than server cache size * Fix CI run command * Fix restart issue * Update nersc.yml (#238) * Since PDCinit returns a uint64_t, 0 should indicate failure (#233) Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Check the return value of `PDC_Client_init` in `PDC_init` (#230) * Check that return value of PDC_Client_init in PDC_init * Change return to 0 This will make is simpler when merging #233 (comment) --------- Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Change `printf` to PDC logger (#232) * Changed all printf to use pdc logger Also removed large blocks of comments and chanegd the pdc logger to print the file name, function, and line number. * Change typo of LOG_INFO to LOG_ERROR * Correct grammar from fail -> failed * update grammer succesfully close -> successfully closed * switch type of LOG_INFO to LOG_ERROR * Add logging docs and fix some LOG_INFO->LOG_JUST_PRINT * update clang formatting --------- Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Malloc correct size for pdc_obj_metadata_pkg (#237) Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * PDCregion_transfer_create validate client buf, local region, and remote regions (#236) Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> --------- Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> Co-authored-by: Noah Lewis <47840925+TheAssembler1@users.noreply.github.com> * Fix return metadata dtype (#246) Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Region info transfer struct type and helper functions (#247) * Fix cache flush (#226) * Fix a thread race issue that may cause memory error when larger than cache max size data is transferred * Add a test that writes more data than server cache size * Fix CI run command * checkpoint * Switch variables such as count_0, start_0, and size0... to arrays This will reduce code duplication, reduce bugs, and make it easier to switch to support n-dimnesional data. * clang format * checkpoint * created better function names and documentation * remove * Committing clang-format changes * clang format * remove file * change for use helper function * fix bug with incorrect helper function call --------- Co-authored-by: Houjun Tang <htang4@lbl.gov> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Fix issues with PDC tools (#249) * Fix issues with PDC tools * Correct LOG_ERROR to LOG_INFO * Committing clang-format changes --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Fix printing in `PGOTO_ERROR` and `PGOTO_ERROR_VOID` (#250) Print new line by default in `PGOTO_ERROR` and `PGOTO_ERROR_VOID` Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Group Tests Into Folders (#252) * Fix cache flush (#226) * Fix a thread race issue that may cause memory error when larger than cache max size data is transferred * Add a test that writes more data than server cache size * Fix CI run command * Grouped commons tests into folders This commit also changes the src/tests/CmakeLists.txt to build tests within their new folders * add deprecated folder remove buf_map folder * Update run_multiple_mpi_test.sh * Update dependencies-macos.sh * Update dependencies-macos.sh --------- Co-authored-by: Houjun Tang <htang4@lbl.gov> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> Co-authored-by: Jean Luca Bez <jeanlucabez@gmail.com> * Return the same obj_id if the obj is just created or already opened (#254) * Return the same obj_id if the obj is just created or already opened * Committing clang-format changes * Update doc * Update dependencies-macos.sh --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * add option to choose interface (#255) * add option to connect to a given network interface * Committing clang-format changes * fix conflict * include header * enable output on failure --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * Fix multithreading compilation (#259) * fix multhreading compilation * Committing clang-format changes --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Fix segmentation fault of calling `PDCobj_create_mpi` twice with duplicate object name (#262) * Validate sucess of PDC_obj_create and PDC_find_id in PDCobj_create_mpi * Committing clang-format changes --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Use `PDC_malloc`, `PDC_free`, `PDC_calloc`, and `PDC_realloc` (#260) * checkpoint * replace free with PDC_free and calloc with PDC_calloc * Committing clang-format changes * fix more mallocs to PDC_malloc * more PDC_free fixes * Committing clang-format changes * Update ubuntu-cache.yml * remove eno1 * fix realloc * Committing clang-format changes * Update ubuntu-no-cache.yaml * Fix several bugs with error checking with object dim allocation * Committing clang-format changes * fix bug * Committing clang-format changes * Update ubuntu-no-cache.yaml * Update ubuntu-cache.yml * Set default value of ndim to 1 in PDCprop_create when using PDC_OBJ_CREATE * Committing clang-format changes * Malloc when defaulting to ndim size 1. Only free hostname when we PDC_malloc the memory because pointers returned by getenv are not malloced and could point to static memory. * Committing clang-format changes * Update README.md minor change to trigger the pipeline --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> Co-authored-by: Jean Luca Bez <jeanlucabez@gmail.com> * Fix Sphinx documentation errors and warnings (#265) * Fix all sphinx warnings and errors. Removed repeat declarations of functions. * Committing clang-format changes * remove def of EXTENSION_MAPPING * gitignore for docs and fix c structs * Committing clang-format changes --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Replace `docs/README.md` -> steps to build docs (#268) * Replace docs/README.md -> steps to build docs * Update README.md --------- Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Use `FUNC_ENTER` and `FUNC_LEAVE` (#270) * use func enter and func leave in all functions * Committing clang-format changes * fix infinite recursion between memory managment, hash table, and per function timing * Committing clang-format changes * add profiling to CI --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * New test macros and code cleanup (#261) * checkpoint * Committing clang-format changes * some tests * Committing clang-format changes * checkpoint * open_obj uses new test macros * Committing clang-format changes * read_obj uses TASSERT * read_obj uses TASSERT * Committing clang-format changes * cont_del and cont_getid use test macros * convert more tests to use macros * convert more tests to macros * Committing clang-format changes * Committing clang-format changes * clang format * use test helper in cont_info and cont_add_del * more tests use macros * Committing clang-format changes * use tests macros in more tests * use PGOTO* macros instead of goto * clang format * more log fixes * logging cleanup and more usage of test macros * Committing clang-format changes * clang format and fix CMakeLists for tests * use tests macros in transfer overlap 2D/3D * use TASSERT in more tests * Committing clang-format changes * use test asserts * all tests on the CI use TASSERT * fix printing and newlines in tests * print time, file name, function name, and line number in debug print --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * Tests logging typo fix (#273) * Fixed logging typos * Committing clang-format changes --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * Rename pdc_server.exe to pdc_server for consistency (#275) Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Update vpicio_mts.c (#276) Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Client Propogate `HG_Finalize` error on `PDCclose` (#263) * all but 4 close errors are fixed * Committing clang-format changes * client side HG_Finalize now passes on serial tests * Committing clang-format changes * cleanup * Committing clang-format changes * Update pdc_region_transfer.c * free bulk handles during region transfer close --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * Standardize ID Lookup Null Checks and Error Handling (#281) * cleanup finding id's * Committing clang-format changes --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * Obj open fix (#279) * Fix seg fault for PDCobj_open on non-existent object * Committing clang-format changes * Remove log from NULL check * Log message when object metadata isn't found. --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Fix multithread (#274) * move hash table mutex to hashtable source filse * Committing clang-format changes * add multithread compile test --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Fix seg fault when mercury initialization fails (#283) * check for NULL paramterse in hash table * Committing clang-format changes --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> --------- Co-authored-by: Noah Lewis <47840925+TheAssembler1@users.noreply.github.com> Co-authored-by: Houjun Tang <htang4@lbl.gov> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>

TheAssembler1 requested a review from a team as a code owner March 20, 2025 00:15

TheAssembler1 temporarily deployed to internal March 20, 2025 00:15 — with GitHub Actions Inactive

TheAssembler1 force-pushed the pdc_logging branch from 9a102b1 to b5f8832 Compare March 20, 2025 00:18

TheAssembler1 temporarily deployed to internal March 20, 2025 00:18 — with GitHub Actions Inactive

TheAssembler1 force-pushed the pdc_logging branch from b5f8832 to cf63b42 Compare March 20, 2025 00:23

TheAssembler1 temporarily deployed to internal March 20, 2025 00:23 — with GitHub Actions Inactive

TheAssembler1 force-pushed the pdc_logging branch from cf63b42 to 71ed74b Compare March 20, 2025 00:25

TheAssembler1 temporarily deployed to internal March 20, 2025 00:26 — with GitHub Actions Inactive

Changed all printf to use pdc logger

1e0534c

Also removed large blocks of comments and chanegd the pdc logger to print the file name, function, and line number.

TheAssembler1 force-pushed the pdc_logging branch from 71ed74b to 1e0534c Compare March 20, 2025 00:41

TheAssembler1 temporarily deployed to internal March 20, 2025 00:41 — with GitHub Actions Inactive

sbyna requested review from houjun and jeanbez March 20, 2025 00:42