Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'debug_blocks' only lists the block distribution on first proc since #602 #718

Closed
phil-blain opened this issue May 12, 2022 · 3 comments · Fixed by #750
Closed

'debug_blocks' only lists the block distribution on first proc since #602 #718

phil-blain opened this issue May 12, 2022 · 3 comments · Fixed by #750
Assignees

Comments

@phil-blain
Copy link
Member

phil-blain commented May 12, 2022

Hi @apcraig, I did not have time to review this PR (#602) before it was merged.

Here:

if (debug_blocks .and. my_task == master_task) then
write(nu_diag,'(2a,3i8)') &
subname,' block id, proc, local_block: ', &
block_ids(distribution%blockLocalID(n)), &
distribution%blockLocation(n), &
distribution%blockLocalID(n)

I notice that you prevented the block distribution from being outputed for all procs. Was this intended ? If we are debugging decompositions, I would expect we want output from all procs....

I was using debug_blocks just now to debug a segfault and was very confused as to why all blocks were listed on proc 1, and I dug up this PR by looking at the history of create_local_block_ids...

Originally posted by @phil-blain in #602 (comment)

@apcraig
Copy link
Contributor

apcraig commented May 12, 2022

This was done to have readable output. Otherwise, all pes are writing to the same file at the same time. I could probably modify this, so it loops over pes then barriers and flushes between pe writes. Or we could write out from each pe to it's own file. Or we can leave as is or something else. Preferences?

@phil-blain
Copy link
Member Author

I think looping over procs + barrier + flushing would be best.

@apcraig
Copy link
Contributor

apcraig commented Aug 8, 2022

Working on a PR now to update the debug_blocks output. New output will include information about the overall decomposition as follows,

> (create_blocks) block ID, iblock, jblock Locations:
> (create_blocks) global block ID, iblock, jblock, tripole:       1       1       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:       2       2       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:       3       3       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:       4       4       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:       5       5       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:       6       6       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:       7       7       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:       8       8       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:       9       9       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:      10      10       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:      11      11       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:      12      12       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:      13      13       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:      14      14       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:      15      15       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:      16      16       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:      17      17       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:      18      18       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:      19      19       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:      20      20       1   F
> (create_blocks) global block ID, iblock, jblock, tripole:      21       1       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      22       2       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      23       3       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      24       4       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      25       5       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      26       6       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      27       7       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      28       8       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      29       9       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      30      10       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      31      11       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      32      12       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      33      13       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      34      14       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      35      15       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      36      16       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      37      17       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      38      18       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      39      19       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      40      20       2   F
> (create_blocks) global block ID, iblock, jblock, tripole:      41       1       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      42       2       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      43       3       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      44       4       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      45       5       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      46       6       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      47       7       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      48       8       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      49       9       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      50      10       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      51      11       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      52      12       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      53      13       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      54      14       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      55      15       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      56      16       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      57      17       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      58      18       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      59      19       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      60      20       3   F
> (create_blocks) global block ID, iblock, jblock, tripole:      61       1       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      62       2       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      63       3       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      64       4       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      65       5       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      66       6       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      67       7       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      68       8       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      69       9       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      70      10       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      71      11       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      72      12       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      73      13       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      74      14       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      75      15       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      76      16       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      77      17       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      78      18       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      79      19       4   F
> (create_blocks) global block ID, iblock, jblock, tripole:      80      20       4   F
526c608
<   debug_blocks          =      F
---
>   debug_blocks          =      T
535a618,783
>   
> (init_domain_distribution) Blocks by Proc:
> (init_domain_distribution) my_task, local block ID, global block ID:        0       1       1
> (init_domain_distribution) my_task, local block ID, global block ID:        0       2       9
> (init_domain_distribution) my_task, local block ID, global block ID:        0       3      17
> (init_domain_distribution) my_task, local block ID, global block ID:        0       4      26
> (init_domain_distribution) my_task, local block ID, global block ID:        0       5      34
> (init_domain_distribution) my_task, local block ID, global block ID:        0       6      42
> (init_domain_distribution) my_task, local block ID, global block ID:        0       7      50
> (init_domain_distribution) my_task, local block ID, global block ID:        0       8      58
> (init_domain_distribution) my_task, local block ID, global block ID:        0       9      66
> (init_domain_distribution) my_task, local block ID, global block ID:        0      10      74
> (init_domain_distribution) my_task, local block ID, global block ID:        1       1       2
> (init_domain_distribution) my_task, local block ID, global block ID:        1       2      10
> (init_domain_distribution) my_task, local block ID, global block ID:        1       3      18
> (init_domain_distribution) my_task, local block ID, global block ID:        1       4      27
> (init_domain_distribution) my_task, local block ID, global block ID:        1       5      35
> (init_domain_distribution) my_task, local block ID, global block ID:        1       6      43
> (init_domain_distribution) my_task, local block ID, global block ID:        1       7      51
> (init_domain_distribution) my_task, local block ID, global block ID:        1       8      59
> (init_domain_distribution) my_task, local block ID, global block ID:        1       9      67
> (init_domain_distribution) my_task, local block ID, global block ID:        1      10      75
> (init_domain_distribution) my_task, local block ID, global block ID:        2       1       3
> (init_domain_distribution) my_task, local block ID, global block ID:        2       2      11
> (init_domain_distribution) my_task, local block ID, global block ID:        2       3      19
> (init_domain_distribution) my_task, local block ID, global block ID:        2       4      28
> (init_domain_distribution) my_task, local block ID, global block ID:        2       5      36
> (init_domain_distribution) my_task, local block ID, global block ID:        2       6      44
> (init_domain_distribution) my_task, local block ID, global block ID:        2       7      52
> (init_domain_distribution) my_task, local block ID, global block ID:        2       8      60
> (init_domain_distribution) my_task, local block ID, global block ID:        2       9      68
> (init_domain_distribution) my_task, local block ID, global block ID:        2      10      76
> (init_domain_distribution) my_task, local block ID, global block ID:        3       1       4
> (init_domain_distribution) my_task, local block ID, global block ID:        3       2      12
> (init_domain_distribution) my_task, local block ID, global block ID:        3       3      20
> (init_domain_distribution) my_task, local block ID, global block ID:        3       4      29
> (init_domain_distribution) my_task, local block ID, global block ID:        3       5      37
> (init_domain_distribution) my_task, local block ID, global block ID:        3       6      45
> (init_domain_distribution) my_task, local block ID, global block ID:        3       7      53
> (init_domain_distribution) my_task, local block ID, global block ID:        3       8      61
> (init_domain_distribution) my_task, local block ID, global block ID:        3       9      69
> (init_domain_distribution) my_task, local block ID, global block ID:        3      10      77
> (init_domain_distribution) my_task, local block ID, global block ID:        4       1       5
> (init_domain_distribution) my_task, local block ID, global block ID:        4       2      13
> (init_domain_distribution) my_task, local block ID, global block ID:        4       3      21
> (init_domain_distribution) my_task, local block ID, global block ID:        4       4      30
> (init_domain_distribution) my_task, local block ID, global block ID:        4       5      38
> (init_domain_distribution) my_task, local block ID, global block ID:        4       6      46
> (init_domain_distribution) my_task, local block ID, global block ID:        4       7      54
> (init_domain_distribution) my_task, local block ID, global block ID:        4       8      62
> (init_domain_distribution) my_task, local block ID, global block ID:        4       9      70
> (init_domain_distribution) my_task, local block ID, global block ID:        4      10      78
> (init_domain_distribution) my_task, local block ID, global block ID:        5       1       6
> (init_domain_distribution) my_task, local block ID, global block ID:        5       2      14
> (init_domain_distribution) my_task, local block ID, global block ID:        5       3      22
> (init_domain_distribution) my_task, local block ID, global block ID:        5       4      31
> (init_domain_distribution) my_task, local block ID, global block ID:        5       5      39
> (init_domain_distribution) my_task, local block ID, global block ID:        5       6      47
> (init_domain_distribution) my_task, local block ID, global block ID:        5       7      55
> (init_domain_distribution) my_task, local block ID, global block ID:        5       8      63
> (init_domain_distribution) my_task, local block ID, global block ID:        5       9      71
> (init_domain_distribution) my_task, local block ID, global block ID:        5      10      79
> (init_domain_distribution) my_task, local block ID, global block ID:        6       1       7
> (init_domain_distribution) my_task, local block ID, global block ID:        6       2      15
> (init_domain_distribution) my_task, local block ID, global block ID:        6       3      23
> (init_domain_distribution) my_task, local block ID, global block ID:        6       4      32
> (init_domain_distribution) my_task, local block ID, global block ID:        6       5      40
> (init_domain_distribution) my_task, local block ID, global block ID:        6       6      48
> (init_domain_distribution) my_task, local block ID, global block ID:        6       7      56
> (init_domain_distribution) my_task, local block ID, global block ID:        6       8      64
> (init_domain_distribution) my_task, local block ID, global block ID:        6       9      72
> (init_domain_distribution) my_task, local block ID, global block ID:        6      10      80
> (init_domain_distribution) my_task, local block ID, global block ID:        7       1       8
> (init_domain_distribution) my_task, local block ID, global block ID:        7       2      16
> (init_domain_distribution) my_task, local block ID, global block ID:        7       3      25
> (init_domain_distribution) my_task, local block ID, global block ID:        7       4      33
> (init_domain_distribution) my_task, local block ID, global block ID:        7       5      41
> (init_domain_distribution) my_task, local block ID, global block ID:        7       6      49
> (init_domain_distribution) my_task, local block ID, global block ID:        7       7      57
> (init_domain_distribution) my_task, local block ID, global block ID:        7       8      65
> (init_domain_distribution) my_task, local block ID, global block ID:        7       9      73
>   
> (init_domain_distribution) Blocks by Global Block ID:
> (init_domain_distribution) global block id, proc, local block ID:        1       1       1
> (init_domain_distribution) global block id, proc, local block ID:        2       2       1
> (init_domain_distribution) global block id, proc, local block ID:        3       3       1
> (init_domain_distribution) global block id, proc, local block ID:        4       4       1
> (init_domain_distribution) global block id, proc, local block ID:        5       5       1
> (init_domain_distribution) global block id, proc, local block ID:        6       6       1
> (init_domain_distribution) global block id, proc, local block ID:        7       7       1
> (init_domain_distribution) global block id, proc, local block ID:        8       8       1
> (init_domain_distribution) global block id, proc, local block ID:        9       1       2
> (init_domain_distribution) global block id, proc, local block ID:       10       2       2
> (init_domain_distribution) global block id, proc, local block ID:       11       3       2
> (init_domain_distribution) global block id, proc, local block ID:       12       4       2
> (init_domain_distribution) global block id, proc, local block ID:       13       5       2
> (init_domain_distribution) global block id, proc, local block ID:       14       6       2
> (init_domain_distribution) global block id, proc, local block ID:       15       7       2
> (init_domain_distribution) global block id, proc, local block ID:       16       8       2
> (init_domain_distribution) global block id, proc, local block ID:       17       1       3
> (init_domain_distribution) global block id, proc, local block ID:       18       2       3
> (init_domain_distribution) global block id, proc, local block ID:       19       3       3
> (init_domain_distribution) global block id, proc, local block ID:       20       4       3
> (init_domain_distribution) global block id, proc, local block ID:       21       5       3
> (init_domain_distribution) global block id, proc, local block ID:       22       6       3
> (init_domain_distribution) global block id, proc, local block ID:       23       7       3
> (init_domain_distribution) global block id, proc, local block ID:       24       0       0
> (init_domain_distribution) global block id, proc, local block ID:       25       8       3
> (init_domain_distribution) global block id, proc, local block ID:       26       1       4
> (init_domain_distribution) global block id, proc, local block ID:       27       2       4
> (init_domain_distribution) global block id, proc, local block ID:       28       3       4
> (init_domain_distribution) global block id, proc, local block ID:       29       4       4
> (init_domain_distribution) global block id, proc, local block ID:       30       5       4
> (init_domain_distribution) global block id, proc, local block ID:       31       6       4
> (init_domain_distribution) global block id, proc, local block ID:       32       7       4
> (init_domain_distribution) global block id, proc, local block ID:       33       8       4
> (init_domain_distribution) global block id, proc, local block ID:       34       1       5
> (init_domain_distribution) global block id, proc, local block ID:       35       2       5
> (init_domain_distribution) global block id, proc, local block ID:       36       3       5
> (init_domain_distribution) global block id, proc, local block ID:       37       4       5
> (init_domain_distribution) global block id, proc, local block ID:       38       5       5
> (init_domain_distribution) global block id, proc, local block ID:       39       6       5
> (init_domain_distribution) global block id, proc, local block ID:       40       7       5
> (init_domain_distribution) global block id, proc, local block ID:       41       8       5
> (init_domain_distribution) global block id, proc, local block ID:       42       1       6
> (init_domain_distribution) global block id, proc, local block ID:       43       2       6
> (init_domain_distribution) global block id, proc, local block ID:       44       3       6
> (init_domain_distribution) global block id, proc, local block ID:       45       4       6
> (init_domain_distribution) global block id, proc, local block ID:       46       5       6
> (init_domain_distribution) global block id, proc, local block ID:       47       6       6
> (init_domain_distribution) global block id, proc, local block ID:       48       7       6
> (init_domain_distribution) global block id, proc, local block ID:       49       8       6
> (init_domain_distribution) global block id, proc, local block ID:       50       1       7
> (init_domain_distribution) global block id, proc, local block ID:       51       2       7
> (init_domain_distribution) global block id, proc, local block ID:       52       3       7
> (init_domain_distribution) global block id, proc, local block ID:       53       4       7
> (init_domain_distribution) global block id, proc, local block ID:       54       5       7
> (init_domain_distribution) global block id, proc, local block ID:       55       6       7
> (init_domain_distribution) global block id, proc, local block ID:       56       7       7
> (init_domain_distribution) global block id, proc, local block ID:       57       8       7
> (init_domain_distribution) global block id, proc, local block ID:       58       1       8
> (init_domain_distribution) global block id, proc, local block ID:       59       2       8
> (init_domain_distribution) global block id, proc, local block ID:       60       3       8
> (init_domain_distribution) global block id, proc, local block ID:       61       4       8
> (init_domain_distribution) global block id, proc, local block ID:       62       5       8
> (init_domain_distribution) global block id, proc, local block ID:       63       6       8
> (init_domain_distribution) global block id, proc, local block ID:       64       7       8
> (init_domain_distribution) global block id, proc, local block ID:       65       8       8
> (init_domain_distribution) global block id, proc, local block ID:       66       1       9
> (init_domain_distribution) global block id, proc, local block ID:       67       2       9
> (init_domain_distribution) global block id, proc, local block ID:       68       3       9
> (init_domain_distribution) global block id, proc, local block ID:       69       4       9
> (init_domain_distribution) global block id, proc, local block ID:       70       5       9
> (init_domain_distribution) global block id, proc, local block ID:       71       6       9
> (init_domain_distribution) global block id, proc, local block ID:       72       7       9
> (init_domain_distribution) global block id, proc, local block ID:       73       8       9
> (init_domain_distribution) global block id, proc, local block ID:       74       1      10
> (init_domain_distribution) global block id, proc, local block ID:       75       2      10
> (init_domain_distribution) global block id, proc, local block ID:       76       3      10
> (init_domain_distribution) global block id, proc, local block ID:       77       4      10
> (init_domain_distribution) global block id, proc, local block ID:       78       5      10
> (init_domain_distribution) global block id, proc, local block ID:       79       6      10
> (init_domain_distribution) global block id, proc, local block ID:       80       7      10
>   
> (init_domain_distribution) ice_distributionGet checks pass
>   

apcraig added a commit to apcraig/CICE that referenced this issue Aug 10, 2022
apcraig added a commit that referenced this issue Aug 15, 2022
* Update/improve debug_blocks output, see #718.

* Add ICE_MEMUSE cice.settings flag for batch memory use
Add set_env.memsmall, memmed, memlarge options
To use, will require changes to the env machine files.  Most machines will probably not use it.
See #674.

* Add setup_machparams.csh to compute batch/launch machine parameters
Update cice.batch.csh and cice.launch.csh to use setup_machparams.csh
See #650

* Update subroutine diagnostic_abort which calls print_state
Update ice_transport_remap and ice_transport_driver to call diagnostic_abort
  during some errors.
See also #622

* Update miniconda install information
See #547

* Code cleanup based on compile with -Wall
Code cleanup based on -std f2003 and f2008 checks
Add -stand f08 to cheyenne_intel debug flags
Add -std f2008 to cheyenne_gnu debug flags
Code consistent with Fortran 2003 except for use of contiguous in
  1d evp code.

* Remove all trailing blank space with script

* Update the cheyenne env so qc testing works
Add configuration/scripts/tests/qctest.yml file
Update documentation

* Update Icepack

* Clean up some output

* fix comments

* update print_state output
dabail10 pushed a commit to ESCOMP/CICE that referenced this issue Oct 4, 2022
* Update/improve debug_blocks output, see CICE-Consortium#718.

* Add ICE_MEMUSE cice.settings flag for batch memory use
Add set_env.memsmall, memmed, memlarge options
To use, will require changes to the env machine files.  Most machines will probably not use it.
See CICE-Consortium#674.

* Add setup_machparams.csh to compute batch/launch machine parameters
Update cice.batch.csh and cice.launch.csh to use setup_machparams.csh
See CICE-Consortium#650

* Update subroutine diagnostic_abort which calls print_state
Update ice_transport_remap and ice_transport_driver to call diagnostic_abort
  during some errors.
See also CICE-Consortium#622

* Update miniconda install information
See CICE-Consortium#547

* Code cleanup based on compile with -Wall
Code cleanup based on -std f2003 and f2008 checks
Add -stand f08 to cheyenne_intel debug flags
Add -std f2008 to cheyenne_gnu debug flags
Code consistent with Fortran 2003 except for use of contiguous in
  1d evp code.

* Remove all trailing blank space with script

* Update the cheyenne env so qc testing works
Add configuration/scripts/tests/qctest.yml file
Update documentation

* Update Icepack

* Clean up some output

* fix comments

* update print_state output
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants