Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support to query local arrays #3824

Closed
wants to merge 45 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
f332881
1) removed unused code sections
guj Sep 23, 2023
1e96b21
clang-format v16
guj Sep 23, 2023
7066902
older clang-format
guj Sep 23, 2023
79528bd
removed extra ;
guj Sep 23, 2023
2149fe7
modified test. it failed because I consolidate block when all subblo…
guj Sep 24, 2023
2884f71
Import tests from bp to staging common, implement memory selection in…
eisenhauer Sep 22, 2023
51f7b6f
Try always using the MPI version
eisenhauer Sep 26, 2023
5bde490
Try always using the MPI version
eisenhauer Sep 27, 2023
c503940
Merge pull request #3823 from eisenhauer/SstMemSel
eisenhauer Sep 27, 2023
700839d
Add prototype testing of remote functionality (#3830)
eisenhauer Sep 30, 2023
107164d
Tweak Remote class and test multi-threaded file remote access (#3834)
eisenhauer Oct 14, 2023
ac07185
MPI: add timeout for conf test for MPI_DP (#3848)
vicentebolea Oct 17, 2023
8707f2b
Update test.cpp
guj Oct 17, 2023
d0fcf10
cmake: update minimum cmake to 3.12 (#3849)
vicentebolea Oct 17, 2023
975e8de
MPI_DP: do not call MPI_Init (#3847)
vicentebolea Oct 17, 2023
897ca42
Improve existing examples
spyridon97 Sep 14, 2023
3bdc1aa
Add examples from ADIOS2-Examples
spyridon97 Sep 18, 2023
fdee65d
Fix formatting issue in DetectOptions.cmake
spyridon97 Sep 19, 2023
b881acc
Remove testing based on ADIOS2-examples
spyridon97 Sep 19, 2023
bf63c65
Disable BUILD_TESTING and ADIOS2_BUILD_EXAMPLES by default
spyridon97 Sep 21, 2023
2425033
Add/Improve the ReadMe.md files in examples directory
spyridon97 Sep 21, 2023
326bb54
ci: enable tau check
vicentebolea Oct 16, 2023
2324daf
cmake: include ctest before detectoptions
vicentebolea Oct 17, 2023
861ccb7
Fix destdir install test (#3850)
vicentebolea Oct 18, 2023
a43d013
Merge pull request #3820 from spyridon97/organize-and-add-examples
spyridon97 Oct 18, 2023
6f33d87
Blosc2 USE ON: Fix Module Fallback (#3774)
ax3l Oct 19, 2023
2f581ec
BP5Serializer: handle local variables that use operators (#3859)
caitlinross Oct 20, 2023
3662939
Examples: Add BeginStep/EndStep wherever it was missing
spyridon97 Oct 11, 2023
e13cc18
inlineMWE example: Close files at the end
spyridon97 Oct 11, 2023
e98f442
Examples: Use BPFile instead of BP3/4/5 for future-proof
spyridon97 Oct 11, 2023
5315ead
Update the bpWriterReadHip example's cmake to run on crusher
spyridon97 Oct 19, 2023
2d4af78
Merge pull request #3856 from spyridon97/update-examples
spyridon97 Oct 20, 2023
2426600
bug fix: syntax error in json output (#3857)
guj Oct 20, 2023
347d356
Remove Slack as a contact mechanism (#3866)
eisenhauer Oct 20, 2023
afdef44
ascent,ci: remove unshallow flag
vicentebolea Oct 20, 2023
432c6ab
Server Improvements (#3862)
eisenhauer Oct 21, 2023
e53a1bf
Fix hip example compilation error (#3865)
spyridon97 Oct 21, 2023
de0ad6e
add changes to the query test file (someone moved to a new dir)
guj Oct 23, 2023
7322c8d
1) removed unused code sections
guj Sep 23, 2023
7402c20
clang-format v16
guj Sep 23, 2023
04a3dbc
older clang-format
guj Sep 23, 2023
75c34a1
removed extra ;
guj Sep 23, 2023
7c21614
modified test. it failed because I consolidate block when all subblo…
guj Sep 24, 2023
a5866d2
touched
guj Oct 23, 2023
a88271b
Merge branch 'queryLocalArray' of https://github.com/guj/ADIOS2 into …
guj Oct 23, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
9 changes: 2 additions & 7 deletions .github/workflows/everything.yml
Original file line number Diff line number Diff line change
Expand Up @@ -448,24 +448,19 @@ jobs:
strategy:
fail-fast: false
matrix:
code: [examples, lammps, tau]
code: [lammps, tau]
include:
- code: examples
repo: ornladios/ADIOS2-Examples
ref: master
- code: lammps
repo: pnorbert/lammps
ref: fix-deprecated-adios-init
- code: tau
repo: ornladios/ADIOS2-Examples
ref: master

defaults:
run:
shell: bash -c "docker exec adios2-ci bash --login -e $(echo {0} | sed 's|/home/runner/work|/__w|g')"

steps:
- uses: actions/checkout@v4
if: ${{ matrix.repo != '' }}
with:
repository: ${{ matrix.repo }}
ref: ${{ matrix.ref }}
Expand Down
8 changes: 4 additions & 4 deletions .gitlab/config/SpackCIBridge.py
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ def list_github_prs(self):
# Check if we should defer pushing/testing this PR because it is based on "too new" of a commit
# of the main branch.
tmp_pr_branch = f"temporary_{pr_string}"
subprocess.run(["git", "fetch", "--unshallow", "github",
subprocess.run(["git", "fetch", "github",
f"refs/pull/{pull.number}/head:{tmp_pr_branch}"], check=True)
# Get the merge base between this PR and the main branch.
try:
Expand Down Expand Up @@ -226,7 +226,7 @@ def list_github_prs(self):
# then we will push the merge commit that was automatically created by GitHub to GitLab
# where it will kick off a CI pipeline.
try:
subprocess.run(["git", "fetch", "--unshallow", "github",
subprocess.run(["git", "fetch", "github",
f"{pull.merge_commit_sha}:{pr_string}"], check=True)
except subprocess.CalledProcessError:
print("Failed to locally checkout PR {0} ({1}). Skipping"
Expand Down Expand Up @@ -306,7 +306,7 @@ def setup_git_repo(self):
self.gitlab_shallow_fetch()

if self.main_branch:
subprocess.run(["git", "fetch", "--unshallow", "github", self.main_branch], check=True)
subprocess.run(["git", "fetch", "github", self.main_branch], check=True)

def get_gitlab_pr_branches(self):
"""Query GitLab for branches that have already been copied over from GitHub PRs.
Expand Down Expand Up @@ -350,7 +350,7 @@ def update_refspecs_for_tags(self, tags, open_refspecs, fetch_refspecs):
def fetch_github_branches(self, fetch_refspecs):
"""Perform `git fetch` for a given list of refspecs."""
print("Fetching GitHub refs for open PRs")
fetch_args = ["git", "fetch", "-q", "--unshallow", "github"] + fetch_refspecs
fetch_args = ["git", "fetch", "-q", "github"] + fetch_refspecs
subprocess.run(fetch_args, check=True)

def build_local_branches(self, protected_branches):
Expand Down
10 changes: 6 additions & 4 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,11 @@ if((NOT BUILD_SHARED_LIBS) AND (NOT DEFINED CMAKE_POSITION_INDEPENDENT_CODE))
set(CMAKE_POSITION_INDEPENDENT_CODE ON)
endif()

# Ctest creates BUILD_TESTING option and sets it to true by default
# Here we disable BUILD_TESTING option by default
option(BUILD_TESTING "Build testing" OFF)
include(CTest)
mark_as_advanced(CLEAR BUILD_TESTING)

adios_option(Blosc2 "Enable support for c-blosc-2 transforms" AUTO)
adios_option(BZip2 "Enable support for BZip2 transforms" AUTO)
Expand Down Expand Up @@ -176,7 +180,7 @@ if(ADIOS2_HAVE_CUDA OR ADIOS2_HAVE_Kokkos_CUDA)
if(DEFINED Kokkos_CUDA_ARCHITECTURES)
set(CMAKE_CUDA_ARCHITECTURES ${Kokkos_CUDA_ARCHITECTURES})
else()
# Mininum common non-deprecated architecture
# Minimum common non-deprecated architecture
set(CMAKE_CUDA_ARCHITECTURES 52)
endif()
endif()
Expand Down Expand Up @@ -296,8 +300,6 @@ endif()
#------------------------------------------------------------------------------#
# Third party libraries
#------------------------------------------------------------------------------#
include(CTest)
mark_as_advanced(BUILD_TESTING)
add_subdirectory(thirdparty)

#------------------------------------------------------------------------------#
Expand All @@ -318,7 +320,7 @@ add_subdirectory(plugins)
#------------------------------------------------------------------------------#
# Examples
#------------------------------------------------------------------------------#
option(ADIOS2_BUILD_EXAMPLES "Build examples" ON)
option(ADIOS2_BUILD_EXAMPLES "Build examples" OFF)
if(ADIOS2_BUILD_EXAMPLES)
add_subdirectory(examples)
endif()
Expand Down
1 change: 0 additions & 1 deletion ReadMe.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,6 @@ Once ADIOS2 is installed refer to:

ADIOS2 is an open source project: Questions, discussion, and contributions are welcome. Join us at:

- Slack workspace: [![Slack](https://img.shields.io/badge/slack-ADIOS2-pink.svg)](https://adios2.spack.io)
- Mailing list: adios-ecp@kitware.com
- Github Discussions: https://github.com/ornladios/ADIOS2/discussions

Expand Down
8 changes: 7 additions & 1 deletion bindings/CXX11/adios2/cxx11/Query.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,11 @@ QueryWorker::QueryWorker(const std::string &configFile, adios2::Engine &reader)
delete m;
}

void QueryWorker::GetResultCoverage(std::vector<size_t> &touched_blockIDs)
{
m_Worker->GetResultCoverage(touched_blockIDs);
}

void QueryWorker::GetResultCoverage(std::vector<adios2::Box<adios2::Dims>> &touched_blocks)
{
adios2::Box<adios2::Dims> empty;
Expand All @@ -26,4 +31,5 @@ void QueryWorker::GetResultCoverage(const adios2::Box<adios2::Dims> &outputSelec
if (m_Worker)
return m_Worker->GetResultCoverage(outputSelection, touched_blocks);
}
}

} // namespace
1 change: 1 addition & 0 deletions bindings/CXX11/adios2/cxx11/Query.h
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ class QueryWorker
// configFile has query, can be either xml or json
QueryWorker(const std::string &configFile, adios2::Engine &engine);

void GetResultCoverage(std::vector<size_t> &touched_block_ids);
// touched_blocks is a list of regions specified by (start, count),
// that contains data that satisfies the query file
void GetResultCoverage(std::vector<adios2::Box<adios2::Dims>> &touched_blocks);
Expand Down
2 changes: 1 addition & 1 deletion bindings/Matlab/README.txt
Original file line number Diff line number Diff line change
Expand Up @@ -43,5 +43,5 @@ $ cd <your adios source>
$ git pull
$ mkdir build.lean
$ cd build.lean
$ cmake -DCMAKE_INSTALL_PREFIX=/opt/adios/lean -DADIOS2_USE_MPI=OFF -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DCMAKE_BUILD_TYPE=RelWithDebInfo -DADIOS2_ENABLE_PIC=ON -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TESTING=OFF -DADIOS2_USE_SST=OFF -DADIOS2_USE_BZip2=OFF -DADIOS2_USE_DataMan=OFF ..
$ cmake -DCMAKE_INSTALL_PREFIX=/opt/adios/lean -DADIOS2_USE_MPI=OFF -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DCMAKE_BUILD_TYPE=RelWithDebInfo -DADIOS2_ENABLE_PIC=ON -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DADIOS2_USE_SST=OFF -DADIOS2_USE_BZip2=OFF -DADIOS2_USE_DataMan=OFF ..

7 changes: 7 additions & 0 deletions bindings/Python/py11Query.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -36,5 +36,12 @@ std::vector<Box<Dims>> Query::GetResult()
return touched_blocks;
}

std::vector<size_t> Query::GetBlockIDs()
{
std::vector<size_t> touched_block_ids;
m_QueryWorker->GetResultCoverage(touched_block_ids);
return touched_block_ids;
}

} // py11
} // adios2
1 change: 1 addition & 0 deletions bindings/Python/py11Query.h
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ class Query
explicit operator bool() const noexcept;

std::vector<Box<Dims>> GetResult();
std::vector<size_t> GetBlockIDs();

private:
Query(adios2::query::Worker *qw);
Expand Down
3 changes: 2 additions & 1 deletion bindings/Python/py11glue.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -357,7 +357,8 @@ PYBIND11_MODULE(ADIOS2_PYTHON_MODULE_NAME, m)
"adios2 query construction, a xml query File and a read engine",
pybind11::arg("queryFile"), pybind11::arg("reader") = true)

.def("GetResult", &adios2::py11::Query::GetResult);
.def("GetResult", &adios2::py11::Query::GetResult)
.def("GetBlockIDs", &adios2::py11::Query::GetBlockIDs);

pybind11::class_<adios2::py11::Variable>(m, "Variable")
// Python 2
Expand Down
23 changes: 17 additions & 6 deletions cmake/DetectOptions.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ if(ADIOS2_USE_Blosc2 STREQUAL AUTO)
endif()
elseif(ADIOS2_USE_Blosc2)
# Prefect CONFIG mode
find_package(Blosc2 2.4 CONFIG REQUIRED)
find_package(Blosc2 2.4 CONFIG)
if(NOT Blosc2_FOUND)
find_package(Blosc2 2.4 MODULE REQUIRED)
endif()
Expand Down Expand Up @@ -216,7 +216,7 @@ if(ADIOS2_USE_Kokkos)
enable_language(HIP)
endif()
if(Kokkos_ENABLE_SYCL)
set(ADIOS2_HAVE_Kokkos_SYCL TRUE)
set(ADIOS2_HAVE_Kokkos_SYCL TRUE)
endif()
set(ADIOS2_HAVE_GPU_Support TRUE)
endif()
Expand Down Expand Up @@ -445,9 +445,11 @@ if(ADIOS2_USE_SST AND NOT WIN32)
endif()
endif()
if(ADIOS2_HAVE_MPI)
set(CMAKE_REQUIRED_LIBRARIES MPI::MPI_C)
include(CheckCSourceRuns)
check_c_source_runs([=[
set(CMAKE_REQUIRED_LIBRARIES "MPI::MPI_C;Threads::Threads")
include(CheckCXXSourceRuns)
check_cxx_source_runs([=[
#include <chrono>
#include <future>
#include <mpi.h>
#include <stdlib.h>

Expand All @@ -457,9 +459,18 @@ if(ADIOS2_USE_SST AND NOT WIN32)

int main()
{
// Timeout after 5 second
auto task = std::async(std::launch::async, []() {
std::this_thread::sleep_for(std::chrono::seconds(5));
exit(EXIT_FAILURE);
});

char* port_name = new char[MPI_MAX_PORT_NAME];
MPI_Init_thread(NULL, NULL, MPI_THREAD_MULTIPLE, NULL);
MPI_Open_port(MPI_INFO_NULL, malloc(sizeof(char) * MPI_MAX_PORT_NAME));
MPI_Open_port(MPI_INFO_NULL, port_name);
MPI_Close_port(port_name);
MPI_Finalize();
exit(EXIT_SUCCESS);
}]=]
ADIOS2_HAVE_MPI_CLIENT_SERVER)
unset(CMAKE_REQUIRED_LIBRARIES)
Expand Down
2 changes: 1 addition & 1 deletion cmake/adios2-config-install.cmake.in
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
cmake_minimum_required(VERSION 3.1)
cmake_minimum_required(VERSION 3.12)

set(_ADIOS2_CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH})
list(INSERT CMAKE_MODULE_PATH 0 "${CMAKE_CURRENT_LIST_DIR}")
Expand Down
2 changes: 1 addition & 1 deletion cmake/install/post/adios2-config-dummy/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
cmake_minimum_required(VERSION 3.0)
cmake_minimum_required(VERSION 3.12)

project(adios2-config-dummy C CXX)

Expand Down
6 changes: 6 additions & 0 deletions cmake/install/post/generate-adios2-config.sh.in
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,12 @@ if [ "${1:0:1}" != "/" ]
then
# Convert relative paths to absolute based on DESTDIR
PREFIX="${DESTDIR:-$(pwd)}/$1"
else
# check if the PREFIX directory exists and prepend DESTDIR if not
if [ ! -d "${PREFIX}" ]
then
PREFIX="${DESTDIR}$1"
fi
fi

shift
Expand Down
4 changes: 2 additions & 2 deletions docs/user_guide/source/setting_up/source/cmake.rst
Original file line number Diff line number Diff line change
Expand Up @@ -103,8 +103,8 @@ In addition to the ``ADIOS2_USE_Feature`` options, the following options are als
CMake VAR Options Values Description |
==================================== =============================================== ===============================
``BUILD_SHARED_LIBS`` **ON**/OFF Build shared libraries.
``ADIOS2_BUILD_EXAMPLES`` **ON**/OFF Build examples.
``BUILD_TESTING`` **ON**/OFF Build test code.
``ADIOS2_BUILD_EXAMPLES`` ON/**OFF** Build examples.
``BUILD_TESTING`` ON/**OFF** Build test code.
``CMAKE_INSTALL_PREFIX`` /path/to/install (``/usr/local``) Installation location.
``CMAKE_BUILD_TYPE`` Debug/**Release**/RelWithDebInfo/MinSizeRel Compiler optimization levels.
==================================== =============================================== ===============================
Expand Down
22 changes: 2 additions & 20 deletions examples/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5,24 +5,6 @@

add_subdirectory(basics)
add_subdirectory(hello)
add_subdirectory(query)
add_subdirectory(useCases)
add_subdirectory(inlineMWE)
add_subdirectory(plugins)
add_subdirectory(fides)

if(ADIOS2_HAVE_MPI)
add_subdirectory(heatTransfer)
endif()


if(ADIOS2_HAVE_CUDA OR ADIOS2_HAVE_Kokkos_CUDA)
add_subdirectory(cuda)
endif()


if(ADIOS2_HAVE_MPI AND ADIOS2_HAVE_HDF5)
if(HDF5_VERSION VERSION_GREATER_EQUAL 1.14)
add_subdirectory(h5subfile)
endif()
endif()
add_subdirectory(simulations)
add_subdirectory(useCases)
16 changes: 16 additions & 0 deletions examples/ReadMe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# ADIOS2 Examples

This directory contains examples of how to use ADIOS2 in different scenarios.
The examples are written mostly in C++, but also C, Fortran, Python, and Julia.

They can be found in the following subdirectories, and they should be explored in the order that they are listed:

1. [hello](hello): The _hello_ examples are meant to introduce you to ADIOS2's IO capabilities and engines.
2. [basics](basics): The _basics_ examples are meant to introduce you to basic concepts of ADIOS2, such as
global/joined/local arrays, values, and variables shapes.
3. [useCases](useCases): The _useCases_ examples are meant to demonstrate how to use ADIOS2 in different scenarios,
such as in situ visualization, and fides schema.
4. [plugins](plugins): The _plugins_ examples are meant to introduce you to the plugin capabilities of ADIOS2, such as
how to develop your own engine or operators.
5. [simulations](simulations): The _simulations_ examples are meant to demonstrate how to integrate ADIOS2 within your
simulation code to read, write and/or stream your simulation data.
7 changes: 6 additions & 1 deletion examples/basics/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,12 @@
# accompanying file Copyright.txt for details.
#------------------------------------------------------------------------------#

add_subdirectory(globalArray)
if(ADIOS2_HAVE_MPI)
add_subdirectory(globalArray1D)
endif()
add_subdirectory(globalArrayND)
add_subdirectory(joinedArray)
add_subdirectory(localArray)
add_subdirectory(queryWorker)
add_subdirectory(values)
add_subdirectory(variablesShapes)
29 changes: 29 additions & 0 deletions examples/basics/ReadMe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
## ADIOS2 basics examples

The _basics_ examples are meant to introduce you to basic concepts of ADIOS2, such as
global/joined/local arrays, values, and variables shapes.

They can be found in the following subdirectories, and they should be explored in the order that they are listed:

1. [globalArray1D](globalArray1D): The _globalArray1D_ example demonstrates how to read and write an
1-D global array with constant dimensions over time from multiple processors using ADIOS2's BP engine.
* Languages: C, Fortran
2. [globalArrayND](globalArrayND): The _globalArrayND_ example demonstrates how to write an N-D global array with
constant dimensions over time from multiple processors using ADIOS2's BP engine.
* Languages: C++
3. [localArray](localArray): The _localArray_ example demonstrates how to write and read a local array per processor
with the same name from multiple processors using ADIOS2's BP engine.
* Languages: C++
4. [joinedArray](joinedArray): The _joinedArray_ example demonstrates how to write local array that is different only in
one dimension so that it can be joined into a global array with the same name from multiple processors at read time
using ADIOS2's ADIOS2's BP engine.
* Languages: C++
5. [values](values): The _values_ example demonstrates how to write and read a multiple types of variables with a single
value, such as global constant, global value, local constant, and local value using ADIOS2's BP engine.
* Languages: C++, Fortran
6. [variablesShapes](variablesShapes): The _variablesShapes_ example demonstrates how to write supported variables
shapes using stepping and ADIOS2's BP engine.
* Languages: C++, C++ using high-level API
7. [queryWorker](queryWorker): The _queryWorker_ example demonstrates how to read variables using ADIOS2's BP engine
and perform queries on the read data and streams the results.
* Languages: C++
12 changes: 0 additions & 12 deletions examples/basics/globalArray/CMakeLists.txt

This file was deleted.

Loading
Loading