Skip to content

Commit

Permalink
Network message v2 (#1089)
Browse files Browse the repository at this point in the history
* Environment Directed Graph
Directed graphs can be defined on the Host and accessed and traversed on device (edges in/out of a given vertex).
Vertices and Edges are referred to by vertex index on device, however a function exists to convert between vertex ID and vertex index.
This approach is the compromise to minimise indirection whilst retaining usability.
Graphs can be exported/imported to a JSON format compatible with d3.js via a dedicated HostAPI function.
Tests added for all current functionality (C/Python/AgentPython).
Also fixed a few GLM_ON build issues, likely a side effect of refactors prior to release.
FLAMEGPU_ENABLE_ADVANCED_API CMake option has been added, this currently exposes a few additional (undocumented) HostAPI features such as a function that returns the CUDA stream handle.
  • Loading branch information
Robadob authored Dec 15, 2023
1 parent e8adb97 commit 24b167f
Show file tree
Hide file tree
Showing 54 changed files with 8,352 additions and 101 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
*.json
*.gv
*.bin
*test_rtcfunc_file

# GoogleTest directories (handled by CMake)
googletest-build/
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,7 @@ cmake --build . --target all
| `FLAMEGPU_VERBOSE_PTXAS` | `ON`/`OFF` | Enable verbose PTXAS output during compilation. Default `OFF`. |
| `FLAMEGPU_CURAND_ENGINE` | `XORWOW` / `PHILOX` / `MRG` | Select the CUDA random engine. Default `XORWOW` |
| `FLAMEGPU_ENABLE_GLM` | `ON`/`OFF` | Experimental feature for GLM type support within models. Default `OFF`. |
| `FLAMEGPU_ENABLE_ADVANCED_API` | `ON`/`OFF` | Enable advanced API functionality (C++ only), providing access to internal sim components for high-performance extensions. No stability guarantees are provided around this interface and the returned objects. Documentation is limited to that found in the source. Default `OFF`. |
| `FLAMEGPU_SHARE_USAGE_STATISTICS` | `ON`/`OFF` | Share usage statistics ([telemetry](https://docs.flamegpu.com/guide/telemetry)) to support evidencing usage/impact of the software. Default `ON`. |
| `FLAMEGPU_TELEMETRY_SUPPRESS_NOTICE` | `ON`/`OFF` | Suppress notice encouraging telemetry to be enabled, which is emitted once per binary execution if telemetry is disabled. Defaults to `OFF`, or the value of a system environment variable of the same name. |
| `FLAMEGPU_TELEMETRY_TEST_MODE` | `ON`/`OFF` | Submit telemetry values to the test mode of TelemetryDeck. Intended for use during development of FLAMEGPU rather than use. Defaults to `OFF`, or the value of a system environment variable of the same name.|
Expand Down
21 changes: 21 additions & 0 deletions include/flamegpu/defines.h
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,27 @@ typedef unsigned int id_t;
* Internal variable name used for IDs
*/
constexpr const char* ID_VARIABLE_NAME = "_id";
/**
* Internal variable name used for source-dest pairs
* @note These are always stored in [dest, source] order, but shown to the user in [source, dest] order. This enables 2D sorting
*/
constexpr const char* GRAPH_SOURCE_DEST_VARIABLE_NAME = "_source_dest";
/**
* Internal variable name used to the pointer to the PBM of edges leaving each vertex
*/
constexpr const char* GRAPH_VERTEX_PBM_VARIABLE_NAME = "_pbm";
/**
* Internal variable name used to the pointer to the (inverted) PBM of edges joining each vertex
*/
constexpr const char* GRAPH_VERTEX_IPBM_VARIABLE_NAME = "_ipbm";
/**
* Edges are not sorted in order of the IPBM, instead the IPBM points to indices in this list of edge indexes
*/
constexpr const char* GRAPH_VERTEX_IPBM_EDGES_VARIABLE_NAME = "_ipbm_edges";
/**
* Accessing an ID within this buffer will return the index of the vertex
**/
constexpr const char* GRAPH_VERTEX_INDEX_MAP_VARIABLE_NAME = "_index_map";
/**
* Internal value used when IDs have not be set
* If this value is changed, things may break
Expand Down
19 changes: 18 additions & 1 deletion include/flamegpu/exception/FLAMEGPUException.h
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ DERIVED_FLAMEGPUException(InvalidInputFile, "Invalid Input File");
/**
* Defines a type of object to be thrown as exception.
* It reports errors that are due to invalid agent variable type.
* This could happen when retriving or setting a variable of differet type.
* This could happen when retrieving or setting a variable of different type.
*/
DERIVED_FLAMEGPUException(InvalidVarType, "Bad variable type in agent instance set/get variable");

Expand Down Expand Up @@ -315,6 +315,11 @@ DERIVED_FLAMEGPUException(DuplicateEnvProperty, "Environment property of same na
* It reports errors that are due invalid environment property names
*/
DERIVED_FLAMEGPUException(InvalidEnvProperty, "Environment property of name does not exist");
/**
* Defines a type of object to be thrown as exception.
* It reports errors that are due invalid environment graphs
*/
DERIVED_FLAMEGPUException(InvalidEnvGraph, "Environment graph of name does not exist");

/**
* Defines a type of object to be thrown as exception.
Expand Down Expand Up @@ -415,10 +420,22 @@ DERIVED_FLAMEGPUException(InvalidDependencyGraph, "Agent function dependency gra
* This should not occur if the shared ID matches ID_NOT_SET
*/
DERIVED_FLAMEGPUException(AgentIDCollision, "Multiple agents of same type share an ID");
DERIVED_FLAMEGPUException(IDCollision, "Generic ID collision");
DERIVED_FLAMEGPUException(IDOutOfBounds, "Generic ID range failure");
DERIVED_FLAMEGPUException(IDNotSet, "Generic ID has not been set failure");
DERIVED_FLAMEGPUException(InvalidID, "Generic ID provided is not valid failure");
/**
* Defines an error when runs fail during an ensemble's execution
*/
DERIVED_FLAMEGPUException(EnsembleError, "One of more runs failed during the ensemble's execution");
/**
* Defines an error when an invalid property for a graph is specified
*/
DERIVED_FLAMEGPUException(InvalidGraphProperty, "Invalid graph property specified");
/**
* Defines an error when an invalid name for a graph is specified
*/
DERIVED_FLAMEGPUException(InvalidGraphName, "Invalid graph name specified");

} // namespace exception
} // namespace flamegpu
Expand Down
1 change: 1 addition & 0 deletions include/flamegpu/flamegpu.h
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
#include "flamegpu/model/SubModelDescription.h"
#include "flamegpu/model/SubAgentDescription.h"
#include "flamegpu/model/SubEnvironmentDescription.h"
#include "flamegpu/model/EnvironmentDirectedGraphDescription.cuh"
#include "flamegpu/simulation/AgentVector.h"
#include "flamegpu/runtime/agent/AgentInstance.h"
#include "flamegpu/simulation/CUDASimulation.h"
Expand Down
33 changes: 33 additions & 0 deletions include/flamegpu/io/JSONGraphReader.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#ifndef INCLUDE_FLAMEGPU_IO_JSONGRAPHREADER_H_
#define INCLUDE_FLAMEGPU_IO_JSONGRAPHREADER_H_

#include <memory>
#include <string>

#include "flamegpu/detail/cuda.cuh"

namespace flamegpu {
namespace detail {
class CUDAEnvironmentDirectedGraphBuffers;
} // namespace detail
namespace io {

class JSONGraphReader {
public:
/**
* Imports the provided graph from the json "adjacency like" format, supported by NetworkX/d3.js
*
* @param filepath The path to load the graph from
* @param directed_graph The graph buffers to import into
* @param stream CUDA stream (required by directed_graph for synchronising device buffers)
*
* @throws exception::InvalidFilePath If the file cannot be opened for reading
* @throws exception::RapidJSONError If JSON fails for any reason (e.g. structure does not match expectations)
*/
static void loadAdjacencyLike(const std::string &filepath, const std::shared_ptr<detail::CUDAEnvironmentDirectedGraphBuffers> &directed_graph, cudaStream_t stream);
};

} // namespace io
} // namespace flamegpu

#endif // INCLUDE_FLAMEGPU_IO_JSONGRAPHREADER_H_
34 changes: 34 additions & 0 deletions include/flamegpu/io/JSONGraphWriter.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#ifndef INCLUDE_FLAMEGPU_IO_JSONGRAPHWRITER_H_
#define INCLUDE_FLAMEGPU_IO_JSONGRAPHWRITER_H_

#include <memory>
#include <string>

#include "flamegpu/detail/cuda.cuh"

namespace flamegpu {
namespace detail {
class CUDAEnvironmentDirectedGraphBuffers;
} // namespace detail
namespace io {

class JSONGraphWriter {
public:
/**
* Exports the provided graph in the json "adjacency like" format, supported by NetworkX/d3.js
*
* @param filepath The path to save the graph to
* @param directed_graph The graph buffers to export
* @param stream CUDA stream (required by directed_graph for synchronising device buffers)
* @param pretty_print Whether JSON should be human readable (vs minified)
*
* @throws exception::InvalidFilePath If the file cannot be opened for writing
* @throws exception::RapidJSONError If conversion to JSON fails for any reason
*/
static void saveAdjacencyLike(const std::string &filepath, const std::shared_ptr<const detail::CUDAEnvironmentDirectedGraphBuffers> &directed_graph, cudaStream_t stream, bool pretty_print = true);
};

} // namespace io
} // namespace flamegpu

#endif // INCLUDE_FLAMEGPU_IO_JSONGRAPHWRITER_H_
5 changes: 5 additions & 0 deletions include/flamegpu/model/EnvironmentData.h
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@

#include "flamegpu/detail/Any.h"
#include "flamegpu/model/ModelData.h"
#include "flamegpu/model/EnvironmentDirectedGraphData.cuh"

namespace flamegpu {

Expand Down Expand Up @@ -108,6 +109,10 @@ struct EnvironmentData {
* Main storage of all macroproperties
*/
std::unordered_map<std::string, MacroPropData> macro_properties{};
/**
* Main storage of all directed graphs
*/
std::unordered_map<std::string, std::shared_ptr<EnvironmentDirectedGraphData>> directed_graphs{};
/**
* Equality operator, checks whether EnvironmentData hierarchies are functionally the same
* @returns True when environments are the same
Expand Down
28 changes: 28 additions & 0 deletions include/flamegpu/model/EnvironmentDescription.h
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@

namespace flamegpu {

class CEnvironmentDirectedGraphDescription;
class EnvironmentDirectedGraphDescription;

class CEnvironmentDescription {
/**
* Data store class for this description, constructs instances of this class
Expand Down Expand Up @@ -101,6 +104,14 @@ class CEnvironmentDescription {
* @throws exception::InvalidEnvProperty If a property of the name does not exist
*/
bool getConst(const std::string &name) const;
/**
* Returns an existing directed graph
*
* Directed graphs are environmental graph structures which can hold properties on their vertices and edges
* Agent's can traverse the graph structure and assign messages to the graph using related communication strategies
* @param graph_name The name used to refer to the graph throughout the model
*/
CEnvironmentDirectedGraphDescription getDirectedGraph(const std::string& graph_name) const;

protected:
/**
Expand Down Expand Up @@ -135,6 +146,7 @@ class EnvironmentDescription : public CEnvironmentDescription {
*/
EnvironmentDescription& operator=(const EnvironmentDescription& other_env) = default;
EnvironmentDescription& operator=(EnvironmentDescription&& other_env) = default;

/**
* Adds a new environment property
* @param name name used for accessing the property
Expand Down Expand Up @@ -200,6 +212,22 @@ class EnvironmentDescription : public CEnvironmentDescription {
template<typename T>
void newMacroProperty_swig(const std::string& name, flamegpu::size_type I = 1, flamegpu::size_type J = 1, flamegpu::size_type K = 1, flamegpu::size_type W = 1);
#endif
/**
* Define a new directed graph
*
* Directed graphs are environmental graph structures which can hold properties on their vertices and edges
* Agent's can traverse the graph structure and assign messages to the graph using related communication strategies
* @param graph_name The name used to refer to the graph throughout the model
*/
EnvironmentDirectedGraphDescription newDirectedGraph(const std::string &graph_name);
/**
* Returns an existing directed graph
*
* Directed graphs are environmental graph structures which can hold properties on their vertices and edges
* Agent's can traverse the graph structure and assign messages to the graph using related communication strategies
* @param graph_name The name used to refer to the graph throughout the model
*/
EnvironmentDirectedGraphDescription getDirectedGraph(const std::string& graph_name);
/**
* Sets an environment property
* @param name name used for accessing the property
Expand Down
73 changes: 73 additions & 0 deletions include/flamegpu/model/EnvironmentDirectedGraphData.cuh
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
#ifndef INCLUDE_FLAMEGPU_MODEL_ENVIRONMENTDIRECTEDGRAPHDATA_CUH_
#define INCLUDE_FLAMEGPU_MODEL_ENVIRONMENTDIRECTEDGRAPHDATA_CUH_

#include <string>
#include <memory>

#include "flamegpu/model/Variable.h"
#include "flamegpu/model/ModelData.h"

namespace flamegpu {

class EnvironmentDescription;

/**
* This is the internal data store for EnvironmentDirectedGraphDescription
* Users should only access that data stored within via an instance of EnvironmentDirectedGraphDescription
*/
struct EnvironmentDirectedGraphData {
friend class EnvironmentDescription;
friend struct EnvironmentData;
/**
* Parent model
*/
std::weak_ptr<const ModelData> model;
/**
* Holds all of the graphs's vertex property definitions
*/
VariableMap vertexProperties{};
/**
* Holds all of the graphs's edge property definitions
*/
VariableMap edgeProperties{};
/**
* Name of the graph, used to refer to the graph in many functions
*/
std::string name;
/**
* Equality operator, checks whether EnvironmentDirectedGraphData hierarchies are functionally the same
* @param rhs Right hand side
* @returns True when directed graphs are the same
* @note Instead compare pointers if you wish to check that they are the same instance
*/
bool operator==(const EnvironmentDirectedGraphData& rhs) const;
/**
* Equality operator, checks whether EnvironmentDirectedGraphData hierarchies are functionally different
* @param rhs Right hand side
* @returns True when directed graphs are not the same
* @note Instead compare pointers if you wish to check that they are not the same instance
*/
bool operator!=(const EnvironmentDirectedGraphData& rhs) const;
/**
* Default copy constructor should not be used
*/
explicit EnvironmentDirectedGraphData(const EnvironmentDirectedGraphData& other) = delete;

protected:
/**
* Copy constructor
* This is unsafe, should only be used internally, use clone() instead
* @param model The parent model of the graph
* @param other Other EnvironmentDirectedGraphData to copy data from
*/
explicit EnvironmentDirectedGraphData(const std::shared_ptr<const ModelData>& model, const EnvironmentDirectedGraphData& other);
/**
* Normal constructor, only to be called by ModelDescription
* @param parent The environment which owns the graph
* @param graph_name Name of the graph
*/
explicit EnvironmentDirectedGraphData(const std::shared_ptr<const EnvironmentData>& parent, const std::string& graph_name);
};
} // namespace flamegpu

#endif // INCLUDE_FLAMEGPU_MODEL_ENVIRONMENTDIRECTEDGRAPHDATA_CUH_
Loading

0 comments on commit 24b167f

Please sign in to comment.