Skip to content

Commit

Permalink
Improved tests and interfaces for benchmarks of LDPC code performance (
Browse files Browse the repository at this point in the history
…#9)

* Adds CmdParser v1.1.0 for parsing command line arguments to the FER simulation benchmark and the  average minimum rate simulation.

Note: original PR intended automatic benchmarks and reports. Development on automatic benchmarks and reports will continue in future PRs.
  • Loading branch information
XQP-Munich authored Oct 19, 2021
1 parent eb4fc12 commit 3a0dfb5
Show file tree
Hide file tree
Showing 21 changed files with 13,564 additions and 419 deletions.
62 changes: 47 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
[![License](https://img.shields.io/github/license/XQP-Munich/LDPC4QKD)](./LICENSE)
# LDPC4QKD: LDPC codes for rate adaptive distributed source coding

Note: This repository is still missing some benchmarks and documentation, which will be added very soon.
Note: This repository is still missing some documentation, which will be added very soon.

## Overview

Expand Down Expand Up @@ -47,36 +47,68 @@ To build the C++ executables (except for unit tests) using CMake (Julia not requ
All executables will be built inside the `build` folder.

For a demo of how to use the C++ header-only library, see the `examples` directory, which shows a basic example ("demo_error_correction") of how to use the C++ header containing the decoder. This example is built by CMake (executable `build/examples/demo_error_correction`. Note: the executable produces no output; look at the C++ source code to see how the header can be used).
For a demo of how to use the C++ header-only library, see the `examples` directory, which shows a basic example ("demo_error_correction") of how to use the C++ header containing the decoder.
This example is built by CMake (executable `build/examples/demo_error_correction`.
Note: the executable produces no output; look at the C++ source code to see how the header can be used).

## How to contribute
This repository is actively maintained. Issues and pull requests will be responded to and processed.
This repository is actively maintained.
Issues and pull requests will be responded to and processed.

Let us know if you're having problems with the provided materials, wish to contribute new ideas or need modifications of our work for your application.

## List of contents

### Data files
- A number of LDPC codes (a list and the actual LDPC matrices are in the folder `codes`). Their parity check matrices are stored in a custom file format (called `CSCMAT`).
- A number of LDPC codes (a list and the actual LDPC matrices are in the folder `codes`).
Their parity check matrices are stored in a custom file format (called `CSCMAT`).
- Simulations results done using [AFF3CT](https://github.com/aff3ct/aff3ct), showing FER of the LDPC matrices at various channel parameters.
- Simulation results done using the decoder in this repository, showing FER of LDPC matrices, their rate adapted versions, and average rate under rate adaption (Work in progress!).
- For each LDPC matrix, a specification of rate adaption. This is a list of pairs of row indices of the matrix that are combined (added mod 2) in each rate adaption step.
- For each LDPC matrix, a specification of rate adaption.
This is a list of pairs of row indices of the matrix that are combined (added mod 2) in each rate adaption step.

### Julia code
- Contained in the folder `codes`.
- Enables loading the LDPC matrices from `CSCMAT` files
- Enables loading the LDPC matrices from `CSCMAT` files (using our custom Julia library ``)
- Enables saving the compressed sparse column (CSC) representation and also exporting to other standard formats, such as `alist`.

### C++ code
- Basic LDPC decoder using belief propagation (BP). Contained in a single header file (`src/rate_adaptive_code.hpp`) and easy to use. Can perform syndrome computation as well. The LDPC code can be embedded into the executable or loaded from a file at program runtime.
- Utility functions for reading LDPC matrices (from `.cscmat` files) and rate adaption (from `.csv` files). These functions are contained in `src/read_cscmat_format.hpp`. Note that these functions require the fully explicit binary form of the LDPC matrix. The LDPC codes given inside `codes/ldpc` use an even more memory-efficient storage
- For applications that only require syndrome computation but no decoding, we provide a separate implementation for multiplication of a sparse binary matrix and a dense binary vector (LDPC syndrome computation). This is also contained in a single header file (`src/encoder.hpp`). This is a very specific application, which you probably don't care about initially.
- A LDPC matrix can be stored within the executable. It is included as a header file defining constant data (for example `tests/fortest_autogen_ldpc_matrix_csc.hpp`). Julia code (see folder `codes`) is provided to generate such a C++ header file for any LDPC matrix.
- Basic LDPC decoder using belief propagation (BP). Contained in a single header file (`src/rate_adaptive_code.hpp`) and easy to use. Can perform syndrome computation as well.
The LDPC code can be embedded into the executable or loaded from a file at program runtime.
- Utility functions for reading LDPC matrices (from `.cscmat` files) and rate adaption (from `.csv` files). These functions are contained in `src/read_cscmat_format.hpp`.
Note that these functions require the fully explicit binary form of the LDPC matrix. The LDPC codes given inside `codes/ldpc` use an even more memory-efficient storage
- For applications that only require syndrome computation but no decoding, we provide a separate implementation for multiplication of a sparse binary matrix and a dense binary vector (LDPC syndrome computation). This is also contained in a single header file (`src/encoder.hpp`).
This is a very specific application, which you probably don't care about initially.
- A LDPC matrix can be stored within the executable.
It is included as a header file defining constant data (for example `tests/fortest_autogen_ldpc_matrix_csc.hpp`).
Julia code (see folder `codes`) is provided to generate such a C++ header file for any LDPC matrix.


## Planned features and improvements
- Code to automatically generate reports on quality of all LDPC codes and rate adapted performance (work in progress)
- More and better LDPC codes with more sizes and rates
- The BP implementation is not state-of-the-art or heavily optimized. We may improve it in the future.
- Using compressed sparse row format may have advantages over the current compressed sparse column format. This needs to be tested.
- the quasi-cyclic structure of LDPC matrices can be exploited to store the matrices more efficiently. For the encoder, this may be combinable with storing individual bits as integers rather than booleans, again saving some memory. The encoding in this case would consist of adding bit-shifted integers.

If you need some feature for your applications, let us know, e.g. by creating an issue.

- [ ] Reports and benchmarks on LDPC code quality and decoding performance
+ [x] Simulation programs with command line interfaces
+ [x] Frame error rate simulations for rate adapted codes (including special case of no rate adaption)
+ [x] Critical rate (codeword-averaged minimum leak rate for successful decoding) computation for rate adapted codes
+ [ ] Automatic running of simulations
+ [ ] Automatic reports with plots
+ [ ] Simulations on multiple threads and/or multiprocessing
- [ ] LDPC codes
+ [x] 3 LDPC codes each (different block sizes) for leak rates 1/2 and 1/3
+ [ ] More sizes, more rates
+ [ ] Very high leak rate codes (for very low signal-to-noise ratios)
- [ ] Decoding and decoding algorithms
+ [x] Basic belief propagation (BP) decoder for Slepian-Wolf setting
+ [ ] Decoder performance improvements (look at [AFF3CT](https://github.com/aff3ct/aff3ct) for inspiration), plausibly achieve 2x runtime speedup at same decoding accuracy
+ [ ] Decoding on GPU
+ [x] Encoder that can be used separately from encoder (e.g. for embedded applications)
+ [ ] Using compressed sparse row format may have advantages over the current compressed sparse column (CSC) format. This needs to be tested.
+ [ ] Save memory by storing QC-exponents of structured codes, rather than CSC storage
+ [ ] Encoder speedup may be possible using QC-exponents by densely packed bits and using bit-shifts (not a priority, as syndrome computation, i.e., sparse matrix-vector multiplication, is fast anyway)


## Attributions

- Some of the simulation/benchmarking programs use [CmdParser](https://github.com/FlorianRappl/CmdParser), a simple command line argument parser (MIT license, the sources are included in this repository at `benchmarks/CmdParser-1.1.0`).
17 changes: 1 addition & 16 deletions benchmarks/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -55,21 +55,6 @@ target_include_directories(benchmark_ra
PRIVATE
)

# ------------------------------------------------------------------------------------------ Frame error rate simulation
add_executable(frame_error_rate_simulation main_frame_error_rate_simulation.cpp
code_simulation_helpers.hpp)

target_compile_features(frame_error_rate_simulation PUBLIC cxx_std_17)

target_link_libraries(frame_error_rate_simulation
PRIVATE
LDPC4QKD::LDPC4QKD
)

target_include_directories(frame_error_rate_simulation
PRIVATE
)

# ---------------------------------------------------------------------------------- Rate adapted performance simulation
add_executable(rate_adapted_simulation main_rate_adapted_simulation.cpp
code_simulation_helpers.hpp)
Expand All @@ -85,7 +70,7 @@ target_include_directories(rate_adapted_simulation
PRIVATE
)

# ------------------------------------------------------------------------------------------ Rate adapted fer simulation
# ------------------------------------------------------------- frame error rate (FER) simulation (allows rate adaption)
add_executable(rate_adapted_fer main_rate_adapted_fer.cpp
code_simulation_helpers.hpp)

Expand Down
32 changes: 32 additions & 0 deletions benchmarks/CmdParser-1.1.0/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Compiled Object files
*.slo
*.lo
*.o
*.obj

# Precompiled Headers
*.gch
*.pch

# Compiled Dynamic libraries
*.so
*.dylib
*.dll

# Fortran module files
*.mod

# Compiled Static libraries
*.lai
*.la
*.a
*.lib

# Executables
*.exe
*.out
*.app

# Biicode
/bin
/bii
14 changes: 14 additions & 0 deletions benchmarks/CmdParser-1.1.0/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
cmake_minimum_required(VERSION 3.9)
project(cmdparser)

enable_testing()

set(CMAKE_CXX_STANDARD 11)
set(CMAKE_CXX_EXTENSIONS OFF)

add_subdirectory(cmdparser.Test EXCLUDE_FROM_ALL)
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} --verbose)
add_test(cmdparserTest cmdparser.Test/cmdparserTest)
add_dependencies(check cmdparserTest)


21 changes: 21 additions & 0 deletions benchmarks/CmdParser-1.1.0/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
The MIT License (MIT)

Copyright (c) 2015 - 2016 Florian Rappl

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
120 changes: 120 additions & 0 deletions benchmarks/CmdParser-1.1.0/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
# Simple C++ command line parser

This project supplies a simple, single-header, command-line parser. It is very lightweight and relies on templates. The easiest way is to use it by adding it to your source code. The parser requires C++11 and works fine on gcc (v4.8.2 or later, some earlier versions should work as well), icc (v14 or later), clang and msvc (v18 or later).

## Using the parser

Using the parser is straight forward. Only include the header (*cmdparser.h*) in your application source file, most likely the one that contains the `main` method. Pass the command line arguments to the parser:

```cpp
int main(int argc, char** argv) {
cli::Parser parser(argc, argv);
/* ... */
}
```
In the following two sections we'll have a look at setting up the parser and using it.
### Setup
Setting up the parser works using the following methods:
* `set_optional<T>()`, to include an optional argument
* `set_required<T>()`, to include a required argument
The third parameter for creating an optional argument is the default value. This value is used if nothing is provided by the user. Otherwise the optional and required methods are pretty similar:
1. The shorthand (if the user uses a single slash) string
2. The longhand (if the user uses two slashes) string
3. The optional description.
The third parameter is the fourth parameter for optional arguments.
Let's look at an example:
```cpp
void configure_parser(cli::Parser& parser) {
parser.set_optional<std::string>("o", "output", "data", "Strings are naturally included.");
parser.set_optional<int>("n", "number", 8, "Integers in all forms, e.g., unsigned int, long long, ..., are possible. Hexadecimal and Ocatl numbers parsed as well");
parser.set_optional<cli::NumericalBase<int, 10>>("t", "temp", 0, "integer parsing restricted only to numerical base 10");
parser.set_optional<double>("b", "beta", 11.0, "Also floating point values are possible.");
parser.set_optional<bool>("a", "all", false, "Boolean arguments are simply switched when encountered, i.e. false to true if provided.");
parser.set_required<std::vector<short>>("v", "values", "By using a vector it is possible to receive a multitude of inputs.");
}
```

Usually it makes sense to pack the Parser's setup in a function. But of course this is not required. The shorthand is not limited to a single character. It could also be the same as the longhand alternative.

### Getting values

Getting values is possible via the `get` method. This is also a template. We need to specify the type of argument. This has to be the same type as defined earlier. It also has to be a valid argument (shorthand) name. At the moment only shorthands are considered here. For instance we could do the following:

```cpp
//auto will be int
auto number = parser.get<int>("n");

//auto will be int, note specification of numerical base same as when set during parser configuration
auto number = parser.get<cli::NumericalBase<int, 10> >("t");

//auto will be std::string
auto output = parser.get<std::string>("o");

//auto will be bool
auto all = parser.get<bool>("a");

//auto will be std::vector<short>
auto values = parser.get<std::vector<short>>("v");
```

However, before we can access these values we also need to check if the provided user input was valid. On construction the `Parser` does not examine the input. The parser waits for setup and a potential call to the `run` method. The `run` method runs a boolean value to indicate if the provided command line arguments match the requirements.

What we usually want is something like:

```cpp
void parse_and_exit(cli::Parser& parser) {
if (parser.parse() == false) {
exit(1);
}
}
```
Writing this function seems to be redundant. Hence the parser includes it already:
```cpp
parser.run_and_exit_if_error();
```

The only difference is that the `run_and_exit_if_error` method does not provide overloads for passing custom output and error streams. The `parse` method has overloads to support such scenarios. By default `std::cout` is used the regular output, e.g., the integrated help. Also `std::cerr` is used for displaying error messages.

## Integrated help

The parser comes with a pre-defined command that has the shorthand `-h` and the longhand `--help`. This is the integrated help, which appears if only a single command line argument is given, which happens to be either the shorthand or longhand form.

Finally our `main` method may look as follows:

```cpp
int main(int argc, char** argv) {
cli::Parser parser(argc, argv);
configure_parser(parser);
parser.run_and_exit_if_error();
/* ... */
}
```
This passes the arguments to the parser, configures the parser and checks for potential errors. In case of any errors the program is exited immediately.
## Contributions
This is not a huge project and the file should remain a small, single-header command-line parser, which may be useful for small to medium projects. Nevertheless, if you find any bugs, add small, yet useful, new features or improve the cross-compiler compatibility, then contributions are more than welcome.
## License
The application is licensed under the MIT License (MIT).
Copyright (c) 2015 - 2016 Florian Rappl
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
5 changes: 5 additions & 0 deletions benchmarks/CmdParser-1.1.0/cmdparser.Test/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
set(SOURCE_FILES TestMain.cpp catch.hpp tests.cpp)
add_executable(cmdparserTest ${SOURCE_FILES})
IF(APPLE)
TARGET_COMPILE_OPTIONS(cmdparserTest PUBLIC INTERFACE "-stdlib=libc++")
ENDIF(APPLE)
8 changes: 8 additions & 0 deletions benchmarks/CmdParser-1.1.0/cmdparser.Test/TestMain.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
//
// Created by marcel on 12/21/17.
//

#define CATCH_CONFIG_MAIN

#include "catch.hpp"

Loading

0 comments on commit 3a0dfb5

Please sign in to comment.