Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cachegrand Benchmark Generator #244

Merged
merged 33 commits into from
Oct 4, 2022
Merged
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
a095ed9
Added base command analyzer
Valkyrie00 Sep 10, 2022
d028f36
Simple refinement of service
Valkyrie00 Sep 10, 2022
d233458
Simple refinement of service
Valkyrie00 Sep 10, 2022
5c6546c
integrated pcre2 as dependecy
Valkyrie00 Sep 23, 2022
20ed190
Removed single match
Valkyrie00 Sep 23, 2022
8aea64b
Fixed recursive extractor
Valkyrie00 Sep 25, 2022
76eb2dd
moved some logics under specific namespace
Valkyrie00 Sep 25, 2022
ab60b34
Improved test lists
Valkyrie00 Sep 25, 2022
5171749
Base builder
Valkyrie00 Sep 25, 2022
5973252
Added function to append command to sections
Valkyrie00 Sep 26, 2022
ba05f79
Managed multi-tests building
Valkyrie00 Sep 27, 2022
7dceead
Fix CS
Valkyrie00 Sep 27, 2022
a535eae
Append namespace in function
Valkyrie00 Sep 27, 2022
b96a448
Moved some logics from anlayzer to main
Valkyrie00 Sep 27, 2022
0522cc7
Added jsonc dependecy
Valkyrie00 Sep 27, 2022
fbb46ca
Moved and renamed benchmark generator
Valkyrie00 Sep 27, 2022
dc9e3a5
Renamed some pointers
Valkyrie00 Sep 28, 2022
d17568f
Added json output
Valkyrie00 Sep 28, 2022
37e2e59
Added test name
Valkyrie00 Sep 28, 2022
d5a32bc
fixed parsing command
Valkyrie00 Sep 28, 2022
d5f1ae3
fix calculated nested section padding
Valkyrie00 Sep 28, 2022
46b0c1c
Implemented saving output to file
Valkyrie00 Sep 28, 2022
c7e02b2
Implemented arguments reader
Valkyrie00 Sep 28, 2022
111f804
Imported auto-generated test lists
Valkyrie00 Sep 29, 2022
79e9332
Fixed header definitions
Valkyrie00 Sep 29, 2022
be6cd2b
Merge branch 'main' into tool-test-analyzer
Valkyrie00 Sep 30, 2022
fce4e9c
Improved quality of memory allocations
Valkyrie00 Sep 30, 2022
ad40acb
Updated build documentation and Dockerfile
Valkyrie00 Sep 30, 2022
5f5cbf9
Updated build documentation and Dockerfile
Valkyrie00 Oct 2, 2022
fb52f65
Fixed some errors in benchmark building flow
Valkyrie00 Oct 2, 2022
7641207
Merge branch 'main' into tool-test-analyzer
danielealbano Oct 4, 2022
89dcf71
Fixed CI build and test
Valkyrie00 Oct 4, 2022
b6d65c4
Fixed lgtm packages
Valkyrie00 Oct 4, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build_and_test.yml
Original file line number Diff line number Diff line change
@@ -48,7 +48,7 @@ jobs:
fetch-depth: 2

- name: Install required dependencies
run: sudo apt-get update && sudo apt-get install -y build-essential cmake pkg-config git libnuma1 libnuma-dev libcurl4-openssl-dev libcurl4 libyaml-0-2 libyaml-dev libmbedtls-dev valgrind
run: sudo apt-get update && sudo apt-get install -y build-essential cmake pkg-config git libnuma1 libnuma-dev libcurl4-openssl-dev libcurl4 libyaml-0-2 libyaml-dev libmbedtls-dev libpcre2-8-0 libpcre2-dev libjson-c-dev valgrind

- name: Create Build Environment
run: cmake -E make_directory ${{github.workspace}}/build
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -2,4 +2,4 @@
cmake-build-*/*

/etc/cachegrand.yaml
/src/cmake_config.c
/src/cmake_config.c
3 changes: 3 additions & 0 deletions .lgtm.yml
Original file line number Diff line number Diff line change
@@ -16,6 +16,9 @@ extraction:
- "libyaml-dev"
- "libcurl4-openssl-dev"
- "libcurl4"
- "libpcre2-8-0"
- "libpcre2-dev"
- "libjson-c-dev"
- "lcov"
- "gcovr"
configure:
1 change: 1 addition & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -10,6 +10,7 @@ project(cachegrand LANGUAGES C CXX ASM)
include(tools/cmake/main.cmake)

add_subdirectory(3rdparty)
add_subdirectory(tools)
add_subdirectory(src)

if (BUILD_TESTS)
28 changes: 17 additions & 11 deletions docs/build-from-source.md
Original file line number Diff line number Diff line change
@@ -15,15 +15,17 @@ Example output

## Required packages

| Package | Min. Version | |
|------------|--------------|---------------|
| pkg-config | | **mandatory** |
| libnuma | \>= 1.1 | **mandatory** |
| libyaml | \>= 1.1 | **mandatory** |
| libmbedtls | \>= 2.28 | **mandatory** |
| libatomic1 | | **mandatory** |
| openssl | \>= 2.0 | **mandatory** |
| curl | \>= 7.0 | **mandatory** |
| Package | Min. Version | |
|---------------|--------------|---------------|
| pkg-config | | **mandatory** |
| libnuma | \>= 1.1 | **mandatory** |
| libyaml | \>= 1.1 | **mandatory** |
| libmbedtls | \>= 2.28 | **mandatory** |
| libatomic1 | | **mandatory** |
| openssl | \>= 2.0 | **mandatory** |
| curl | \>= 7.0 | **mandatory** |
| libpcre2 | | **mandatory** |
| libjson-c-dev | | **mandatory** |

## Install the required packages

@@ -38,7 +40,9 @@ sudo apt install \
libnuma1 libnuma-dev \
libcurl4-openssl-dev libcurl4 \
libyaml-0-2 libyaml-dev \
libmbedtls-dev libmbedtls14
libmbedtls-dev libmbedtls14 \
libpcre2-8-0 libpcre2-dev \
libjson-c-dev
```

### Ubuntu 20.04
@@ -52,7 +56,9 @@ sudo apt install \
libnuma1 libnuma-dev \
libcurl4-openssl-dev libcurl4 \
libyaml-0-2 libyaml-dev \
libmbedtls-dev libmbedtls12
libmbedtls-dev libmbedtls12 \
libpcre2-8-0 libpcre2-dev \
libjson-c-dev
```

### Debian 11
1 change: 1 addition & 0 deletions tools/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
add_subdirectory(cachegrand-benchmark-generator)
1 change: 1 addition & 0 deletions tools/cachegrand-benchmark-generator/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
add_subdirectory(src)
43 changes: 43 additions & 0 deletions tools/cachegrand-benchmark-generator/src/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
include(FindPkgConfig)

pkg_check_modules(LIBPCRE2 REQUIRED libpcre2-8)
pkg_check_modules(LIBJSONC REQUIRED json-c)

##################################################

file(GLOB_RECURSE CACHEGRAND_TESTS "../../../tests/unit_tests/modules/redis/command/*.cpp")
foreach(t ${CACHEGRAND_TESTS})
set(ALL_TESTS "${ALL_TESTS} \"${t}\",\n")
endforeach()

configure_file(all_tests.h.in all_tests.h)

##################################################
# Build the executable
##################################################
file(GLOB_RECURSE SRC_FILES_ANALYZER "*.c")

add_executable(
cachegrand-benchmark-generator
analyzer.c
${SRC_FILES_ANALYZER})

set_target_properties(
cachegrand-benchmark-generator
PROPERTIES
LINKER_LANGUAGE C)

target_link_libraries(
cachegrand-benchmark-generator
PUBLIC
${LIBPCRE2_LIBRARIES} ${LIBJSONC_LIBRARIES})

target_link_directories(
cachegrand-benchmark-generator
PUBLIC
${LIBPCRE2_LIBRARY_DIRS} ${LIBJSONC_LIBRARY_DIRS} ${CMAKE_CURRENT_BINARY_DIR})

target_include_directories(
cachegrand-benchmark-generator
PUBLIC
${LIBPCRE2_INCLUDE_DIRS} ${LIBJSONC_INCLUDE_DIRS} ${CMAKE_CURRENT_BINARY_DIR})
8 changes: 8 additions & 0 deletions tools/cachegrand-benchmark-generator/src/all_tests.h.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#ifndef CACHEGRAND_BENCHMARK_GENERATOR_ALL_TESTS_H
#define CACHEGRAND_BENCHMARK_GENERATOR_ALL_TESTS_H

const char* all_tests[] = {
@ALL_TESTS@
};

#endif //CACHEGRAND_BENCHMARK_GENERATOR_ALL_TESTS_H
90 changes: 90 additions & 0 deletions tools/cachegrand-benchmark-generator/src/analyzer.c
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
/**
* Copyright (C) 2018-2022 Vito Castellano
* All rights reserved.
*
* This software may be modified and distributed under the terms
* of the BSD license. See the LICENSE file for details.
**/

#include <stdio.h>
#include <stdbool.h>
#include <stdlib.h>
#include <string.h>
#include <json.h>

#include "matcher.h"
#include "builder.h"
#include "support.h"

#include "analyzer.h"

test_t* analyzer_analyze(
const char *file_path) {
char *body = support_read_file(file_path);
char *test_name = matcher_get_test_name(body);
if (NULL == test_name) return NULL;

test_t *test = builder_new_test_p();
test->name = strdup(test_name);

analyzer_recursive_match(
body,
ANALYZER_START_PADDING,
test,
NULL);

return test;
}

int analyzer_recursive_match(
const char *body,
int padding,
test_t *current_test,
section_t *father_section) {
// Match SECTIONS
matcher_t *sections = matcher_get_sections(body, padding);
if (sections->n_matches <= 0) {
free(sections);
return 0;
}

for (int i = 0; i < sections->n_matches; ++i) {
section_t *current_section = builder_new_section_p();

// Match SECTION name
char *section_name = matcher_get_section_name(sections->matches[i]);
if (NULL == section_name) continue;
current_section->name = section_name;

// Match REQUIRE section
matcher_t *section_requires = matcher_get_requires_section(sections->matches[i], padding);
if (section_requires->n_matches > 0) {
for (int j = 0; j < section_requires->n_matches; ++j) {
// Match COMMAND
char *command = matcher_get_require_command(section_requires->matches[j]);
if (NULL == command) continue;
builder_section_append_command(
current_section, command);
}
}
free(section_requires);

if (NULL != father_section) {
builder_section_append_subsection(
father_section,
current_section);
} else {
builder_test_append_section(
current_test,
current_section);
}

analyzer_recursive_match(
sections->matches[i],
padding+4,
current_test,
current_section);
}

free(sections);
}
15 changes: 15 additions & 0 deletions tools/cachegrand-benchmark-generator/src/analyzer.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
#ifndef CACHEGRAND_BENCHMARK_GENERATOR_ANALYZER_H
#define CACHEGRAND_BENCHMARK_GENERATOR_ANALYZER_H

#define ANALYZER_START_PADDING 4

test_t* analyzer_analyze(
const char *file_path);

int analyzer_recursive_match(
const char *body,
int padding,
test_t *current_test,
section_t *current_section);

#endif //CACHEGRAND_BENCHMARK_GENERATOR_ANALYZER_H
103 changes: 103 additions & 0 deletions tools/cachegrand-benchmark-generator/src/builder.c
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
/**
* Copyright (C) 2018-2022 Vito Castellano
* All rights reserved.
*
* This software may be modified and distributed under the terms
* of the BSD license. See the LICENSE file for details.
**/

#include <stdlib.h>
#include <stdbool.h>

#include "builder.h"

section_t* builder_new_section_p() {
section_t *section;
section = malloc(sizeof(section_t));
section->subsections = NULL;
section->commands = NULL;
section->n_subsections = 0;
section->n_commands = 0;
return section;
}

bool builder_section_append_subsection(
section_t *section,
section_t *subsections) {
if (NULL == subsections) {
return false;
}

section->subsections = realloc(
section->subsections,
(section->n_subsections+1) * sizeof(section_t*));
section->subsections[section->n_subsections] = subsections;
section->n_subsections++;

return true;
}

bool builder_section_append_command(
section_t *section,
char *command) {
if (NULL == command) {
return false;
}

section->commands = realloc(
section->commands,
(section->n_commands+1) * sizeof(char*));
section->commands[section->n_commands] = command;
section->n_commands++;

return true;
}


test_t* builder_new_test_p() {
test_t *test;
test = malloc(sizeof(test_t));
test->sections = NULL;
test->n_sections = 0;
return test;
}

bool builder_test_append_section(
test_t *test,
section_t *section) {
if (NULL == section) {
return false;
}

test->sections = realloc(
test->sections,
(test->n_sections+1) * sizeof(section_t*));
test->sections[test->n_sections] = section;
test->n_sections++;

return true;
}

tests_t* builder_new_tests_p() {
tests_t *tests;
tests = malloc(sizeof(tests_t));
tests->tests = NULL;
tests->n_tests = 0;
return tests;
}

bool builder_tests_append_test(
tests_t *tests,
test_t *test) {
if (NULL == test) {
return false;
}

tests->tests = realloc(
tests->tests,
(tests->n_tests+1) * sizeof(test_t*));
tests->tests[tests->n_tests] = test;
tests->n_tests++;

return true;
}
Loading