Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🔥 feat: Add LevelDB driver #1565

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
178 changes: 178 additions & 0 deletions leveldb/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
---
id: leveldb
title: LevelDB
---

![Release](https://img.shields.io/github/v/tag/gofiber/storage?filter=leveldb*)
[![Discord](https://img.shields.io/discord/704680098577514527?style=flat&label=%F0%9F%92%AC%20discord&color=00ACD7)](https://gofiber.io/discord)
![Test](https://img.shields.io/github/actions/workflow/status/gofiber/storage/test-leveldb.yml?label=Tests)
![Security](https://img.shields.io/github/actions/workflow/status/gofiber/storage/gosec.yml?label=Security)
![Linter](https://img.shields.io/github/actions/workflow/status/gofiber/storage/linter.yml?label=Linter)

A fast key-value DB using [syndtr/goleveldb](https://github.com/syndtr/goleveldb)

**Note: Requires Go 1.23.1 and above**
SadikSunbul marked this conversation as resolved.
Show resolved Hide resolved

### Table of Contents

- [Signatures](#signatures)
- [Installation](#installation)
- [Examples](#examples)
- [Config](#config)
- [Default Config](#default-config)

### Signatures

```go
func New(config ...Config) Storage
func (s *Storage) Get(key string) ([]byte, error)
func (s *Storage) Set(key string, val []byte, exp time.Duration) error
func (s *Storage) Delete(key string) error
func (s *Storage) Reset() error
func (s *Storage) Close() error
func (s *Storage) Conn() *leveldb.DB
```

### Installation

LevelDB is tested on the 2 last [Go versions](https://golang.org/dl/) with support for modules. So make sure to initialize one first if you didn't do that yet:

```bash
go mod init github.com/<user>/<repo>
```

And then install the leveldb implementation:

```bash
go get github.com/gofiber/storage/leveldb
```

### Examples

Import the storage package.

```go
import "github.com/gofiber/storage/leveldb"
```

You can use the following possibilities to create a storage:

```go
// Initialize default config
store := leveldb.New()

// Initialize custom config
store := leveldb.New(leveldb.Config{
Database: "./fiber.leveldb",
Reset: false,
GCInterval: 10 * time.Second,
})
```

### Config

```go
type Config struct {
// DBPath is the filesystem path for the database
//
// Optional. Default is "./fiber.leveldb"
DBPath string

// CacheSize is the size of LevelDB's cache (in MB)
//
// Optional. Default is 8MB
CacheSize int

// BlockSize is the size of data blocks (in KB)
//
// Optional. Default is 4KB
BlockSize int

// WriteBuffer is the size of write buffer (in MB)
//
// Optional. Default is 4MB
WriteBuffer int

// CompactionL0Trigger is the number of level-0 tables that triggers compaction
//
// Optional. Default is 4
CompactionL0Trigger int

// WriteL0PauseTrigger is the number of level-0 tables that triggers write pause
//
// Optional. Default is 12
WriteL0PauseTrigger int

// WriteL0SlowdownTrigger is the number of level-0 tables that triggers write slowdown
//
// Optional. Default is 8
WriteL0SlowdownTrigger int

// MaxOpenFiles is the maximum number of open files that can be held
//
// Optional. Default is 200 on MacOS, 500 on others
MaxOpenFiles int

// CompactionTableSize is the size of compaction table (in MB)
//
// Optional. Default is 2MB
CompactionTableSize int

// BloomFilterBits is the number of bits used in bloom filter
//
// Optional. Default is 10 bits/key
BloomFilterBits int

// NoSync completely disables fsync
//
// Optional. Default is false
NoSync bool

// ReadOnly opens the database in read-only mode
//
// Optional. Default is false
ReadOnly bool

// ErrorIfMissing returns error if database doesn't exist
//
// Optional. Default is false
ErrorIfMissing bool

// ErrorIfExist returns error if database exists
//
// Optional. Default is false
ErrorIfExist bool

// GCInterval is the garbage collection interval
//
// Optional. Default is 10 minutes
GCInterval time.Duration
}
```

### Default Config

```go
var ConfigDefault = Config{
DBPath: "./fiber.leveldb",
CacheSize: 8, // 8 MB
BlockSize: 4, // 4 KB
WriteBuffer: 4, // 4 MB
CompactionL0Trigger: 4,
WriteL0PauseTrigger: 12,
WriteL0SlowdownTrigger: 8,
MaxOpenFiles: func() int {
if runtime.GOOS == "darwin" {
return 200 // MacOS
}
return 500 // Unix/Linux
}(),
CompactionTableSize: 2, // 2 MB
BloomFilterBits: 10, // 10 bits per key
NoSync: false,
ReadOnly: false,
ErrorIfMissing: false,
ErrorIfExist: false,
GCInterval: 10 * time.Minute,
}
```
163 changes: 163 additions & 0 deletions leveldb/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
package leveldb

import (
"runtime"
"time"
)

// Config holds the configuration options for LevelDB database
type Config struct {
// DBPath is the filesystem path for the database
//
// Optional. Default is "./fiber.leveldb"
DBPath string
SadikSunbul marked this conversation as resolved.
Show resolved Hide resolved

// CacheSize is the size of LevelDB's cache (in MB)
//
// Optional. Default is 8MB
CacheSize int

// BlockSize is the size of data blocks (in KB)
//
// Optional. Default is 4KB
BlockSize int

// WriteBuffer is the size of write buffer (in MB)
//
// Optional. Default is 4MB
WriteBuffer int

// CompactionL0Trigger is the number of level-0 tables that triggers compaction
//
// Optional. Default is 4
CompactionL0Trigger int

// WriteL0PauseTrigger is the number of level-0 tables that triggers write pause
//
// Optional. Default is 12
WriteL0PauseTrigger int

// WriteL0SlowdownTrigger is the number of level-0 tables that triggers write slowdown
//
// Optional. Default is 8
WriteL0SlowdownTrigger int

// MaxOpenFiles is the maximum number of open files that can be held
//
// Optional. Default is 200 on MacOS, 500 on others
MaxOpenFiles int

// CompactionTableSize is the size of compaction table (in MB)
//
// Optional. Default is 2MB
CompactionTableSize int

// BloomFilterBits is the number of bits used in bloom filter
//
// Optional. Default is 10 bits/key
BloomFilterBits int

// NoSync completely disables fsync
//
// Optional. Default is false
NoSync bool

// ReadOnly opens the database in read-only mode
//
// Optional. Default is false
ReadOnly bool

// ErrorIfMissing returns error if database doesn't exist
//
// Optional. Default is false
ErrorIfMissing bool

// ErrorIfExist returns error if database exists
//
// Optional. Default is false
ErrorIfExist bool

// GCInterval is the garbage collection interval
//
// Optional. Default is 10 minutes
GCInterval time.Duration
}

// ConfigDefault is the default config
var ConfigDefault = Config{
DBPath: "./fiber.leveldb",
CacheSize: 8, // 8 MB
BlockSize: 4, // 4 KB
WriteBuffer: 4, // 4 MB
CompactionL0Trigger: 4,
WriteL0PauseTrigger: 12,
WriteL0SlowdownTrigger: 8,
MaxOpenFiles: func() int {
if runtime.GOOS == "darwin" {
return 200 // MacOS
}
return 500 // Unix/Linux
}(),
CompactionTableSize: 2, // 2 MB
BloomFilterBits: 10, // 10 bits per key
NoSync: false,
ReadOnly: false,
ErrorIfMissing: false,
ErrorIfExist: false,
GCInterval: 10 * time.Minute,
}

// configDefault is a helper function to set default values for the config
func configDefault(config ...Config) Config {
if len(config) < 1 {
return ConfigDefault
}

cfg := config[0]

if cfg.DBPath == "" {
cfg.DBPath = ConfigDefault.DBPath
}

if cfg.CacheSize <= 0 {
cfg.CacheSize = ConfigDefault.CacheSize
}

if cfg.BlockSize <= 0 {
cfg.BlockSize = ConfigDefault.BlockSize
}

if cfg.WriteBuffer <= 0 {
cfg.WriteBuffer = ConfigDefault.WriteBuffer
}

if cfg.CompactionL0Trigger <= 0 {
cfg.CompactionL0Trigger = ConfigDefault.CompactionL0Trigger
}

if cfg.WriteL0PauseTrigger <= 0 {
cfg.WriteL0PauseTrigger = ConfigDefault.WriteL0PauseTrigger
}

if cfg.WriteL0SlowdownTrigger <= 0 {
cfg.WriteL0SlowdownTrigger = ConfigDefault.WriteL0SlowdownTrigger
}

if cfg.MaxOpenFiles <= 0 {
cfg.MaxOpenFiles = ConfigDefault.MaxOpenFiles
}

if cfg.CompactionTableSize <= 0 {
cfg.CompactionTableSize = ConfigDefault.CompactionTableSize
}

if cfg.BloomFilterBits <= 0 {
cfg.BloomFilterBits = ConfigDefault.BloomFilterBits
}

if cfg.GCInterval <= 0 {
cfg.GCInterval = ConfigDefault.GCInterval
}

return cfg
}
19 changes: 19 additions & 0 deletions leveldb/config_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
package leveldb

import (
"testing"

"github.com/stretchr/testify/assert"
SadikSunbul marked this conversation as resolved.
Show resolved Hide resolved
)

func TestConfigConfigMaxOpenFiles(t *testing.T) {
cfg := Config{
MaxOpenFiles: 1000,
}
assert.Equal(t, 1000, cfg.MaxOpenFiles)
}

func TestConfigDefaultDarwin(t *testing.T) { // MacOS
cfg := configDefault()
assert.Equal(t, 200, cfg.MaxOpenFiles)
}
12 changes: 12 additions & 0 deletions leveldb/go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
module github.com/gofiber/storage/leveldb

go 1.23.1
SadikSunbul marked this conversation as resolved.
Show resolved Hide resolved

require (
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/stretchr/testify v1.10.0 // indirect
github.com/syndtr/goleveldb v1.0.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
Loading
Loading