Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(database): 切换后端数据库操作库至gorm #1060

Merged
merged 83 commits into from
Nov 7, 2024
Merged
Show file tree
Hide file tree
Changes from 81 commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
9b078dc
break: 尝试举家迁移gorm
PaienNate Sep 30, 2024
e56ac98
gorm: 进一步优化代码
PaienNate Oct 1, 2024
ba55147
gorm: 修正主键标签错误BUG以及部分查询错误BUG。
PaienNate Oct 2, 2024
ea0626d
gorm: 标签修改为兼容性更好的text,并修正了绑卡查询的一个BUG
PaienNate Oct 2, 2024
c92e6b0
Update readme.md
PaienNate Oct 3, 2024
4b00266
fix: 兼容部分零值和NULL值引发的异常
PaienNate Oct 3, 2024
c0874fc
fix: 合并上游分支的新提交
PaienNate Oct 3, 2024
6288298
Merge branch 'sealdice:master' into master
PaienNate Oct 8, 2024
a7e5138
Merge branch 'sealdice:master' into master
PaienNate Oct 10, 2024
7053397
chores:合并上游分支
PaienNate Oct 10, 2024
545036a
chores: 突然发现不小心把master里混入了这个错误的说明,本来以为不会合并的,这下傻眼了.jpg
PaienNate Oct 10, 2024
de2f52c
fix: 使用CommandInfoStr和RawMsgIDStr替代原本的interface{}用于参与数据库处理。
PaienNate Oct 11, 2024
9970e1b
fix: 忽略测试版注释描述,转而根据代码逻辑进行处理。
PaienNate Oct 11, 2024
cec7a91
fix: 修正John佬指明的,错误未使用事物的插入
PaienNate Oct 11, 2024
b6b19c0
fix: 修正部分结构体,以解决零值问题。
PaienNate Oct 11, 2024
01eaf93
fix: 修正sql.NullInt等相关使用导致的异常,同时gorm添加Info日志。
PaienNate Oct 11, 2024
c800d64
fix: 修正sql.NullInt等相关使用导致的异常,同时gorm添加Info日志。 [skip ci]
PaienNate Oct 11, 2024
04484ba
Merge remote-tracking branch 'origin/feat-gorm' into feat-gorm [skip ci]
PaienNate Oct 11, 2024
dac0566
fix: 做修正1
PaienNate Oct 13, 2024
341452e
fix: 默认init保持sqlite.[skip ci]
PaienNate Oct 13, 2024
efc3543
0
PaienNate Oct 15, 2024
eb9e80c
Merge remote-tracking branch 'sealdice/master' into feat-gorm
PaienNate Oct 15, 2024
77fa5e4
chores: 跟主线进度
PaienNate Oct 15, 2024
d266a0a
fix: 修正所有建表索引。
PaienNate Oct 15, 2024
974c7fc
chores: 添加gorm日志到zap框架。
PaienNate Oct 15, 2024
2a6c85b
Revert "0"
PaienNate Oct 15, 2024
941f21e
fix: 修正部分检查策略,添加缓存策略。
PaienNate Oct 22, 2024
f6913b4
远端合并
PaienNate Oct 22, 2024
f9c4b57
fix: 去除不好看的gorm zap logger,添加缓存,并试图make lint happy.
PaienNate Oct 22, 2024
1064ba4
make lint happy.
PaienNate Oct 22, 2024
55787fc
[skip ci]make lint happy.
PaienNate Oct 22, 2024
fddd550
make lint unhappy.
PaienNate Oct 22, 2024
7a1a69c
Merge branch 'feat-gorm' of github.com:PaienNate/sealdice-core into f…
PaienNate Oct 22, 2024
2f9e663
Merge remote-tracking branch 'sealdice/master' into feat-gorm
PaienNate Oct 22, 2024
8bc278c
make lint happy again.
PaienNate Oct 22, 2024
bcc532c
fix: 调整日志相关,日志输出到database.log,修正John佬说的各种问题
PaienNate Oct 22, 2024
aa77f61
111
PaienNate Oct 23, 2024
a8f86e4
Revert "111"
PaienNate Oct 23, 2024
31a94ab
chores: 跟踪分支
PaienNate Oct 23, 2024
91defd5
暂存
PaienNate Oct 24, 2024
8a0261c
暂存
PaienNate Oct 24, 2024
c074385
fix: 修正旧代码忽略错误导致的GORM异常,并使用事务保证数据一致性。
PaienNate Oct 25, 2024
95700a4
fix:合并上游
PaienNate Oct 25, 2024
5796559
合并新代码
PaienNate Oct 25, 2024
440da01
fix: 绑卡等各种落库问题尝试进行修正
PaienNate Oct 25, 2024
d8e4156
fix: 修正GORM绑定问题导致跨群绑卡检查pc list无效的BUG。
PaienNate Oct 26, 2024
6acbc42
chores: 合并远端分支[skip ci]
PaienNate Oct 26, 2024
4a87586
fix: 修改Table为Model,优化部分数据库请求逻辑,修正ban_info的错误请求逻辑
PaienNate Oct 26, 2024
75d281b
Merge remote-tracking branch 'TRULY远端/master' into feat-gorm
PaienNate Oct 26, 2024
677bfca
fix: 修正endpoint_info和group_info可能不落盘的问题。
PaienNate Oct 26, 2024
af8767e
[skip ci] Add Some Comments.
PaienNate Oct 26, 2024
6f6bd84
Revert "feat: 增加多个子模块 vm engine 开关 (#1086)"
PaienNate Oct 27, 2024
ed527b3
暂存开发进度
PaienNate Oct 27, 2024
3e75f71
Merge remote-tracking branch 'TRULY远端/master' into feat-gorm
PaienNate Oct 27, 2024
544e22b
Revert "Revert "feat: 增加多个子模块 vm engine 开关 (#1086)""
PaienNate Oct 27, 2024
3ae19c4
1
PaienNate Oct 27, 2024
eea444c
0
PaienNate Oct 28, 2024
28da557
shit
PaienNate Oct 28, 2024
47acb87
fix: 这次有数据了吧……
PaienNate Oct 28, 2024
ee7f2f5
fix: 指定type bool值
PaienNate Oct 28, 2024
e325d09
Merge branch 'feat-gorm' of github.com:PaienNate/sealdice-core into f…
PaienNate Oct 28, 2024
4d27ad7
Merge remote-tracking branch 'TRULY远端/master' into feat-gorm
PaienNate Oct 28, 2024
6846374
fmt清理
PaienNate Oct 28, 2024
5b11b00
fix(config): 修复保留在 Dice 上的个别配置项的设置
JustAnotherID Oct 28, 2024
d30053f
fix(config): 修复退出不活跃群组相关的两个配置项在ui不展示的问题
JustAnotherID Oct 28, 2024
a54425d
Merge branch 'master' into fix/config-set
JustAnotherID Oct 28, 2024
d49f80c
Merge remote-tracking branch 'TRULY远端/master' into feat-gorm
PaienNate Oct 28, 2024
9486a2f
无事发生。
PaienNate Oct 28, 2024
77f8947
fix: 去除错误的调用Save函数方法
PaienNate Oct 28, 2024
7718f53
Merge remote-tracking branch 'IDlao/fix/config-set' into feat-gorm
PaienNate Oct 28, 2024
d4cd9b4
fix: make lint happy!
PaienNate Oct 28, 2024
9c6b66d
fix: 根据木落指导,修改部分代码
PaienNate Oct 29, 2024
efeae85
chores: 大远端追逐术!
PaienNate Oct 29, 2024
f6f7ee3
fix: 修改搞错的判断逻辑,以保证仍然能正确落盘。
PaienNate Oct 30, 2024
d493a7b
Merge remote-tracking branch 'TRULY远端/master' into feat-gorm
PaienNate Oct 30, 2024
05e2339
fix: 去除WAL文件删除,修正日志分页错误和条数显示错误,并确认不能改绑卡代码
PaienNate Oct 30, 2024
9211c43
chores: 添加LogDeleteByID的索引,并更换修正了建表的GORM-SQLITE驱动(包括CGO和非CGO)
PaienNate Nov 2, 2024
cc5a3d3
refactor: 根据gorm的特点,去除wal删除代码并使用线程安全的文件锁进行多开限制。
PaienNate Nov 5, 2024
a5c91f2
make review dog happy.
PaienNate Nov 5, 2024
08624e1
review dog happy.
PaienNate Nov 5, 2024
285d547
review dog happy again!
PaienNate Nov 5, 2024
68db294
fix: 去除冗余的err判断,并添加注释
PaienNate Nov 6, 2024
d023cdc
fix: 修正部分代码
PaienNate Nov 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -42,3 +42,4 @@ _help_cache

.vscode/
!.vscode/settings.json
sealdice-lock.lock
18 changes: 15 additions & 3 deletions api/api_bind.go
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,11 @@ func forceStop(c echo.Context) error {
dbData := d.DBData
if dbData != nil {
d.DBData = nil
_ = dbData.Close()
db, err := dbData.DB()
if err != nil {
return
}
_ = db.Close()
}
})()

Expand All @@ -201,7 +205,11 @@ func forceStop(c echo.Context) error {
dbLogs := d.DBLogs
if dbLogs != nil {
d.DBLogs = nil
_ = dbLogs.Close()
db, err := dbLogs.DB()
if err != nil {
return
}
_ = db.Close()
}
})()

Expand All @@ -213,7 +221,11 @@ func forceStop(c echo.Context) error {
if cm != nil && cm.DB != nil {
dbCensor := cm.DB
cm.DB = nil
_ = dbCensor.Close()
db, err := dbCensor.DB()
if err != nil {
return
}
_ = db.Close()
}
})()
}
Expand Down
9 changes: 8 additions & 1 deletion api/censor.go
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,14 @@ func censorStop(c echo.Context) error {

(&myDice.Config).EnableCensor = false
myDice.MarkModified()
_ = myDice.CensorManager.DB.Close()
db, err2 := myDice.CensorManager.DB.DB()
if err2 != nil {
return Error(&c, "关闭拦截引擎失败", Response{})
}
err = db.Close()
if err != nil {
return err
}
myDice.CensorManager = nil

return Success(&c, Response{})
Expand Down
2 changes: 1 addition & 1 deletion dice/builtin_commands.go
Original file line number Diff line number Diff line change
Expand Up @@ -1917,7 +1917,7 @@ func (d *Dice) registerCoreCommands() {
setCurPlayerName(b)
}
attrs.LastModifiedTime = time.Now().Unix()
attrs.SaveToDB(am.db, nil) // 直接保存
attrs.SaveToDB(am.db) // 直接保存
ReplyToSender(ctx, msg, "操作完成")
} else {
ReplyToSender(ctx, msg, "此角色名已存在")
Expand Down
16 changes: 9 additions & 7 deletions dice/dice.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,13 @@ import (
"github.com/dop251/goja_nodejs/eventloop"
"github.com/dop251/goja_nodejs/require"
"github.com/go-creed/sat"
"github.com/jmoiron/sqlx"
wr "github.com/mroth/weightedrand"
"github.com/robfig/cron/v3"
ds "github.com/sealdice/dicescript"
"github.com/tidwall/buntdb"
rand2 "golang.org/x/exp/rand"
"golang.org/x/exp/slices"
"gorm.io/gorm"

"sealdice-core/dice/logger"
"sealdice-core/dice/model"
Expand Down Expand Up @@ -133,8 +133,8 @@ type Dice struct {
LastUpdatedTime int64 `yaml:"-"`
TextMap map[string]*wr.Chooser `yaml:"-"`
BaseConfig BaseConfig `yaml:"-"`
DBData *sqlx.DB `yaml:"-"` // 数据库对象
DBLogs *sqlx.DB `yaml:"-"` // 数据库对象
DBData *gorm.DB `yaml:"-"` // 数据库对象
DBLogs *gorm.DB `yaml:"-"` // 数据库对象
Logger *log.Helper `yaml:"-"` // 日志
LogWriter *log.WriterX `yaml:"-"` // 用于api的log对象
IsDeckLoading bool `yaml:"-"` // 正在加载中
Expand All @@ -160,10 +160,12 @@ type Dice struct {
AliveNoticeEntry cron.EntryID `yaml:"-" json:"-"`
JsPrinter *PrinterFunc `yaml:"-" json:"-"`
JsRequire *require.RequireModule `yaml:"-" json:"-"`
JsLoop *eventloop.EventLoop `yaml:"-" json:"-"`
JsScriptList []*JsScriptInfo `yaml:"-" json:"-"`
JsScriptCron *cron.Cron `yaml:"-" json:"-"`
JsScriptCronLock *sync.Mutex `yaml:"-" json:"-"`
// 如果我们重用它呢?
JSRegistry *require.Registry `yaml:"-" json:"-"`
JsLoop *eventloop.EventLoop `yaml:"-" json:"-"`
JsScriptList []*JsScriptInfo `yaml:"-" json:"-"`
JsScriptCron *cron.Cron `yaml:"-" json:"-"`
JsScriptCronLock *sync.Mutex `yaml:"-" json:"-"`
// 重载使用的互斥锁
JsReloadLock sync.Mutex `yaml:"-" json:"-"`
// 内置脚本摘要表,用于判断内置脚本是否有更新
Expand Down
24 changes: 9 additions & 15 deletions dice/dice_attrs_manager.go
Original file line number Diff line number Diff line change
@@ -1,20 +1,19 @@
package dice

import (
"database/sql"
"errors"
"fmt"
"time"

"github.com/jmoiron/sqlx"
ds "github.com/sealdice/dicescript"
"gorm.io/gorm"

"sealdice-core/dice/model"
log "sealdice-core/utils/kratos"
)

type AttrsManager struct {
db *sqlx.DB
db *gorm.DB
logger *log.Helper
m SyncMap[string, *AttributesItem]
}
Expand Down Expand Up @@ -169,24 +168,18 @@ func (am *AttrsManager) CheckForSave() (int, int) {
return 0, 0
}

tx, err := db.Begin()
if err != nil {
if am.logger != nil {
am.logger.Errorf("定期写入用户数据出错(创建事务): %v", err)
}
return 0, 0
}
tx := db.Begin()

am.m.Range(func(key string, value *AttributesItem) bool {
if !value.IsSaved {
saved += 1
value.SaveToDB(db, tx)
value.SaveToDB(tx)
}
times += 1
return true
})

err = tx.Commit()
err := tx.Commit().Error
if err != nil {
if am.logger != nil {
am.logger.Errorf("定期写入用户数据出错(提交事务): %v", err)
Expand All @@ -210,7 +203,8 @@ func (am *AttrsManager) CheckAndFreeUnused() {
am.m.Range(func(key string, value *AttributesItem) bool {
if value.LastUsedTime-currentTime > 60*10 {
prepareToFree[key] = 1
value.SaveToDB(am.db, nil)
// 直接保存
value.SaveToDB(am.db)
}
return true
})
Expand Down Expand Up @@ -279,13 +273,13 @@ type AttributesItem struct {
SheetType string
}

func (i *AttributesItem) SaveToDB(db *sqlx.DB, tx *sql.Tx) {
func (i *AttributesItem) SaveToDB(db *gorm.DB) {
// 使用事务写入
rawData, err := ds.NewDictVal(i.valueMap).V().ToJSON()
if err != nil {
return
}
err = model.AttrsPutById(db, tx, i.ID, rawData, i.Name, i.SheetType)
err = model.AttrsPutById(db, i.ID, rawData, i.Name, i.SheetType)
if err != nil {
log.Error("保存数据失败", err.Error())
return
Expand Down
4 changes: 2 additions & 2 deletions dice/dice_censor.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import (
"sort"
"strings"

"github.com/jmoiron/sqlx"
"gorm.io/gorm"

"sealdice-core/dice/censor"
"sealdice-core/dice/model"
Expand Down Expand Up @@ -54,7 +54,7 @@ type CensorManager struct {
IsLoading bool
Parent *Dice
Censor *censor.Censor
DB *sqlx.DB
DB *gorm.DB
SensitiveWordsFiles map[string]*censor.WordFile
}

Expand Down
5 changes: 3 additions & 2 deletions dice/im_session.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import (
"strings"
"time"

"gorm.io/gorm"

"sealdice-core/dice/model"
"sealdice-core/message"
log "sealdice-core/utils/kratos"
Expand All @@ -20,7 +22,6 @@ import (
rand2 "golang.org/x/exp/rand"

"github.com/dop251/goja"
"github.com/jmoiron/sqlx"
"golang.org/x/time/rate"
"gopkg.in/yaml.v3"
)
Expand Down Expand Up @@ -192,7 +193,7 @@ func (group *GroupInfo) IsActive(ctx *MsgContext) bool {
return false
}

func (group *GroupInfo) PlayerGet(db *sqlx.DB, id string) *GroupPlayerInfo {
func (group *GroupInfo) PlayerGet(db *gorm.DB, id string) *GroupPlayerInfo {
if group.Players == nil {
group.Players = new(SyncMap[string, *GroupPlayerInfo])
}
Expand Down
2 changes: 2 additions & 0 deletions dice/model/attr.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ import (
"github.com/jmoiron/sqlx"
)

// 废弃代码先不改

func attrGetAllBase(db *sqlx.DB, bucket string, key string) []byte {
var buf []byte

Expand Down
Loading