Skip to content

Commit

Permalink
remove WAL from sqlite to avoid bugs
Browse files Browse the repository at this point in the history
also log how long it took to load or synch a file

Why remove WAL?
It was faster yes, but often the engine would not have the up-to-date info and then crash.
Adding walcheckpoints all over the place didnt seem to help

the slowdown is a bit noticeable with >3000k columns, but given that we didnt support them before shouldnt be a big problem.

For speed comparisons:

        WAL     NO-WAL
3KC      4s        10s
10KC    28s        54s
200KR   30s        30s
1MR    146s       147s
1KR5KC 347s       356s

Where the "R" sets have 2 columns and the "C" sets have a single row.
As can be seen, there *is* a delay for 1K cols and above, but it seems constant.
And any fixes for the synching-db problem with the engine most likely includes adding new delays...
So lets just turn it off
  • Loading branch information
JorisGoosen committed Sep 12, 2024
1 parent 2ff3871 commit 3102e9e
Show file tree
Hide file tree
Showing 5 changed files with 11 additions and 23 deletions.
16 changes: 0 additions & 16 deletions CommonData/databaseinterface.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1589,8 +1589,6 @@ void DatabaseInterface::create()
}
else
Log::log() << "Opened internal sqlite database for creation at '" << dbFile() << "'." << std::endl;

dbStartUpPragmas();

transactionWriteBegin();
runStatements(_dbConstructionSql);
Expand All @@ -1615,20 +1613,6 @@ void DatabaseInterface::load()
else
Log::log() << "Opened internal sqlite database for loading at '" << dbFile() << "'." << std::endl;

dbStartUpPragmas();
}

void DatabaseInterface::dbStartUpPragmas()
{
runStatements("pragma journal_mode = WAL;");
runStatements("pragma synchronous = normal;");
}


void DatabaseInterface::doWALCheckpoint()
{
// https://www.sqlite.org/pragma.html#pragma_wal_checkpoint
runStatements("PRAGMA wal_checkpoint(TRUNCATE);");
}

void DatabaseInterface::close()
Expand Down
6 changes: 1 addition & 5 deletions CommonData/databaseinterface.h
Original file line number Diff line number Diff line change
Expand Up @@ -158,10 +158,7 @@ class DatabaseInterface
void transactionWriteEnd(bool rollback = false); ///< runs COMMIT or ROLLBACK based on rollback and ends the transaction. Tracks whether nested and only does BEGIN+COMMIT at lowest depth
void transactionReadBegin(); ///< runs BEGIN DEFERRED and waits for sqlite to not be busy anymore if some other process is writing Tracks whether nested and only does BEGIN+COMMIT at lowest depth
void transactionReadEnd(); ///< runs COMMIT and ends the transaction. Tracks whether nested and only does BEGIN+COMMIT at lowest depth

//WAL handling, was added in https://github.com/jasp-stats/jasp-desktop/commit/4cfe6197714440b0ab6936891fa5ff7cc8ac19b6, see docs: https://www.sqlite.org/wal.html
void doWALCheckpoint(); ///< Writes all pending changes to the db-file


private:
void _doubleTroubleBinder(sqlite3_stmt *stmt, int param, double dbl); ///< Needed to work around the lack of support for NAN, INF and NEG_INF in sqlite, converts those to string to make use of sqlite flexibility
double _doubleTroubleReader(sqlite3_stmt *stmt, int colI); ///< The reading counterpart to _doubleTroubleBinder to convert string representations of NAN, INF and NEG_INF back to double
Expand All @@ -172,7 +169,6 @@ class DatabaseInterface
void load(); ///< Loads a sqlite database from sessiondir (after loading a jaspfile)
void close(); ///< Closes the loaded database and disconnects
bool tableHasColumn(const std::string & tableName, const std::string & columnName);
void dbStartUpPragmas();

int _transactionWriteDepth = 0,
_transactionReadDepth = 0;
Expand Down
1 change: 0 additions & 1 deletion Desktop/data/exporters/jaspexporter.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,5 @@ void JASPExporter::saveAnalyses(archive *a)

void JASPExporter::saveDatabase(archive * a)
{
DatabaseInterface::singleton()->doWALCheckpoint();
saveTempFile(a, DatabaseInterface::singleton()->dbFile(true));
}
2 changes: 1 addition & 1 deletion Desktop/data/importers/csvimporter.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ CSVImporter::CSVImporter() : Importer()
ImportDataSet* CSVImporter::loadFile(const string &locator, std::function<void(int)> progressCallback)
{
JASPTIMER_RESUME(CSVImporter::loadFile);

ImportDataSet* result = new ImportDataSet(this);
stringvec colNames;
CSV csv(locator);
Expand Down
9 changes: 9 additions & 0 deletions Desktop/data/importers/importer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ Importer::~Importer() {}

void Importer::loadDataSet(const std::string &locator, std::function<void(int)> progressCallback)
{
long timeBeginS = Utils::currentSeconds();

DataSetPackage::pkg()->beginLoadingData();

_synching = false;
Expand Down Expand Up @@ -52,6 +54,9 @@ void Importer::loadDataSet(const std::string &locator, std::function<void(int)>
importDataSet->clearColumns();
delete importDataSet;
DataSetPackage::pkg()->endLoadingData();

long totalS = (Utils::currentSeconds() - timeBeginS);
Log::log() << "Loading '" << locator << "' took " << totalS << "s or " << (totalS / 60) << "m" << std::endl;
}

void Importer::initColumn(QVariant colId, ImportColumn *importColumn)
Expand All @@ -73,6 +78,7 @@ void Importer::initColumnWithStrings(QVariant colId, const std::string &newName,
void Importer::syncDataSet(const std::string &locator, std::function<void(int)> progress)
{
_synching = true;
long timeBeginS = Utils::currentSeconds();

ImportDataSet * importDataSet = loadFile(locator, progress);
bool rowCountChanged = importDataSet->rowCount() != DataSetPackage::pkg()->dataRowCount();
Expand Down Expand Up @@ -135,6 +141,9 @@ void Importer::syncDataSet(const std::string &locator, std::function<void(int)>

DataSetPackage::pkg()->setManualEdits(false);
delete importDataSet;

long totalS = (Utils::currentSeconds() - timeBeginS);
Log::log() << "Synching '" << locator << "' took " << totalS << "s or " << (totalS / 60) << "m" << std::endl;
}

void Importer::_syncPackage(
Expand Down

0 comments on commit 3102e9e

Please sign in to comment.