Breaking API changes to land soon

[rhashimoto]

UPDATE 2023-04-21: The changes have been merged to the trunk.

I'm going to make a few API changes, not big changes but incompatible with some existing usage. Work is currently proceeding in a branch, and I'll wait at least a few weeks to merge it. But please keep an eye on this thread (and maybe the branch also) if you may be affected so you aren't surprised when you update and your code breaks.

The changes correct some poor design and an ugly hack in the existing APIs. Migration should be straightforward and you can use the repo examples as a guide, but if anything is unclear post questions here.

Here's a list of intended changes:

SQLite3, i.e. the main programming API

• Who will be affected? Anyone writing programs that use blobs.
• What is changing? Blobs are changing from Int8Array to Uint8Array. Any API call that receives a blob as an argument (e.g. bind_blob()) or returns a blob (e.g. column_blob()) will use an Uint8Array for the blob value.
• Why is it changing? This was a bad choice. It's more painful debugging signed raw bytes than unsigned raw bytes.
• Status: Completed in the branch.

Virtual File System API

• Who will be affected? Anyone using their own VFS class for custom storage, including those who have copied an example VFS into their tree.

• What is changing?

  • xRead and xWrite buffers will be passed as Uint8Array instead of a weird ad-hoc object containing a Int8Array.
  • Pointers to values to be set by xAccess, xFileSize, xCheckReservedLock, and xOpen, will be passed as DataView instead of a weird ad-hoc object containing a setter.
  • Operation-dependent data in xFileControl will be passed as DataView instead of a weird ad-hoc object containing a Int8Array.

  See the MemoryVFS.js diffs for examples.

• Why is it changing? It was bad design. The objects were weird and clumsy, and can't be structured cloned for proxying over a MessageChannel.

• Status: Completed in the branch.

Module API

• Who will be affected? Anyone writing module classes, aka virtual tables.
• What is changing? Currently table schemas in xCreate and xConnect are specified by returning them in the error string. This was a lazy and sloppy way around the dilemma that these methods are supposed to call the SQLite function declare_vtab but Asyncify doesn't allow calling back to WebAssembly while it is suspended. If you have an existing module with an asynchronous xCreate/xConnect, it turns out there is a less horrible workaround for that. If you have an existing module with a synchronous xCreate/xConnect, it should be a 1-line change to use declare_vtab.
• Why is it changing? Yuck.
• Status: Completed in the branch.

[rhashimoto]

I was targeting March 14 to merge these changes to coincide with Firefox 111, which shipped worker ES6 module support. Unfortunately, I didn't realize that support wouldn't include dynamic import or that it would break with Dev Tools open.

Although the library itself doesn't require dynamic import, changes I made to the demo and intend to make to the benchmarks do use dynamic import in ES6 module workers. And if there are any Firefox-specific issues with that, I don't want to investigate without Dev Tools available. So I'm waiting until these Firefox bugs are fixed, hopefully by Firefox 113 scheduled for May 9.

Update 4/12/23: Demos and benchmarks work in the latest Firefox Nightly so it looks like things are on track for Firefox 113.

[rhashimoto]

The merge has been done. The new APIs are in place, WASM artifacts are in dist/, and the buildless branch is obsolete. Please be prepared to make the appropriate changes to your code when you update.

Unfortunately, ES6 module workers are not enabled in the Firefox beta, so it didn't look possible for them to appear in the next release, especially considering their primary developer is on leave until October. I removed the rewritten and new demos that need module workers from the merge. You can still access that source code on the breaking-changes branch and they are live at an alternate path:

• demo/
• demo/benchmarks.html
• demo/contention.html

UPDATE 5/10/23: Firefox module workers are targeted for Firefox 114, scheduled for 6/6/23.
UPDATE 6/11/23: New demos still don't work on Firefox 114. They do work on Firefox 115 beta scheduled to go stable 7/4/23 so maybe then?