Anyway to save SQL DB to IndexedDB without calling export() for persistence?

[LeeRob97]

I am currently having to call export every time I need to save to indexedDB for persistence. The problem is export greatly increases the save time taken and I cannot save just the SQLJS object due to the new SQL.database constructor not being able to handle that? Is there any way around this or could this be a possible enhancement.

[theSherwood]

I'm also interested in this. Anyone have an idea what the barriers to implementation might be? I'm not familiar with sql.js but I'd be interested in using it if this problem can be sorted out. If someone could give me a heads up as to the what they think might be the best route for fixing this, I'd probably take a swing at it. Cheers!

[lovasoa]

Yes, or should not be too hard to implement this feature using

https://emscripten.org/docs/api_reference/Filesystem-API.html

If you are ready to work on a PR, I can assign you to this, @theSherwood !

[theSherwood]

I can give it a try. Just a heads up: I'm well outside my wheelhouse here. Maybe we should talk design. What kind of granularity would be right? Would every row in sqlLite be a document in IndexedDB? That sounds like the best way to reduce write volume for a database that needs to be persisted frequently. But to my mind, it also means coopting the run and exec commands to also write to indexedDB, which may not be ideal/may be better as a wrapper library.

@lovasoa, what are your thoughts on this? How would you go about it?

[lovasoa]

We should just use IDBFS from emscripten; you can read the documentation I linked to in my previous message. It shouldn't be hard to implement.

We could simply add an optional options argument to the Database constructor containing an optional database name. When it is provided, we don't an IDBDS and create a file with the given name in it.

[theSherwood]

Maybe I'm misunderstanding the way IDBFS works. Does it save each file as a separate document in IndexedDB? Or does it handle chunking out of the box?

[lovasoa]

no, it does not chunk files.

see https://github.com/emscripten-core/emscripten/blob/master/src/library_idbfs.js

[theSherwood]

Then if sql.js is treating the entire database as a single file, won't that defeat the purpose? It sounds like IDBFS would take the database and save it as a single document, which can also simply be done by using export and storing it in indexedDB manually? Or am I missing something?

[lovasoa]

no, you are getting it right

[theSherwood]

But how can that be done incrementally so as to not be saving the entire database to indexedDB at once, but only updating the records that changed and persisting each row in the sql.js database as a document in indexedDB? Do you have any idea on an approach for that?

[lovasoa]

This would be much more complex to implement. In practice, exporting the whole database and persisting it never takes more than a few seconds for huge databases, and it can be done in the background. Did you find a case where persisting the database faster would have a positive impact on the user ?

[kaizhu256]

i agree with lovasoa its probably not feasible to re-invent new mechanism to persist individual sqlite-records to indexeddb. what we really need are native file-api's like "seek" exposed in browser to exploit sqlite's native file-persistence mechanism (like wal).

it appears google is working on nativeio filesystem for browsers @ https://github.com/fivedots/nativeio-explainer. they have an experimental sqlite-wasm implementation @ https://github.com/jabolopes/sqlite-wasm

i managed to compile the demo @ https://kaizhu256.github.io/sqlite-wasm/dist/index.html, but couldn't get the nativeio benchmark working.