[GH-ISSUE #24] Storing a complete object in a single db row/value.

gitea-mirror commented

2026-05-23 08:25:53 -06:00

Owner

Originally created by @clibu on GitHub (Mar 22, 2021).
Original GitHub issue: https://github.com/appy-one/acebase/issues/24

I understand that Acebase stores certain objects properties in multiple db rows to speed up read/write performance. These include object properties whose value is longer than maxInlineValueSize and nested or child objects.

In the app I'm working on I store complex JSON documents in the db. Even a small document has resulted in 60 rows in the db. These JSON documents are always retrieved and saved in their entirety ie. as single JSON objects.

My concerns here are the considerable extra space required to save a document along with the overhead of breaking up and reassembling the documents. The current Browser DB I use, nanoSQL stores all values in a single db row.

Here is an example of part of an JSON Doc in IndexedDb:

I could JSON.stringify() etc. which should resolve this issue but that seems like unnecessary extra overhead and on the flip side doing JSON.parse(). The DB is updated frequently as the user is editing a document, so performance is important.

I think you can see where this is going.😀

Would you consider providing an option to store value's in a single db row. This could be an AceBase constructor option and/or a update() and set() or even a ref option. What do you think?

Originally created by @clibu on GitHub (Mar 22, 2021). Original GitHub issue: https://github.com/appy-one/acebase/issues/24 I understand that Acebase stores certain objects properties in multiple db rows to speed up read/write performance. These include object properties whose value is longer than ``maxInlineValueSize`` and nested or child objects. In the app I'm working on I store complex JSON documents in the db. Even a small document has resulted in 60 rows in the db. These JSON documents are always retrieved and saved in their entirety ie. as single JSON objects. My concerns here are the considerable extra space required to save a document along with the overhead of breaking up and reassembling the documents. The current Browser DB I use, nanoSQL stores all values in a single db row. Here is an example of part of an JSON Doc in IndexedDb: ![image](https://user-images.githubusercontent.com/12797550/111953320-8de6a880-8b3a-11eb-9e14-86529425875d.png) I could JSON.stringify() etc. which should resolve this issue but that seems like unnecessary extra overhead and on the flip side doing JSON.parse(). The DB is updated frequently as the user is editing a document, so performance is important. I think you can see where this is going.😀 Would you consider providing an option to store value's in a single db row. This could be an AceBase constructor option and/or a ``update()`` and ``set()`` or even a ``ref`` option. What do you think?

gitea-mirror closed this issue

2026-05-23 08:25:54 -06:00

gitea-mirror commented

2026-05-23 08:25:57 -06:00

Author

Owner

@appy-one commented on GitHub (Mar 22, 2021):

I see you are using lots of arrays, even nested. Arrays are a pain for synchronization, because they require to be rewritten entirely when items are inserted, removed, or sorted. For example, when removing an array item at index 2, all items with index 2 and up will have different data and have to be rewritten. When using a live data proxy, array rewriting is done automatically behind the scenes by rewriting the topmost array that has a change in its tree. But this is expensive and can be prevented.

If you'd use object collections instead, they would have their own id's so they can be changed, removed and inserted individually, which is ideal for synchronization and performance: only changing objects have to be updated/inserted/deleted and synchronized. Storing those objects in their own records is key to make this fast and efficient. Yes, a couple more bytes needed for storage, but the user is able to make a change to "clibu_notes/id/value/content/id/content/id/marks/id/attributes/id" and only 1 small string value has to be overwritten in the database and synchronized with the server and other clients. This is what makes realtime databases fast - tiny updates that flow from a to b (and c, d, e) with the speed of light. It also prevents many synchronization conflicts, because it's not entire documents being synchronized, only the actual data being changed.

Note that if you'd JSON.stringify objects for storage in 1 record, you will not be able to use them as objects with a live data proxy because the value is a string. If you'd JSON.parse that string, updating properties will not update the database because it's not a proxied object. Also, updating child properties through db.ref('object/child/prop').set(...) will not be possible anymore.

I'll think about your request some more, but right now I don't see the added value.
How much data are you typically storing in the browser database?

@appy-one commented on GitHub (Mar 22, 2021): I see you are using lots of arrays, even nested. Arrays are a pain for synchronization, because they require to be rewritten entirely when items are inserted, removed, or sorted. For example, when removing an array item at index 2, all items with index 2 and up will have different data and have to be rewritten. When using a live data proxy, array rewriting is done automatically behind the scenes by rewriting the topmost array that has a change in its tree. But this is expensive and can be prevented. If you'd use object collections instead, they would have their own id's so they can be changed, removed and inserted individually, which is ideal for synchronization and performance: only changing objects have to be updated/inserted/deleted and synchronized. Storing those objects in their own records is key to make this fast and efficient. Yes, a couple more bytes needed for storage, but the user is able to make a change to `"clibu_notes/id/value/content/id/content/id/marks/id/attributes/id"` and only 1 small string value has to be overwritten in the database and synchronized with the server and other clients. This is what makes realtime databases fast - tiny updates that flow from a to b (and c, d, e) with the speed of light. It also prevents many synchronization conflicts, because it's not entire documents being synchronized, only the actual data being changed. Note that if you'd `JSON.stringify` objects for storage in 1 record, you will not be able to use them as objects with a live data proxy because the value is a string. If you'd `JSON.parse` that string, updating properties will not update the database because it's not a proxied object. Also, updating child properties through `db.ref('object/child/prop').set(...)` will not be possible anymore. I'll think about your request some more, but right now I don't see the added value. How much data are you typically storing in the browser database?

gitea-mirror commented

2026-05-23 08:25:58 -06:00

Author

Owner

@clibu commented on GitHub (Mar 22, 2021):

Ok, I should have provided some more detail. I don't have any control over the content of the JSON documents. These come from the Prosemirror editor. Document Structure toJSON() function. Also I'm not using a live data proxy for this, just update().

Further I have no interest or need to access nested document content in the database ex. "clibu_notes/id/value/content/id/content/id/marks/id/attributes/id"

Prosemirror has a transaction callback that is called every time the editor document changes. I debounce these calls and every 500ms save the current JSON document to the database.

I have no control of and don't impose any limits on document size or the number of documents a user can create and edit. This will be constrained by the amount of storage their device has available to the Browser.

The Browser database is synchronized to the server in real time if they are connected to it, otherwise changes are synchronized via. an append-only-log next time they connect. So if they haven't connected for some time they could potentially loose all work since their last connection if the Browser evicts the apps data. That said they can also export and therefore back the Browser DB.

Hopefully that sheds some more light on this use case.

@clibu commented on GitHub (Mar 22, 2021): Ok, I should have provided some more detail. I don't have any control over the content of the JSON documents. These come from the [Prosemirror](https://prosemirror.net/) editor. [Document Structure](https://prosemirror.net/docs/ref/#model.Document_Structure) toJSON() function. Also I'm not using a live data proxy for this, just ``update()``. Further I have no interest or need to access nested document content in the database ex. ```"clibu_notes/id/value/content/id/content/id/marks/id/attributes/id"``` Prosemirror has a transaction callback that is called every time the editor document changes. I debounce these calls and every 500ms save the current JSON document to the database. I have no control of and don't impose any limits on document size or the number of documents a user can create and edit. This will be constrained by the amount of storage their device has available to the Browser. The Browser database is synchronized to the server in real time if they are connected to it, otherwise changes are synchronized via. an append-only-log next time they connect. So if they haven't connected for some time they could potentially loose all work since their last connection if the Browser evicts the apps data. That said they can also export and therefore back the Browser DB. Hopefully that sheds some more light on this use case.

gitea-mirror commented

2026-05-23 08:26:00 -06:00

Author

Owner

@clibu commented on GitHub (Mar 22, 2021):

PS. These documents can contain images and attachments (PDF's etc.) so a single doc could easily be in the MB's.

@clibu commented on GitHub (Mar 22, 2021): PS. These documents can contain images and attachments (PDF's etc.) so a single doc could easily be in the MB's.

gitea-mirror commented

2026-05-23 08:26:01 -06:00

Author

Owner

@appy-one commented on GitHub (Mar 22, 2021):

Thanks for the context info.. In that case, the JSON.stringify route might not be a bad idea. Where are you syncing to? If it's not an AceBase server, may I ask why you would switch from nanoSQL to AceBase for the browser storage of those text notes?

@appy-one commented on GitHub (Mar 22, 2021): Thanks for the context info.. In that case, the JSON.stringify route might not be a bad idea. Where are you syncing to? If it's not an AceBase server, may I ask why you would switch from nanoSQL to AceBase for the browser storage of those text notes?

gitea-mirror commented

2026-05-23 08:26:05 -06:00

Author

Owner

@clibu commented on GitHub (Mar 22, 2021):

My current live app Clibu uses MongoDB on the server and has no offline capability. The app I'm working on to replace it has been designed to be offline from the very start. I also want to make it easier for people to host it on their own server or home PC. For that reason I've abstracted the server db to make it easy to switch db engines.

My prime interest in AceBase is that it can be embedded in a node app and the user doesn't need to install a separate db like MongoDB. Also AceBase capabilities are impressive. So I started by using AceBase on the server and that works fine.

Next I wanted to get a feel for how well or otherwise AceBase worked in the Browser. I started by replacing a Store which tracks and manages complex UI state etc. That Store used nanoSql along with on-change which "watches an object or array for changes". This combo worked similarly to AceBase's live proxy but is somewhat messier.

The Store rewrite to use AceBase has gone well and is catering for a more complex store than the original. The live proxy code worked perfectly without me having to change my code that previously used on-change. I've also written new user management code built on AceBase in the Browser.

The next step was to see if AceBase can replace the Notes documents db, which is where this post started. Ideally I'd use one database lib for everything, to reduce code size, and for ease of development and support. Right now I've got AceBase, NanoSQL and a small IndexedDb lib which is part of Logux.io

Finally the capabilities that AceBase provides better addresses my needs than NanoSQL and being newer and actively worked on and supported is also attractive.

@clibu commented on GitHub (Mar 22, 2021): My current live app [Clibu](clibu.com) uses MongoDB on the server and has no offline capability. The app I'm working on to replace it has been designed to be offline from the very start. I also want to make it easier for people to host it on their own server or home PC. For that reason I've abstracted the server db to make it easy to switch db engines. My prime interest in AceBase is that it can be embedded in a node app and the user doesn't need to install a separate db like MongoDB. Also AceBase capabilities are impressive. So I started by using AceBase on the server and that works fine. Next I wanted to get a feel for how well or otherwise AceBase worked in the Browser. I started by replacing a Store which tracks and manages complex UI state etc. That Store used nanoSql along with [on-change](https://www.npmjs.com/package/on-change) which "watches an object or array for changes". This combo worked similarly to AceBase's live proxy but is somewhat messier. The Store rewrite to use AceBase has gone well and is catering for a more complex store than the original. The live proxy code worked perfectly without me having to change my code that previously used on-change. I've also written new user management code built on AceBase in the Browser. The next step was to see if AceBase can replace the Notes documents db, which is where this post started. Ideally I'd use one database lib for everything, to _reduce code size_, and for _ease of development and support_. Right now I've got AceBase, NanoSQL and a small IndexedDb lib which is part of Logux.io Finally the capabilities that AceBase provides better addresses my needs than NanoSQL and being newer and actively worked on and supported is also attractive.

gitea-mirror commented

2026-05-23 08:26:06 -06:00

Author

Owner

@appy-one commented on GitHub (Mar 22, 2021):

That is a great testimonial, thanks! Happy to help, things can only get better! 🙂

@appy-one commented on GitHub (Mar 22, 2021): That is a great testimonial, thanks! Happy to help, things can only get better! 🙂

gitea-mirror commented

2026-05-23 08:26:08 -06:00

Author

Owner

@clibu commented on GitHub (Mar 22, 2021):

Yes, agreed and I'm very happy to help as best I can. So you'll think about my single row request then.😀

I've can foresee a couple more issues converting from NanoSQL to AceBase with functionality which may not (yet) be present in AceBase and will report as I progress.

@clibu commented on GitHub (Mar 22, 2021): Yes, agreed and I'm very happy to help as best I can. So you'll think about my single row request then.😀 I've can foresee a couple more issues converting from NanoSQL to AceBase with functionality which may not _(yet)_ be present in AceBase and will report as I progress.

gitea-mirror commented

2026-05-23 08:26:09 -06:00

Author

Owner

@appy-one commented on GitHub (Mar 28, 2021):

I was thinking... What you could do to have those notes stored in 1 record is having them serialized automatically using a type mapping. Code below will cause any object stored in clibu_notes to automatically be serialized and persisted as a string (1 record), and be instantiated to your object again when read. To save even more space you could even choose to compress the json string for storage.

db.types.bind(
    "clibu_notes", 
    Object, { 
        serializer: (ref, note) => {
            // Serialize note for storage
            let json = JSON.stringify(note);
            // optionally compress json here
            return json;
        },
        creator: (snap) => {
            // Deserialize stored string to object again
            let json = snap.val();
            // optionally decompress here
            return JSON.parse(json);
        }
    }
);

// Now you can do this:
await db.ref('clibu_notes/someid').set(noteObject); // Persisted as string
// And:
let note = await db.ref('clibu_notes/someid').get(); // Deserialized to your object automatically

@appy-one commented on GitHub (Mar 28, 2021): I was thinking... What you could do to have those notes stored in 1 record is having them serialized automatically using a [type mapping](https://www.npmjs.com/package/acebase#mapping-data-to-custom-classes). Code below will cause any object stored in `clibu_notes` to automatically be serialized and persisted as a string (1 record), and be instantiated to your object again when read. To save even more space you could even choose to compress the json string for storage. ```js db.types.bind( "clibu_notes", Object, { serializer: (ref, note) => { // Serialize note for storage let json = JSON.stringify(note); // optionally compress json here return json; }, creator: (snap) => { // Deserialize stored string to object again let json = snap.val(); // optionally decompress here return JSON.parse(json); } } ); // Now you can do this: await db.ref('clibu_notes/someid').set(noteObject); // Persisted as string // And: let note = await db.ref('clibu_notes/someid').get(); // Deserialized to your object automatically ```

gitea-mirror commented

2026-05-23 08:26:11 -06:00

Author

Owner

@clibu commented on GitHub (Mar 29, 2021):

Thanks, that would be helpful however there is only one property value that needs to be stored in one record. The other "clibu_notes" properties are stored normally.

It is easy enough for me to do the JSON.stringify() etc. however my concerns are performance and the possibility the JSON conversion process could loose or change values.

@clibu commented on GitHub (Mar 29, 2021): Thanks, that would be helpful however there is only one property value that needs to be stored in one record. The other "clibu_notes" properties are stored normally. It is easy enough for me to do the ``JSON.stringify()`` etc. however my concerns are performance and the possibility the JSON conversion process could loose or change values.

gitea-mirror commented

2026-05-23 08:26:12 -06:00

Author

Owner

@appy-one commented on GitHub (Mar 30, 2021):

I'm confused. I thought you said there was no limit to the number of documents the user could store, and now it's only one? You were worried about storage?

@appy-one commented on GitHub (Mar 30, 2021): I'm confused. I thought you said there was no limit to the number of documents the user could store, and now it's only one? You were worried about storage?

gitea-mirror commented

2026-05-23 08:26:13 -06:00

Author

Owner

@clibu commented on GitHub (Mar 30, 2021):

Sorry for the confustion. A clibu_note value looks like this:

{ id: string   
  path: array,
  title: Object
  userSortKey: Object
  body: { large json document },
  dateCreated: isoDate
  ...
  ...
}

There can be an unlimited number of these records. The body value is the large json document that I wont stored as a single value / row. All the other value's need to be stored as is.

Hopefully that clears things up.

@clibu commented on GitHub (Mar 30, 2021): Sorry for the confustion. A clibu_note value looks like this: ```` { id: string path: array, title: Object userSortKey: Object body: { large json document }, dateCreated: isoDate ... ... } ```` There can be an unlimited number of these records. The ``body`` value is the large json document that I wont stored as a single value / row. All the other value's need to be stored as is. Hopefully that clears things up.

gitea-mirror commented

2026-05-23 08:26:14 -06:00

Author

Owner

@appy-one commented on GitHub (Mar 30, 2021):

Aha ok.. You could serialize/deserialize the objects differently then, only changing the body property:

db.types.bind(
    "clibu_notes", 
    Object, { 
        serializer: (ref, note) => {
            // Serialize note for storage.
            note.body = JSON.stringify(note.body); // Only stringify body
            return note;
        },
        creator: (snap) => {
            // Deserialize
            const note = snap.val();
            note.body = JSON.parse(note.body);
            return note;
        }
    }
);

The only catch in above example is that serializer alters the original note object, which in your case would not likely be a problem since it was generated by your editor's toJSON function. Cloning the original object before making changes to it would be advisable in other circumstances: const clone = {}; Object.assign(clone, note); clone.body = JSON.stringify(clone.body); return clone;

@appy-one commented on GitHub (Mar 30, 2021): Aha ok.. You could serialize/deserialize the objects differently then, only changing the body property: ```js db.types.bind( "clibu_notes", Object, { serializer: (ref, note) => { // Serialize note for storage. note.body = JSON.stringify(note.body); // Only stringify body return note; }, creator: (snap) => { // Deserialize const note = snap.val(); note.body = JSON.parse(note.body); return note; } } ); ``` The only catch in above example is that `serializer` alters the original note object, which in your case would not likely be a problem since it was generated by your editor's toJSON function. Cloning the original object before making changes to it would be advisable in other circumstances: `const clone = {}; Object.assign(clone, note); clone.body = JSON.stringify(clone.body); return clone;`

Rows
Columns

[GH-ISSUE #24] Storing a complete object in a single db row/value. #23