[GH-ISSUE #67] Indices with custom storage not working

gitea-mirror commented

2026-05-23 08:28:10 -06:00

Owner

Originally created by @gryphonmyers on GitHub (Feb 5, 2022).
Original GitHub issue: https://github.com/appy-one/acebase/issues/67

Originally assigned to: @appy-one on GitHub.

I have written a custom storage class and it seems to be working just fine, except that if I try to create an index I get file access errors indicating it is trying to use non-custom storage for those operations, e.g.:

const db = new AceBase('testdb', { logLevel: 'error', storage: testStorage  })
await db.indexes.create("*", "sys/id")

yields:

  Error {
    code: 'ENOENT',
    errno: -2,
    path: './testdb.acebase/[undefined]-#-sys/id.idx.build',
    syscall: 'open',
    message: 'ENOENT: no such file or directory, open \'./testdb.acebase/[undefined]-#-sys/id.idx.build\'',
  }

You can see it seems to be trying to open a .acebase file but there isn't one in this case. Is this a known issue? Am I doing something wrong?

Originally created by @gryphonmyers on GitHub (Feb 5, 2022). Original GitHub issue: https://github.com/appy-one/acebase/issues/67 Originally assigned to: @appy-one on GitHub. I have written a custom storage class and it seems to be working just fine, except that if I try to create an index I get file access errors indicating it is trying to use non-custom storage for those operations, e.g.: ```javascript const db = new AceBase('testdb', { logLevel: 'error', storage: testStorage }) await db.indexes.create("*", "sys/id") ``` yields: ``` Error { code: 'ENOENT', errno: -2, path: './testdb.acebase/[undefined]-#-sys/id.idx.build', syscall: 'open', message: 'ENOENT: no such file or directory, open \'./testdb.acebase/[undefined]-#-sys/id.idx.build\'', } ``` You can see it seems to be trying to open a .acebase file but there isn't one in this case. Is this a known issue? Am I doing something wrong?

gitea-mirror

2026-05-23 08:28:10 -06:00

closed this issue
added the
bug

indexes

feature request
labels

gitea-mirror commented

2026-05-23 08:28:14 -06:00

Author

Owner

@appy-one commented on GitHub (Feb 5, 2022):

Creating indexes should work for custom storage, the only requirement is that the filesystem is available. You are getting an error because the key to index contains a slash, which is not allowed.

I don't know what data you are trying to index, but your target doesn't seem right. If you want to index every id of every sys child of every root child (eg "/child1/sys/id", "/child2/sys/id", "/child34/sys/id" etc) I'm afraid that is not possible. The key to index cannot be a path, and the index path is expected to be a collection. To work around this limitation, you'd have to move the id to reside in the index path target: "/child1/sys_id", "/child23/sys_id" etc and then create the index on "", "sys_id" - that will index "*/sys_id".

I do wonder where the [undefined] in the target file path originates from, I will check that out.

I'm curious what type of custom storage you created, would you like to share?

@appy-one commented on GitHub (Feb 5, 2022): Creating indexes should work for custom storage, the only requirement is that the filesystem is available. You are getting an error because the key to index contains a slash, which is not allowed. I don't know what data you are trying to index, but your target doesn't seem right. If you want to index every `id` of every `sys` child of every root child (eg `"/child1/sys/id"`, `"/child2/sys/id"`, `"/child34/sys/id"` etc) I'm afraid that is not possible. The key to index cannot be a path, and the index path is expected to be a collection. To work around this limitation, you'd have to move the `id` to reside in the index path target: `"/child1/sys_id"`, `"/child23/sys_id"` etc and then create the index on `"", "sys_id"` - that will index `"*/sys_id"`. I do wonder where the `[undefined]` in the target file path originates from, I will check that out. I'm curious what type of custom storage you created, would you like to share?

gitea-mirror commented

2026-05-23 08:28:15 -06:00

Author

Owner

@gryphonmyers commented on GitHub (Feb 5, 2022):

It doesn't work if I remove the slash either:

  const db = new AceBase('testdb', { logLevel: 'error', storage: testStorage  })
  await db.indexes.create("*", "id")

Error:

  Error {
    code: 'ENOENT',
    errno: -2,
    path: './testdb.acebase/[undefined]-#-id.idx.build',
    syscall: 'open',
    message: 'ENOENT: no such file or directory, open \'./testdb.acebase/[undefined]-#-id.idx.build\'',
  }

This is the storage class I'm using:


export function createTestStorage() {
  const data :Record<string, ICustomStorageNode> = {};

  class TestStorageTransaction extends CustomStorageTransaction {
    async commit() {
      return;
    }
    async rollback() {
      return;
    }
    async set(path: string, value: ICustomStorageNode) {
      data[path] = value;
    }
    async remove(path: string) {
      delete data[path]
    }
    async get(path: string) {
      return data[path] || null;
    }
    async childrenOf(path: string, include: { value: boolean, metadata: boolean }, checkCallback: Function, addCallback: Function) {
      const pathInfo = CustomStorageHelpers.PathInfo.get(path);
      
      for (var key in data) {
        const otherPath = key;
        if (pathInfo.isParentOf(otherPath) && checkCallback(otherPath)) {
          let node;
          if (include.metadata || include.value) {
            node = data[key]
          }
          const keepGoing = addCallback(otherPath, node);
          if (!keepGoing) { break; }
        }
      }
    }
    async descendantsOf(path: string, include: { value: boolean, metadata: boolean }, checkCallback: Function, addCallback: Function) {
      const pathInfo = CustomStorageHelpers.PathInfo.get(path);
      
      for (var key in data) {
        const otherPath = key;
        if (pathInfo.isAncestorOf(otherPath) && checkCallback(otherPath)) {
          let node;
          if (include.metadata || include.value) {
            node = data[key]
          }
          const keepGoing = addCallback(otherPath, node);
          if (!keepGoing) { break; }
        }
      }
    }
  }
  
  return new CustomStorageSettings({
    name: 'TestStorage',
    locking: false, // Let AceBase handle resource locking to prevent multiple simultanious updates to the same data
    ready() {
        return Promise.resolve();
    },
    async getTransaction(target) {
        const transaction = new TestStorageTransaction(target);
        return Promise.resolve(transaction);
    }
  });
}

As an aside, that is unfortunate that the slash is unsupported because I'm working with a third party structure that put its primary keys in nested objects. This is the structure I am forced to work with. Note that it has the top level keys "metadata", "sys", "fields" and all data is stored under these (including the primary key sys.id). Acebase should have adequate functionality to let me query this data, as you've already demonstrated, but the situation I'm trying to figure out now is how to resolve these entry links (see under fields.content below - it's an object with an id but no reference to which collection it is referencing). This is why I was trying to create an index so I could do a wildcard query to find the referenced entry, but apparently that's not going to work out for me if indices can't be created on nested keys. Do you have any other suggestions for how I might accomplish this without modifying the structure (as I mentioned this is a structure coming from a third party tool and I'd like to preserve compatibility with it)?

    {
    "metadata": {
      "tags": [
        {
          "sys": {
            "type": "Link",
            "linkType": "Tag",
            "id": "n22"
          }
        }
      ]
    },
    "sys": {
      "space": {
        "sys": {
          "type": "Link",
          "linkType": "Space",
          "id": "1a6ngf98576c"
        }
      },
      "id": "7zSFH41eNtOzNPP8a4gmxJ",
      "type": "Entry",
      "createdAt": "2021-09-28T19:15:28.275Z",
      "updatedAt": "2022-01-28T15:48:27.626Z",
      "environment": {
        "sys": {
          "id": "master",
          "type": "Link",
          "linkType": "Environment"
        }
      },
      "revision": 24,
      "contentType": {
        "sys": {
          "type": "Link",
          "linkType": "ContentType",
          "id": "BcLink"
        }
      }
    },
    "fields": {
      "internalName": {
        "en-US": "N22 | Mobile | Social Sharing Facebook"
      },
      "content": {
        "en-US": {
          "sys": {
            "type": "Link",
            "linkType": "Entry",
            "id": "4R7vPQnJf05PyMVeLDyy2V"
          }
        }
      },
      "title": {
        "en-US": "NBA 2K Mobile Facebook "
      },
      "href": {
        "en-US": "https://www.facebook.com/NBA2KMobile/"
      }
    }
  }

@gryphonmyers commented on GitHub (Feb 5, 2022): It doesn't work if I remove the slash either: ```javascript const db = new AceBase('testdb', { logLevel: 'error', storage: testStorage }) await db.indexes.create("*", "id") ``` Error: ``` Error { code: 'ENOENT', errno: -2, path: './testdb.acebase/[undefined]-#-id.idx.build', syscall: 'open', message: 'ENOENT: no such file or directory, open \'./testdb.acebase/[undefined]-#-id.idx.build\'', } ``` This is the storage class I'm using: ```javascript export function createTestStorage() { const data :Record<string, ICustomStorageNode> = {}; class TestStorageTransaction extends CustomStorageTransaction { async commit() { return; } async rollback() { return; } async set(path: string, value: ICustomStorageNode) { data[path] = value; } async remove(path: string) { delete data[path] } async get(path: string) { return data[path] || null; } async childrenOf(path: string, include: { value: boolean, metadata: boolean }, checkCallback: Function, addCallback: Function) { const pathInfo = CustomStorageHelpers.PathInfo.get(path); for (var key in data) { const otherPath = key; if (pathInfo.isParentOf(otherPath) && checkCallback(otherPath)) { let node; if (include.metadata || include.value) { node = data[key] } const keepGoing = addCallback(otherPath, node); if (!keepGoing) { break; } } } } async descendantsOf(path: string, include: { value: boolean, metadata: boolean }, checkCallback: Function, addCallback: Function) { const pathInfo = CustomStorageHelpers.PathInfo.get(path); for (var key in data) { const otherPath = key; if (pathInfo.isAncestorOf(otherPath) && checkCallback(otherPath)) { let node; if (include.metadata || include.value) { node = data[key] } const keepGoing = addCallback(otherPath, node); if (!keepGoing) { break; } } } } } return new CustomStorageSettings({ name: 'TestStorage', locking: false, // Let AceBase handle resource locking to prevent multiple simultanious updates to the same data ready() { return Promise.resolve(); }, async getTransaction(target) { const transaction = new TestStorageTransaction(target); return Promise.resolve(transaction); } }); } ``` As an aside, that is unfortunate that the slash is unsupported because I'm working with a third party structure that put its primary keys in nested objects. This is the structure I am forced to work with. Note that it has the top level keys "metadata", "sys", "fields" and all data is stored under these (including the primary key `sys.id`). Acebase should have adequate functionality to let me query this data, as you've already demonstrated, but the situation I'm trying to figure out now is how to resolve these entry links (see under `fields.content` below - it's an object with an id _but no reference to which collection it is referencing_). This is why I was trying to create an index so I could do a wildcard query to find the referenced entry, but apparently that's not going to work out for me if indices can't be created on nested keys. Do you have any other suggestions for how I might accomplish this without modifying the structure (as I mentioned this is a structure coming from a third party tool and I'd like to preserve compatibility with it)? ```json { "metadata": { "tags": [ { "sys": { "type": "Link", "linkType": "Tag", "id": "n22" } } ] }, "sys": { "space": { "sys": { "type": "Link", "linkType": "Space", "id": "1a6ngf98576c" } }, "id": "7zSFH41eNtOzNPP8a4gmxJ", "type": "Entry", "createdAt": "2021-09-28T19:15:28.275Z", "updatedAt": "2022-01-28T15:48:27.626Z", "environment": { "sys": { "id": "master", "type": "Link", "linkType": "Environment" } }, "revision": 24, "contentType": { "sys": { "type": "Link", "linkType": "ContentType", "id": "BcLink" } } }, "fields": { "internalName": { "en-US": "N22 | Mobile | Social Sharing Facebook" }, "content": { "en-US": { "sys": { "type": "Link", "linkType": "Entry", "id": "4R7vPQnJf05PyMVeLDyy2V" } } }, "title": { "en-US": "NBA 2K Mobile Facebook " }, "href": { "en-US": "https://www.facebook.com/NBA2KMobile/" } } } ```

gitea-mirror commented

2026-05-23 08:28:16 -06:00

Author

Owner

@appy-one commented on GitHub (Feb 6, 2022):

Thanks for the details, I see you created a cool in-memory storage class there!👌🏼

I'll have to dive into the indexing (and query) code to see if there is a way to implement nested key indexing. Implementing such a feature would obviously take some time. If you want me to prioritise this, please consider sponsoring me for my efforts! 😇

I will check out why it fails to create the index file for a custom storage class asap.

@appy-one commented on GitHub (Feb 6, 2022): Thanks for the details, I see you created a cool in-memory storage class there!👌🏼 I'll have to dive into the indexing (and query) code to see if there is a way to implement nested key indexing. Implementing such a feature would obviously take some time. If you want me to prioritise this, please consider sponsoring me for my efforts! 😇 I will check out why it fails to create the index file for a custom storage class asap.

gitea-mirror commented

2026-05-23 08:28:19 -06:00

Author

Owner

@appy-one commented on GitHub (Feb 20, 2022):

I think there is good news for you on the horizon! I changed parts of the code, looks like indexing and querying on subkeys will become possible...

@appy-one commented on GitHub (Feb 20, 2022): I think there is good news for you on the horizon! I changed parts of the code, looks like indexing and querying on subkeys will become possible...

gitea-mirror commented

2026-05-23 08:28:19 -06:00

Author

Owner

@appy-one commented on GitHub (Feb 21, 2022):

I just published acebase version 1.15.0, The error should be gone and indexing on 'sys/id' possible: await db.indexes.create('', 'sys/id'). Let me know if it all works!

@appy-one commented on GitHub (Feb 21, 2022): I just published acebase version 1.15.0, The error should be gone and indexing on 'sys/id' possible: `await db.indexes.create('', 'sys/id')`. Let me know if it all works!

gitea-mirror commented

2026-05-23 08:28:20 -06:00

Author

Owner

@gryphonmyers commented on GitHub (Feb 22, 2022):

Awesome! Will check it out

@gryphonmyers commented on GitHub (Feb 22, 2022): Awesome! Will check it out

gitea-mirror commented

2026-05-23 08:28:20 -06:00

Author

Owner

@gryphonmyers commented on GitHub (Feb 27, 2022):

@appy-one hmm so after updating to 1.15.1 things have changed a bit, but still not working for my use case. It now seems to be generating a testdb.acebase directory in the same directory as the script where I'm creating the index (before it was just complaining that the testdb.acebase file didn't exist, now it's actually creating it even though I'm using custom storage). After creating this directory with an index file within it, if I run my test a second time I get this error:

[testdb] Error: Index "./testdb.acebase/#-sys~id.idx" version 0 is not supported by this version of AceBase. npm update your acebase packages
    at Function.readFromFile (/Users/gryphon.myers@2k.com/code/git-repos/content-service-client/node_modules/acebase/src/data-index.js:269:23)
    at async Object.add (/Users/gryphon.myers@2k.com/code/git-repos/content-service-client/node_modules/acebase/src/storage.js:309:35)
    at async Promise.all (index 0)
    at async Object.load (/Users/gryphon.myers@2k.com/code/git-repos/content-service-client/node_modules/acebase/src/storage.js:304:17)
    at async CustomStorage._init (/Users/gryphon.myers@2k.com/code/git-repos/content-service-client/node_modules/acebase/src/storage-custom.js:369:13)

@gryphonmyers commented on GitHub (Feb 27, 2022): @appy-one hmm so after updating to 1.15.1 things have changed a bit, but still not working for my use case. It now seems to be generating a testdb.acebase directory in the same directory as the script where I'm creating the index (before it was just complaining that the testdb.acebase file didn't exist, now it's actually creating it even though I'm using custom storage). After creating this directory with an index file within it, if I run my test a second time I get this error: ``` [testdb] Error: Index "./testdb.acebase/#-sys~id.idx" version 0 is not supported by this version of AceBase. npm update your acebase packages at Function.readFromFile (/Users/gryphon.myers@2k.com/code/git-repos/content-service-client/node_modules/acebase/src/data-index.js:269:23) at async Object.add (/Users/gryphon.myers@2k.com/code/git-repos/content-service-client/node_modules/acebase/src/storage.js:309:35) at async Promise.all (index 0) at async Object.load (/Users/gryphon.myers@2k.com/code/git-repos/content-service-client/node_modules/acebase/src/storage.js:304:17) at async CustomStorage._init (/Users/gryphon.myers@2k.com/code/git-repos/content-service-client/node_modules/acebase/src/storage-custom.js:369:13) ```

gitea-mirror commented

2026-05-23 08:28:21 -06:00

Author

Owner

@appy-one commented on GitHub (Feb 28, 2022):

Yes, indexes are created on disk regardless of your custom storage implementation. If you'd switch to AceBase's default binary storage for your data, you can specify a path to store the files. Is there a particular reason you are using in-memory storage?

Looking at the error you are getting now, it sounds like the index file was not correctly written somehow. Did you get an error while executing indexes.create?

@appy-one commented on GitHub (Feb 28, 2022): Yes, indexes are created on disk regardless of your custom storage implementation. If you'd switch to AceBase's default binary storage for your data, you can specify a path to store the files. Is there a particular reason you are using in-memory storage? Looking at the error you are getting now, it sounds like the index file was not correctly written somehow. Did you get an error while executing `indexes.create`?

gitea-mirror commented

2026-05-23 08:28:21 -06:00

Author

Owner

@appy-one commented on GitHub (Mar 19, 2022):

Can you give me an update on this?

@appy-one commented on GitHub (Mar 19, 2022): Can you give me an update on this?

gitea-mirror commented

2026-05-23 08:28:22 -06:00

Author

Owner

@gryphonmyers commented on GitHub (Mar 30, 2022):

Sorry for the delay. I got routed to another project. Will look into this soon.

@gryphonmyers commented on GitHub (Mar 30, 2022): Sorry for the delay. I got routed to another project. Will look into this soon.

gitea-mirror commented

2026-05-23 08:28:22 -06:00

Author

Owner

@gryphonmyers commented on GitHub (Apr 12, 2022):

Looking at the error you are getting now, it sounds like the index file was not correctly written somehow. Did you get an error while executing indexes.create?

No, no error is thrown there. If something is going wrong, it's not letting me know about it. I think I've accurately described the issue, and it is still happening with v1.16. This error is not thrown the first time I run the test, only on subsequent executions after the file has already been written.

Yes, indexes are created on disk regardless of your custom storage implementation. If you'd switch to AceBase's default binary storage for your data, you can specify a path to store the files. Is there a particular reason you are using in-memory storage?

I'm finding it odd that the custom storage functionality cannot customize the storage implementation for indices. What about scenarios where file persistence is not viable? We have no way to reliably persist our indices along with the rest of the data?

@gryphonmyers commented on GitHub (Apr 12, 2022): > Looking at the error you are getting now, it sounds like the index file was not correctly written somehow. Did you get an error while executing `indexes.create`? No, no error is thrown there. If something is going wrong, it's not letting me know about it. I think I've accurately described the issue, and it is still happening with v1.16. This error is not thrown the first time I run the test, only on subsequent executions after the file has already been written. > Yes, indexes are created on disk regardless of your custom storage implementation. If you'd switch to AceBase's default binary storage for your data, you can specify a path to store the files. Is there a particular reason you are using in-memory storage? I'm finding it odd that the custom storage functionality cannot customize the storage implementation for indices. What about scenarios where file persistence is not viable? We have no way to reliably persist our indices along with the rest of the data?

gitea-mirror commented

2026-05-23 08:28:23 -06:00

Author

Owner

@appy-one commented on GitHub (May 9, 2022):

This error is not thrown the first time I run the test, only on subsequent executions after the file has already been written.

My apologies for the late reply. If you can provide code that reproduces this, I'll be able to check what might be going wrong.

I'm finding it odd that the custom storage functionality cannot customize the storage implementation for indices.

Indexing is a feature that has nothing to do with the ability to create custom storage adapters for your data. An index is highly specialized to quickly deliver query results and not something you can just stick into any datastore for it to work. I'd be happy to build customization features that will allow you to handle index operations (create/remove/insert/update/delete/query etc) yourself, but this is not on my todo list. If you want it to be, consider sponsoring me!

What about scenarios where file persistence is not viable? We have no way to reliably persist our indices along with the rest of the data?

I added other storage options to allow your data to be stored elsewhere, indexing was added later and works regardless of the used storage engine. I understand you'd prefer to store indexes along with the data, but it does not affect the reliability in any way. Your data and indexes are always only altered by AceBase, so they won't go out of sync.

@appy-one commented on GitHub (May 9, 2022): > This error is not thrown the first time I run the test, only on subsequent executions after the file has already been written. My apologies for the late reply. If you can provide code that reproduces this, I'll be able to check what might be going wrong. > I'm finding it odd that the custom storage functionality cannot customize the storage implementation for indices. Indexing is a feature that has nothing to do with the ability to create custom storage adapters for your data. An index is highly specialized to quickly deliver query results and not something you can just stick into any datastore for it to work. I'd be happy to build customization features that will allow you to handle index operations (create/remove/insert/update/delete/query etc) yourself, but this is not on my todo list. If you want it to be, consider sponsoring me! [![Sponsor AceBase](https://user-images.githubusercontent.com/26569719/168233053-8e56b243-4140-40ab-9a30-4cb3cc149bfe.svg)](https://github.com/sponsors/appy-one) > What about scenarios where file persistence is not viable? We have no way to reliably persist our indices along with the rest of the data? I added other storage options to allow your data to be stored elsewhere, indexing was added later and works regardless of the used storage engine. I understand you'd prefer to store indexes along with the data, but it does not affect the reliability in any way. Your data and indexes are always only altered by AceBase, so they won't go out of sync.

gitea-mirror commented

2026-05-23 08:28:23 -06:00

Author

Owner

@appy-one commented on GitHub (May 19, 2022):

Closing this for now, feel free to reopen or comment again later

@appy-one commented on GitHub (May 19, 2022): Closing this for now, feel free to reopen or comment again later [![Spread the word](https://user-images.githubusercontent.com/26569719/169265089-3d593555-e1ad-4390-986b-877ac2c38a47.svg)](https://twitter.com/intent/tweet?button=&url=https://github.com/appy-one/acebase&text=I'm+using+@AcebaseRealtime+in+my+project+to+make+my+life+easier!&button=) [![Contribute](https://user-images.githubusercontent.com/26569719/169265318-30c4c6a5-7c89-46a0-a7a2-ef433a8192f4.svg)](https://github.com/appy-one/acebase#contributing) [![Sponsor AceBase](https://user-images.githubusercontent.com/26569719/168233053-8e56b243-4140-40ab-9a30-4cb3cc149bfe.svg)](https://github.com/sponsors/appy-one)

Rows
Columns

[GH-ISSUE #67] Indices with custom storage not working #50