[GH-ISSUE #183] Error: Attempt to read non-existing records #87

Closed
opened 2026-05-23 08:30:02 -06:00 by gitea-mirror · 16 comments
Owner

Originally created by @futurGH on GitHub (Nov 28, 2022).
Original GitHub issue: https://github.com/appy-one/acebase/issues/183

Encountering a strange issue — there's a specific set of what I believe to be 5 consecutive records in my database that I'm calling "ghost records". Trying to read, write, or delete them is throwing the error Attempt to read non-existing records. Even setting their parent record to null causes an unending read lock on the record before eventually causing the same error.

I apologize for the vague issue without a repro, at the moment I'd just appreciate any advice on how to get rid of them! Trying to export the database, manually remove them, then import it back isn't working either; the export process seems to freeze upon reaching those records.

Attempt to read non-existing records
at NodeReader._treeDataReader (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/storage/binary/index.js:2933:19)
at BinaryReader.more (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/btree/binary-reader.js:110:38)
at BinaryReader.assert (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/btree/binary-reader.js:139:24)
at BinaryReader.get (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/btree/binary-reader.js:78:20)
at BinaryBPlusTree._readChild (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/btree/binary-tree.js:205:36)
at async leaf.getNext (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/btree/binary-tree.js:675:35)
at async createStreamFromBinaryTree (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/storage/binary/index.js:2629:56)
at async NodeReader.getAllocation (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/storage/binary/index.js:2179:9)
at async /project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/storage/binary/index.js:2187:45
at async Promise.all (index 0)
Originally created by @futurGH on GitHub (Nov 28, 2022). Original GitHub issue: https://github.com/appy-one/acebase/issues/183 Encountering a strange issue — there's a specific set of what I believe to be 5 consecutive records in my database that I'm calling "ghost records". Trying to read, write, or delete them is throwing the error `Attempt to read non-existing records`. Even setting their parent record to `null` causes an unending read lock on the record before eventually causing the same error. I apologize for the vague issue without a repro, at the moment I'd just appreciate any advice on how to get rid of them! Trying to export the database, manually remove them, then import it back isn't working either; the export process seems to freeze upon reaching those records. ``` Attempt to read non-existing records at NodeReader._treeDataReader (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/storage/binary/index.js:2933:19) at BinaryReader.more (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/btree/binary-reader.js:110:38) at BinaryReader.assert (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/btree/binary-reader.js:139:24) at BinaryReader.get (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/btree/binary-reader.js:78:20) at BinaryBPlusTree._readChild (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/btree/binary-tree.js:205:36) at async leaf.getNext (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/btree/binary-tree.js:675:35) at async createStreamFromBinaryTree (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/storage/binary/index.js:2629:56) at async NodeReader.getAllocation (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/storage/binary/index.js:2179:9) at async /project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/storage/binary/index.js:2187:45 at async Promise.all (index 0) ```
Author
Owner

@futurGH commented on GitHub (Nov 28, 2022):

Right after submitting this issue, I did manage to erase the records (and their sibling records as acceptable collateral damage) with

db.query("path/to/parent").forEach(snapshot => snapshot.ref.remove())

Leaving the issue up in case it ever becomes possible to consistently reproduce or if you have any idea what may be causing it.

<!-- gh-comment-id:1329375013 --> @futurGH commented on GitHub (Nov 28, 2022): Right after submitting this issue, I did manage to erase the records (and their sibling records as acceptable collateral damage) with ```js db.query("path/to/parent").forEach(snapshot => snapshot.ref.remove()) ``` Leaving the issue up in case it ever becomes possible to consistently reproduce or if you have any idea what may be causing it.
Author
Owner

@appy-one commented on GitHub (Nov 30, 2022):

@futurGH thanks for reporting, I'll add more info to the error thrown to make an investigation easier. If you still have a backup with the "ghost records" in it, would be great if you can test with that code!

<!-- gh-comment-id:1332663571 --> @appy-one commented on GitHub (Nov 30, 2022): @futurGH thanks for reporting, I'll add more info to the error thrown to make an investigation easier. If you still have a backup with the "ghost records" in it, would be great if you can test with that code!
Author
Owner

@futurGH commented on GitHub (Nov 30, 2022):

Thanks for looking into this! I'm now getting:

Error: Attempt to read non-existing records of path "/maps/15000367/matches": 211 to 220 for index 27001 + 1024 bytes. Node has 211 allocated records in the following ranges: [object Object]

Couldn't get an object out of the [object Object], I'm pretty sure that's coming from the initial concatenation where the error is being thrown.

Let me know if you'd like me to send the db file.

<!-- gh-comment-id:1332746987 --> @futurGH commented on GitHub (Nov 30, 2022): Thanks for looking into this! I'm now getting: ``` Error: Attempt to read non-existing records of path "/maps/15000367/matches": 211 to 220 for index 27001 + 1024 bytes. Node has 211 allocated records in the following ranges: [object Object] ``` Couldn't get an object out of the `[object Object]`, I'm pretty sure that's coming from the initial concatenation where the error is being thrown. Let me know if you'd like me to send the db file.
Author
Owner

@appy-one commented on GitHub (Nov 30, 2022):

Thanks! Sending the db would be really appreciated! me@appy.one

<!-- gh-comment-id:1332765738 --> @appy-one commented on GitHub (Nov 30, 2022): Thanks! Sending the db would be really appreciated! me@appy.one
Author
Owner

@futurGH commented on GitHub (Nov 30, 2022):

Email sent!

<!-- gh-comment-id:1332770610 --> @futurGH commented on GitHub (Nov 30, 2022): Email sent!
Author
Owner

@appy-one commented on GitHub (Dec 1, 2022):

@futurGH I managed to reproduce the issue with your db, digging into it now

<!-- gh-comment-id:1334260111 --> @appy-one commented on GitHub (Dec 1, 2022): @futurGH I managed to reproduce the issue with your db, digging into it now
Author
Owner

@appy-one commented on GitHub (Dec 1, 2022):

Ok, the issue seems to be that the B+Tree that stores all child properties has not been assigned sufficient amount of records to store its data, causing the leaf stored at the very end to have been cut off. I'm now investigating how that might have happened.

<!-- gh-comment-id:1334427053 --> @appy-one commented on GitHub (Dec 1, 2022): Ok, the issue seems to be that the B+Tree that stores all child properties has not been assigned sufficient amount of records to store its data, causing the leaf stored at the very end to have been cut off. I'm now investigating how that might have happened.
Author
Owner

@appy-one commented on GitHub (Dec 1, 2022):

@futurGH Is it possible your app ever crashed (eg because of an unhandled promise rejection under Node 15+), had a power failure while it was writing to your database, or because you manually terminated the process?

I'm onto a possible cause, where data for the B+Tree has to be moved elsewhere to allow growth. There would be an in-between state where the data hasn't moved yet, but the allocated size has already been assigned. If the app would have crashed in that specific moment, writing new data upon restart might have caused data to be written in unallocated space.

Let me know if that sounds like a possible cause

<!-- gh-comment-id:1334540979 --> @appy-one commented on GitHub (Dec 1, 2022): @futurGH Is it possible your app ever crashed (eg because of an unhandled promise rejection under Node 15+), had a power failure while it was writing to your database, or because you manually terminated the process? I'm onto a possible cause, where data for the B+Tree has to be moved elsewhere to allow growth. There would be an in-between state where the data hasn't moved yet, but the allocated size has already been assigned. If the app would have crashed in that specific moment, writing new data upon restart might have caused data to be written in unallocated space. Let me know if that sounds like a possible cause
Author
Owner

@futurGH commented on GitHub (Dec 1, 2022):

A crash is entirely possible. The application is run with pm2 so I wouldn't have noticed it.

There was also a short period of time when writes were failing because the server was out of available storage space; now that I think about it, I only noticed these ghost records after that incident.

<!-- gh-comment-id:1334542396 --> @futurGH commented on GitHub (Dec 1, 2022): A crash is entirely possible. The application is run with pm2 so I wouldn't have noticed it. There was also a short period of time when writes were failing because the server was out of available storage space; now that I think about it, I only noticed these ghost records after that incident.
Author
Owner

@appy-one commented on GitHub (Dec 1, 2022):

Thanks for your feedback, a full drive could certainly also have caused this!

I'll focus on two things:

  • Fix the issue by making it impossible to have (persistent) in-between state when moving data.
  • Add repair option to db.recovery for this specific issue to recover as much data possible.

I'll probably be working on this on Monday, I'll keep you posted

<!-- gh-comment-id:1334562557 --> @appy-one commented on GitHub (Dec 1, 2022): Thanks for your feedback, a full drive could certainly also have caused this! I'll focus on two things: * Fix the issue by making it impossible to have (persistent) in-between state when moving data. * Add repair option to `db.recovery` for this specific issue to recover as much data possible. I'll probably be working on this on Monday, I'll keep you posted
Author
Owner

@futurGH commented on GitHub (Dec 2, 2022):

Much appreciated!

<!-- gh-comment-id:1335395460 --> @futurGH commented on GitHub (Dec 2, 2022): Much appreciated!
Author
Owner

@appy-one commented on GitHub (Dec 5, 2022):

@futurGH I've added a new recovery option to fix your db: await db.recovery.repairNodeTree('path/to/broken/collection')

Let me know if it works!

<!-- gh-comment-id:1337632457 --> @appy-one commented on GitHub (Dec 5, 2022): @futurGH I've added a new recovery option to fix your db: `await db.recovery.repairNodeTree('path/to/broken/collection')` Let me know if it works!
Author
Owner

@futurGH commented on GitHub (Dec 7, 2022):

Sorry for the delay, and thank you so much for putting this much work into my (probably self-induced) issue! I'm now getting:

Starting node tree repair for path "/maps/15000368"
Failed to repair node tree for path "/maps/15000368": Error: Node at maps/15000368 does not have a B+Tree key index
<!-- gh-comment-id:1341663451 --> @futurGH commented on GitHub (Dec 7, 2022): Sorry for the delay, and thank you so much for putting this much work into my (probably self-induced) issue! I'm now getting: ``` Starting node tree repair for path "/maps/15000368" Failed to repair node tree for path "/maps/15000368": Error: Node at maps/15000368 does not have a B+Tree key index ```
Author
Owner

@appy-one commented on GitHub (Dec 8, 2022):

@futurGH do it on the nodes you're unable to read specific children from. I've tested with the path you sent me through e-mail: "maps/15000367/matches"

<!-- gh-comment-id:1342216559 --> @appy-one commented on GitHub (Dec 8, 2022): @futurGH do it on the nodes you're unable to read specific children from. I've tested with the path you sent me through e-mail: `"maps/15000367/matches"`
Author
Owner

@futurGH commented on GitHub (Dec 8, 2022):

Whoops, my bad! It caught the broken records and successfully repaired them. No longer getting any errors trying to read — I can't know whether the data has been accurately restored but the structure is certainly correct. Thanks for the help!

<!-- gh-comment-id:1342754596 --> @futurGH commented on GitHub (Dec 8, 2022): Whoops, my bad! It caught the broken records and successfully repaired them. No longer getting any errors trying to read — I can't know whether the data has been accurately restored but the structure is certainly correct. Thanks for the help!
Author
Owner

@appy-one commented on GitHub (Dec 12, 2022):

Awesome, closing the issue!

Spread the word contribute Sponsor AceBase

<!-- gh-comment-id:1346165828 --> @appy-one commented on GitHub (Dec 12, 2022): Awesome, closing the issue! [![Spread the word](https://user-images.githubusercontent.com/26569719/169265089-3d593555-e1ad-4390-986b-877ac2c38a47.svg)](https://twitter.com/intent/tweet?button=&url=https://github.com/appy-one/acebase&text=I'm+using+@AcebaseRealtime+in+my+project+to+make+my+life+easier!&button=) [![contribute](https://user-images.githubusercontent.com/26569719/169265318-30c4c6a5-7c89-46a0-a7a2-ef433a8192f4.svg)](https://github.com/appy-one/acebase#contributing) [![Sponsor AceBase](https://user-images.githubusercontent.com/26569719/168233053-8e56b243-4140-40ab-9a30-4cb3cc149bfe.svg)](https://github.com/sponsors/appy-one)
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/acebase#87
No description provided.