mirror of
https://github.com/appy-one/acebase.git
synced 2026-05-25 06:02:14 -06:00
[GH-ISSUE #35] time out / internal server error when trying to get data #30
Labels
No labels
IndexedDB
browser
bug
dependencies
documentation
duplicate
enhancement
feature request
indexes
indexes
invalid
pull-request
query
question
transaction logging
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/acebase#30
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @janphilippfranken on GitHub (Jun 29, 2021).
Original GitHub issue: https://github.com/appy-one/acebase/issues/35
dear all,
i have the following function for requesting data from my acebase database:
const fs = require("fs");
const { AceBaseClient } = require("acebase-client");
const db = new AceBaseClient({
dbname: process.env.ACEBASE_DBNAME,
host: 'localhost',
port: process.env.ACEBASE_PORT,
https: false
});
db.root.get(
x => fs.writeFile("test.json", JSON.stringify(x.val(),null,4), y => {})
);
Thus far, this worked without problems when running it on my server to get my data. However, since today (and without making changes to the code), I get Websocket timeouts and an internal server error that I do not understand. Might it be related to an increased size of the data file i am trying to fetch and if so, is there a way to increase the timeout for requests?
many thanks,
philipp
@appy-one commented on GitHub (Jul 6, 2021):
Hi sorry for the late response, I didn't get a notification email. This is not a timeout issue, but the server failing to respond to requests: "Internal Server Error". What is the error in the response?
@janphilippfranken commented on GitHub (Jul 6, 2021):
Dear Ewout,
Thanks for getting back! We just sorted the issue-it was indeed a timeout
problem and some other tweaks.
Many thanks,
Philipp
On Tue, 6 Jul 2021 at 12:36, Ewout Stortenbeker @.***>
wrote:
@appy-one commented on GitHub (Jul 7, 2021):
Ok, so I assume this was not a problem in AceBase itself. If so, please let me know what was causing it.
@janphilippfranken commented on GitHub (Jul 8, 2021):
hey again.
the thing is that i am not sure if it is an acebase issue - we used a hack
to solve our specific problem but maybe other people encounter it as well.
and now there is a new problem.
let me try to describe it:
i have data in our acebase which is a bunch of nodes.
i am reading the data from our data base using the following .js file and
try to write them to a json object:
const fs = require("fs");const { AceBase } = require("acebase");const
options = { logLevel: 'error' };const db = new
AceBase("NAME_OF_DATA_BASE", options);db.ready(() => { db.root.get(
data => fs.writeFile( "TEST.json",
JSON.stringify(data.val(),null,4), y => {process.exit(0)} ) );});
Now the problem is in the node-modules/acebase/src/node-lock.js file -
here, we get time-outs for some nodes resulting in an error / termination
of the process. I already increased the time out from the initial 15
minutes (see below) to 60:
[image: image.png]
What I don't understand is how the timeout can happen at all though. All
nodes should be the same as they are written to the database using the same
function and the data type is the same. Additionally, most nodes are
accessed and read within seconds, but then for some, it takes 30min or so
(although they should be exactly the same in terms of their content / size
/ data-types).
Maybe our approach above is incorrect?
Any help would be much appreciated, as the data is from a large participant
sample that completed a study and I need to process them swiftly to pay
performance boni etc.
Many thanks,
Philipp
On Wed, Jul 7, 2021 at 8:07 AM Ewout Stortenbeker @.***>
wrote:
@janphilippfranken commented on GitHub (Jul 8, 2021):
i am currently running the above js script using nohup with the extended
time out to see if that solves my problem.
however, i think its an inconvenient solution and would prefer to know why
it takes long to access the data - i am also very new to using acebase so
you might just spot an error in the
js function we use to get the data from the data base?
On Thu, Jul 8, 2021 at 10:48 AM Jan-Philipp Fränken <
@.***> wrote:
@janphilippfranken commented on GitHub (Jul 8, 2021):
this is an example of the nohup.out logs i get and if i wasn't running it
in the background / with higher time outs, it might already have stopped:
[image: image.png]
sorry for the spam, just tried to give all details i am aware of and hope
you might know why it is taking so long.
On Thu, Jul 8, 2021 at 10:54 AM Jan-Philipp Fränken <
@.***> wrote:
@appy-one commented on GitHub (Jul 8, 2021):
The images are missing, can you edit your posts to include them?
@janphilippfranken commented on GitHub (Jul 8, 2021):
[image: image.png]
[image: image.png]
can you see them now? they are attached above.
thanks, philipp
On Thu, Jul 8, 2021 at 1:05 PM Ewout Stortenbeker @.***>
wrote:
@appy-one commented on GitHub (Jul 8, 2021):
No sorry, still no images.
I've been thinking about this, this might be caused by an issue in older versions of the
acebasepackage. (prior to 1.6.4, see this issue). The last publishedacebase-serveruses an olderacebasedependency (v1.4.1) which might also have that locking issue. I'll publish a newacebase-serverpackage that has some fixes and the updatedacebasedependency. I'll keep you updated.@janphilippfranken commented on GitHub (Jul 8, 2021):
thanks a lot.
ps i temporarily solved the issue for me by changing the state to LOCKED in
the time out part of the node-lock.js file. that way, everytime when there
is a time out issue, it just behaves as if there was none. it allowed me to
get the data within a few seconds.
but i dont know if its reliable and also think kts not a nice solution. i
believe what you said might solve or easily!
On Thu, 8 Jul 2021 at 15:59, Ewout Stortenbeker @.***>
wrote:
@appy-one commented on GitHub (Jul 8, 2021):
Yeah, locking prevents concurrent read/write actions on the same data so it's definitely not a good idea to tamper with it - that might damage your data beyond repair.
I've published
acebase-serverv1.2.3 that now usesacebasev1.6.4 - which is known to fix a prior locking issue.Let me know if updating fixes your issue 🙂
@janphilippfranken commented on GitHub (Jul 8, 2021):
thanks - i have used both new packages now. it went much faster and almost
completed everything. however, i still got a similar locking error in the
end and could not complete the download. this is the console output. when
trying to access node *T3Ij0yBny *with child *kqtjez9c01iz599qephcok6i, *i
get a completion time warning. thats okay, but then for the next child it
gives me this error lock ... has expired, not allowed to continue. let me
know if this makes sense? :) and thanks for responding so fast.
T3Ij0yBny/kqtjez9c01iz599qephcok6i/planetSelections/TBPLCRdESJ2dysaiAACT[3]/socialTrainingStructure")
is taking a long time to complete [1]
[s1938897_three_players] NodeReader.getValue:child error: Error: lock on
tid kqv3142n0000i79qdeffyvl3 has expired, not allowed to continue
(/home/s1938897/three_players_game_server_skeleton/node_modules/acebase
/src/node-lock.js:141:19)
(/home/s1938897/three_players_game_server_skeleton/node_modules/acebase
/src/ipc/ipc.js:349:49)
(/home/s1938897/three_players_game_server_skeleton/node_modules/acebase
/src/storage.js:80:33)
(/home/s1938897/three_players_game_server_skeleton/node_modules/acebase
/src/storage-acebase.js:1601:67)
acebase/src/storage-acebase.js:1663:55
(/home/s1938897/three_players_game_server_skeleton/node_modules/acebase
/src/storage-acebase.js:1904:38)
(/home/s1938897/three_players_game_server_skeleton/node_modules/acebase
/src/storage-acebase.js:1648:21)
(/home/s1938897/three_players_game_server_skeleton/node_modules/acebase
/src/storage-acebase.js:1635:37)
(/home/s1938897/three_players_game_server_skeleton/node_modules/acebase
/src/storage-acebase.js:1679:21)
On Thu, Jul 8, 2021 at 4:36 PM Ewout Stortenbeker @.***>
wrote:
@appy-one commented on GitHub (Jul 8, 2021):
That all looks very familiar. Locking is not easy, I've spent many many hours on it and am about to do a complete overhaul (code is 95% ready and allows for more concurrent reads/writes) - but before I do I have to make 100% sure the current locking mechanism is working flawlessly.
I think at this stage the best way for me to find out what might be causing your locking problem is being able to run your code on your (anonymised) data - would you consider mailing me that (eg via WeTransfer)? Otherwise, if you can provide minimal code and data to reproduce would be also suffice. Let me know.
@janphilippfranken commented on GitHub (Jul 8, 2021):
okay that would be super helpful.
i am just chatting to my supervisor if he is okay with this. problem is
that we stored ip addresses and prolific ids so not sure if we can share
the data online just chatting with at this moment and will respond asap.
On Thu, Jul 8, 2021 at 5:20 PM Ewout Stortenbeker @.***>
wrote:
@janphilippfranken commented on GitHub (Jul 8, 2021):
can we maybe chat 1min about what data you need for reproducing the
problem?
On Thu, Jul 8, 2021 at 5:28 PM Jan-Philipp Fränken <
@.***> wrote:
@janphilippfranken commented on GitHub (Jul 8, 2021):
To join the meeting on Google Meet, click this link:
meet.google.com/ayn-pkqa-dbc
Or open Meet and enter this code: ayn-pkqa-dbc
On Thu, Jul 8, 2021 at 5:37 PM Jan-Philipp Fränken <
@.***> wrote:
@appy-one commented on GitHub (Jul 8, 2021):
I'm in the supermarket right now. 😂 Only data I need is something that will cause the locking issue to happen
@janphilippfranken commented on GitHub (Jul 8, 2021):
haha okay no problem :D
the problem is that the .db file is around 100MB and i can't split it or
anything 😂
it does only have anonymous data but its confidential experimental data so
please delete it afterwards 😋
this is the content of the *get-data.js *file i run on our server using node
get-data.js. after accessing around 90% of our nodes, the locking problem
occurs.
const fs = require("fs");
const { AceBase } = require("acebase");
const options = {logLevel: 'error' };
const db = new AceBase("NAME_OF_DB", options);
db.ready(() => {
db.root.get(
);
});
this file is in the same directory as the NAME_OF_DB.acebase directory
which includes the data.db file and rule.js file that i just sent
to @.*
and @.*** *as i was not sure which email you are using.
i guess that should be everything you need?
enjoy shopping!
On Thu, Jul 8, 2021 at 5:41 PM Ewout Stortenbeker @.***>
wrote:
@janphilippfranken commented on GitHub (Jul 8, 2021):
actually that email does not work on wtransfer
which email should i use to send it to you?
On Thu, Jul 8, 2021 at 5:56 PM Jan-Philipp Fränken <
@.***> wrote:
@appy-one commented on GitHub (Jul 8, 2021):
No problem, I'll delete all once I've got enough info to fix the issue.
You can send it to: me [at] appy.one
@janphilippfranken commented on GitHub (Jul 8, 2021):
just sent the link in the mail
above - let me know if it works!
On Thu, 8 Jul 2021 at 18:11, Ewout Stortenbeker @.***>
wrote:
@appy-one commented on GitHub (Jul 8, 2021):
Got it!
@appy-one commented on GitHub (Jul 8, 2021):
I'm getting the errors right now, thanks for sending!
I'll debug tonight or tomorrow, kids have to go to bed now 😉
@janphilippfranken commented on GitHub (Jul 8, 2021):
brilliant, thank you so much!
On Thu, 8 Jul 2021 at 18:22, Ewout Stortenbeker @.***>
wrote:
@appy-one commented on GitHub (Jul 9, 2021):
I've been doing mutiple tests, it doesn't seem to be a locking issue but rather a performance issue that causes the locks to expire while all nested data is read. Initially I thought using
exportinstead ofgetto stream the output to your json file instead would be a faster, less memory requiring solution, but I ran into other issues when testing.What you can do for now to get the data to export successfully, is to change the
LOCK_TIMEOUTconst in node-lock.js to:const LOCK_TIMEOUT = 90 * MINUTE;. It takes a while, but the export will finish before any locks expire. Note that any writes to the data are not possible while the locks are in place, so it's best to do it on a downloaded copy of the db.This is a very good case for me to improve performance with reading larger datasets, but that will take some time. Also, it brought the
exportstreaming issue to my attention, so that's another thing that has to be fixed. I'll keep you updated on the progress!@janphilippfranken commented on GitHub (Jul 9, 2021):
great, thank you for sharing this. excited to
hear how it goes.
will continue with a downloded version and keep you posted if there are any
issues.
On Fri, 9 Jul 2021 at 08:21, Ewout Stortenbeker @.***>
wrote:
@janphilippfranken commented on GitHub (Jul 9, 2021):
just to double check for running it offline - can i just run it from the
same folder?
like one directory with the acebase folder with data.db and rules.js inside
and then in the get-data.js inside the same main folder? (and also
node-modules)?
i tried running it online but it seems as if it does not find the data base
that way but just returns an empty .json object
On Fri, Jul 9, 2021 at 9:17 AM Jan-Philipp Fränken <
@.***> wrote:
@appy-one commented on GitHub (Jul 9, 2021):
You will have to set the storage path to
'..'like so:new AceBace('mydb', { logLevel: 'error', storage: { path: '..' } }). It will then open the database from../mydb.acebase/data.db. Otherwise, it'll create a new (empty) db directory in the current path. Note that the current path depends on your working directory. Executingnode get-data.jsin the database folder will have your current result, but executingnode mydb.acebase/get-data.jsfrom the project dir should work without having to change thepathsetting.@appy-one commented on GitHub (Jul 9, 2021):
I managed to fix the
exportissue I mentioned earlier. I will publish the changed code in the next version of acebase, but if you want to be able to export your entire db to json very quickly right now (takes about 1 minute on my machine), follow these steps:Use this code instead of your
db.root.get(...)in get-data.js:The current implementation of
exportproduces invalid json output on your data, so you will have to replace theexportNodefunction in storage.js with the following:I'll commit this and other changes later today.
@janphilippfranken commented on GitHub (Jul 9, 2021):
great thanks.
it gives me an error for the await component:
"await is only valid in async functions and the top level bodies of modules"
this is my code now in the show-data.js which is in the same directory as
the s1938897_three_players acebase folder which has rules.js and data.db. i
guess something is wrong? and thanks for your help, really appreciate it.
this is also my first acebase project thats why i probably lack some
understanding of the basics. if you could help me sorting this last one
that would be great.
const fs = require("fs");
const { AceBase } = require("acebase");
const options = { logLevel: 'error' };
const db = new AceBase('s1938897_three_players', { logLevel: 'error',
storage: { path: './' } })
db.ready(() => {
const fstream = fs.createWriteStream('export.json', { flags: 'w+' });
const stream = {
write: chunk => {
const ok = fstream.write(chunk);
if (!ok) {
return new Promise(resolve => fstream.once('drain', resolve));
}
}
};
await db.root.export(stream);
fstream.close();
});
On Fri, Jul 9, 2021 at 12:00 PM Ewout Stortenbeker @.***>
wrote:
@appy-one commented on GitHub (Jul 9, 2021):
Change
db.ready(() =>todb.ready(async () =>@janphilippfranken commented on GitHub (Jul 9, 2021):
it worked!
On Fri, Jul 9, 2021 at 12:18 PM Ewout Stortenbeker @.***>
wrote:
@appy-one commented on GitHub (Jul 13, 2021):
FYI I've published these changes in
acebasev1.7.0, which is now also used byacebase-serverv1.2.4@janphilippfranken commented on GitHub (Jul 13, 2021):
perfect, thank you for keeping me posted.
have a great day :-)
On Tue, Jul 13, 2021 at 1:12 PM Ewout Stortenbeker @.***>
wrote: