I have this project based on Amber. It spawns some background processes that basically just syncs data from an external service.
Works fine in general, but recently I’ve discovered that it fails if the process is long running (a full sync rather than update). But only in production.
Or rather, no so much“fail” as simply exiting. No errors, no shutdown message, nothing caught in Sentry by raven. It just exits and docker restarts the container.
Doesn’t happen when I run the same docker image locally, with the same parameters.
I’m pretty stumped. The complete lack of error messages makes it seem that some code somewhere is doing a quiet exit, but the fact that it only happens on the production server suggests that some error ought to have happened.
How is the app running in production? Is it possible it’s getting killed because it’s hitting some resource limit or something that local doesn’t have?
A simple docker-compose up. Yeah, I was considering that, but I would expect it to be killed with some sort of clue. In K8n the smoking gun is the 137 exit code (means that it was killed by the OOM killer), and I believe the same goes for base docker. But docker claims it just exited with a zero exit code.
The production server is running CoreOS, so I’m a bit off my turf, but it shouldn’t be set up particularly restricted.
What you can do, is run your project without docker, compile it with debug symbols, push it to your server (–cross-compile may be necessary) and run it using gdb, and when it crashes, run backtrace on gdb to see where the program died. That should give you some clue
That is what I had did to find which thing made my program hang.
