WARNING: THIS SITE IS A MIRROR OF GITHUB.COM / IT CANNOT LOGIN OR REGISTER ACCOUNTS / THE CONTENTS ARE PROVIDED AS-IS / THIS SITE ASSUMES NO RESPONSIBILITY FOR ANY DISPLAYED CONTENT OR LINKS / IF YOU FOUND SOMETHING MAY NOT GOOD FOR EVERYONE, CONTACT ADMIN AT ilovescratch@foxmail.com
Skip to content

database: improve handling of connection failure #203

@mdonadoni

Description

@mdonadoni

It can happen that connecting to external database takes a long time or never completes:

$ psql -h db.example.org -p 1234  -U reana reana
[... wait forever ...]

When this happens, all the REANA components that depend on the database get stuck. As an example, the REST API of r-server will stop replying, and the logs will show:

*** uWSGI listen queue of socket "127.0.0.1:45841" (fd: 3) full !!! (100/100) ***
*** uWSGI listen queue of socket "127.0.0.1:45841" (fd: 3) full !!! (101/100) ***
*** uWSGI listen queue of socket "127.0.0.1:45841" (fd: 3) full !!! (101/100) ***
[...]

Note that these error messages might appear after many minutes, when the connections queue is full. In the meantime, nothing is printed to the logs, which makes it difficult to understand what is going wrong. As an example, one might think that the server is not reachable, maybe due to Ingresses that are not configured correctly.

Also note that in this case the components do not self-heal and a restart of the pod is needed.

We should improve this, maybe by tweaking the configuration to put a timeout on database connections, so that we can at least return a 500 error in a timely manner and if possible self-heal when the database is back online.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions