Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement unix domain socket support #7615

Merged
merged 3 commits into from
Jun 6, 2023

Conversation

ThomasWaldmann
Copy link
Member

@ThomasWaldmann ThomasWaldmann commented May 29, 2023

Server side (listening side):

borg serve --socket=/path/to/socket
or
borg serve --socket  # finds socket at `get_runtime_dir()/borg.sock`

Client:

borg --socket=/path/to/socket --repo=socket:///path/to/repo rcreate ...
or
borg [--socket] --repo socket:///path/to/repo create ...  # finds socket at `get_runtime_dir()/borg.sock`

BORG_RUNTIME_DIR env var can be used to override BORG_BASE_DIR and platformdirs.user_runtime_dir, as usual.

fixes #6183.

@codecov-commenter
Copy link

codecov-commenter commented May 29, 2023

Codecov Report

Merging #7615 (ae97584) into master (9f51600) will increase coverage by 0.95%.
The diff coverage is 76.62%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##           master    #7615      +/-   ##
==========================================
+ Coverage   82.89%   83.85%   +0.95%     
==========================================
  Files          66       66              
  Lines       11748    11802      +54     
  Branches     2120     2140      +20     
==========================================
+ Hits         9739     9896     +157     
+ Misses       1433     1342      -91     
+ Partials      576      564      -12     
Impacted Files Coverage Δ
src/borg/archiver/serve_cmd.py 100.00% <ø> (ø)
src/borg/remote.py 83.44% <67.92%> (-1.23%) ⬇️
src/borg/helpers/fs.py 84.21% <85.71%> (+4.21%) ⬆️
src/borg/archiver/_common.py 91.05% <100.00%> (+0.04%) ⬆️
src/borg/helpers/__init__.py 100.00% <100.00%> (ø)
src/borg/helpers/parseformat.py 90.97% <100.00%> (+0.39%) ⬆️
src/borg/logger.py 70.90% <100.00%> (+0.35%) ⬆️

... and 15 files with indirect coverage changes

@ThomasWaldmann ThomasWaldmann force-pushed the serve-socket2 branch 3 times, most recently from 9c94bfe to a2f4d60 Compare May 30, 2023 15:52
@ThomasWaldmann
Copy link
Member Author

ThomasWaldmann commented May 30, 2023

hmm, the windows CI seems to hang. is this a borg or a github issue?

Update:

Logs after killing the workflow run:

...
src/borg/testsuite/archiver/serve_cmd.py::test_with_socket_default Terminate batch job (Y/N)? 

Same thing for the other socket test:

src/borg/testsuite/archiver/serve_cmd.py::test_with_socket_option Terminate batch job (Y/N)? 

Same for the permissions test:

src/borg/testsuite/archiver/serve_cmd.py::test_socket_permissions Terminate batch job (Y/N)? 

So it seems to hang infinitely there, but only on win32.

Update: code works on windows cygwin without hanging.

@RayyanAnsari can you have a look?

@PhrozenByte
Copy link
Contributor

hmm, the windows CI seems to hang. is this a borg or a github issue?

It's not not very common for a GitHub workflow to hang, but I've seen it before. GitHub will kill it after 6 hours runtime. You can either wait or cancel it manually, then restart the workflow - if it hangs again it might be an issue with Borg. The logs of the hanging step might be lost though.

@ThomasWaldmann ThomasWaldmann force-pushed the serve-socket2 branch 2 times, most recently from ef734c3 to 4238f10 Compare May 31, 2023 21:54
@ThomasWaldmann ThomasWaldmann added this to the 2.0.0b6 milestone May 31, 2023
def serve_socket(monkeypatch):
have_a_short_runtime_dir(monkeypatch)
# use a random unique socket filename, so tests can run in parallel.
socket_file = tempfile.mktemp(suffix=".sock", prefix="borg-", dir=get_runtime_dir())

Check failure

Code scanning / CodeQL

Insecure temporary file

Call to deprecated function tempfile.mktemp may be insecure.
@ThomasWaldmann ThomasWaldmann force-pushed the serve-socket2 branch 2 times, most recently from 1175056 to 7db5a94 Compare June 6, 2023 16:55
server (listening) side:
borg serve --socket  # default location
borg serve --socket=/path/to/socket

client side:
borg -r socket:///path/to/repo create ...
borg --socket=/path/to/socket -r socket:///path/to/repo ...

served connections:
- for ssh: proto: one connection
- for socket: proto: many connections (one after the other)

The socket has user and group permissions (770).

skip socket tests on win32, they hang infinitely, until
github CI terminates them after 60 minutes.

socket tests: use unique socket name

don't use the standard / default socket name, otherwise tests
running in parallel would interfere with each other by using
the same socket / the same borg serve process.

write a .pid file, clean up .pid and .sock file at exit

add stderr print for accepted/finished socket connection
shutting down logging is problematic as it is global
and we do multi-threaded execution, e.g. in tests.

thus, rather just flush the important loggers and keep
them alive.
@ThomasWaldmann ThomasWaldmann merged commit cad57e7 into borgbackup:master Jun 6, 2023
@ThomasWaldmann ThomasWaldmann deleted the serve-socket2 branch June 6, 2023 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add UNIX domain socket transport
3 participants