node: support reloading node attributes with SIGHUP #3005

End-rey · 2024-11-08T12:17:35Z

Closes #1870.

Also fix bug from #2998, that incorrectly checked when to reconnect.

codecov · 2024-11-08T12:20:00Z

Codecov Report

Attention: Patch coverage is 17.72152% with 65 lines in your changes missing coverage. Please review.

Project coverage is 22.85%. Comparing base (f1b6982) to head (d430daa).
Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
cmd/neofs-node/netmap.go	0.00%	39 Missing ⚠️
cmd/neofs-node/config.go	0.00%	13 Missing ⚠️
cmd/neofs-node/attributes.go	70.00%	6 Missing ⚠️
cmd/neofs-node/container.go	0.00%	3 Missing ⚠️
pkg/core/object/fmt.go	0.00%	2 Missing and 1 partial ⚠️
pkg/morph/client/reload.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3005      +/-   ##
==========================================
- Coverage   22.85%   22.85%   -0.01%     
==========================================
  Files         791      791              
  Lines       58603    58684      +81     
==========================================
+ Hits        13395    13411      +16     
- Misses      44312    44376      +64     
- Partials      896      897       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests

roman-khimov

You can compare old/new and avoid updates if nothing changed.
You're likely to have concurrency problems here when updating c.
Not sure what updateLocalState gives us, we want to force node netmap update mostly (send an appropriate tx) and it works the other way around.

roman-khimov

Maybe an atomic pointer would be easier to handle here, but locks can be used too.

cmd/neofs-node/config.go

cmd/neofs-node/attributes.go

cmd/neofs-node/netmap.go

cmd/neofs-node/attributes.go

cmd/neofs-node/config.go

cmd/neofs-node/netmap.go

cmd/neofs-node/container.go

Incorrect expression to check for reconnection. Before that, it worked the other way around, now is fixed. Signed-off-by: Andrey Butusov <[email protected]>

End-rey · 2024-11-18T15:26:20Z

@cthulhu-rider Can you help me please.
I have problems with the tests after I updated the sdk:
1.

--- FAIL: TestFormatValidator_Validate (0.01s)
    --- FAIL: TestFormatValidator_Validate/incorrect_session_token (0.00s)
        --- FAIL: TestFormatValidator_Validate/incorrect_session_token/wrong_signature (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x8c2558]

goroutine 25 [running]:
testing.tRunner.func1.2({0xa05da0, 0x10a8350})
        /snap/go/current/src/testing/testing.go:1632 +0x230
testing.tRunner.func1()
        /snap/go/current/src/testing/testing.go:1635 +0x35e
panic({0xa05da0?, 0x10a8350?})
        /snap/go/current/src/runtime/panic.go:785 +0x132
github.com/nspcc-dev/neofs-sdk-go/crypto.PublicKeyBytes({0x0, 0x0})
        /home/endrey/go/pkg/mod/github.com/nspcc-dev/[email protected]/crypto/util.go:16 +0x18
github.com/nspcc-dev/neofs-sdk-go/session.commonData.AssertAuthKey({0x1, {0x81, 0x78, 0x56, 0x98, 0x1e, 0x55, 0x45, 0x7e, 0xab, ...}, ...}, ...)
        /home/endrey/go/pkg/mod/github.com/nspcc-dev/[email protected]/session/common.go:339 +0x25
github.com/nspcc-dev/neofs-node/pkg/core/object.(*FormatValidator).validateSignatureKey(0xc000218498?, 0xc000200a80)
        /home/endrey/neo/neofs-node/pkg/core/object/fmt.go:198 +0x1b8
github.com/nspcc-dev/neofs-node/pkg/core/object.(*FormatValidator).Validate(0xc000266010, 0xc000200a80, 0x0)
        /home/endrey/neo/neofs-node/pkg/core/object/fmt.go:164 +0x6e8
github.com/nspcc-dev/neofs-node/pkg/core/object.TestFormatValidator_Validate.func6.1(0xc00025cea0)
        /home/endrey/neo/neofs-node/pkg/core/object/fmt_test.go:129 +0x18c

This one maybe after this commit, don't return error that public key is missing.
Make check for nil in this func?

neofs-node/pkg/core/object/fmt.go

Line 185 in f1b6982

func (v *FormatValidator) validateSignatureKey(obj *object.Object) error {

--- FAIL: TestBlobStor_Put_Overflow (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xa05e1f]

goroutine 12 [running]:
testing.tRunner.func1.2({0xaadd60, 0x11e6110})
        /snap/go/current/src/testing/testing.go:1632 +0x230
testing.tRunner.func1()
        /snap/go/current/src/testing/testing.go:1635 +0x35e
panic({0xaadd60?, 0x11e6110?})
        /snap/go/current/src/runtime/panic.go:785 +0x132
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor.(*BlobStor).Put(0xc00011a310, {{{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...}, ...})
        /home/endrey/neo/neofs-node/pkg/local_object_storage/blobstor/put.go:34 +0x15f
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor_test.TestBlobStor_Put_Overflow(0xc0002f6340)
        /home/endrey/neo/neofs-node/pkg/local_object_storage/blobstor/put_test.go:48 +0x19f

May be after this commit.
Check for nil here?

neofs-node/pkg/local_object_storage/blobstor/put.go

Line 34 in f1b6982

prm.RawData = prm.Object.Marshal()

--- FAIL: TestShardReload (0.27s)
    logger.go:146: 2024-11-18T12:57:28.906+0300 DEBUG   opening...      {"component": "BlobStor"}
    logger.go:146: 2024-11-18T12:57:28.916+0300 DEBUG   initializing... {"component": "BlobStor"}
    logger.go:146: 2024-11-18T12:57:28.942+0300 INFO    local object storage operation  {"component": "BlobStor", "address": "3q1D7ykZnBWGnAQtt5T3RRiSpAqWAGFiLPdmXJCWRG6L/CZQuQVBCTmjtDuB6n42Pm7RKB4ebzdwTFRgoEbpddTmz", "op": "PUT", "type": "fstree", "storage_id": ""}
    logger.go:146: 2024-11-18T12:57:28.973+0300 INFO    local object storage operation  {"component": "BlobStor", "address": "7QRMDrx1kFBaq7R2pwu1b1LncrF2XbktRBZipvWRbPkq/3V9Kmqk36yuRmtYAAxTbwh4LbQsStwL2zhGs7QmSq1zc", "op": "PUT", "type": "fstree", "storage_id": ""}
    logger.go:146: 2024-11-18T12:57:29.007+0300 INFO    local object storage operation  {"component": "BlobStor", "address": "2ZL5irCwvkdksinQHbjrD4A7NvhSbr36R5qDbtLZXmXv/9f8rcbsQYCUHckcHcj3WprsQ4eJdyph15xPV75tDrvWR", "op": "PUT", "type": "fstree", "storage_id": ""}
    logger.go:146: 2024-11-18T12:57:29.037+0300 INFO    local object storage operation  {"component": "BlobStor", "address": "29k7ZTwNGAU6aUMnfGivecwRqFDhEx2KhaXGST5ycQe5/5UUzS5Hk3omMmVHCudnwn2FZ5ZeuAASg3vSFa1VNSVE", "op": "PUT", "type": "fstree", "storage_id": ""}
    logger.go:146: 2024-11-18T12:57:29.073+0300 INFO    local object storage operation  {"component": "BlobStor", "address": "6EDxrhcCuXDMDo59zW6hVwK1MaNea8CacM6rmhuBXuuX/GUUEtSt4VGun2TXwb5kP6vm6zF27g2V2Qc3vVbPfsiC4", "op": "PUT", "type": "fstree", "storage_id": ""}
    logger.go:146: 2024-11-18T12:57:29.094+0300 INFO    trying to restore read-write mode
    logger.go:146: 2024-11-18T12:57:29.094+0300 INFO    setting shard mode      {"old_mode": "READ_WRITE", "new_mode": "READ_WRITE"}
    logger.go:146: 2024-11-18T12:57:29.094+0300 INFO    shard mode set successfully     {"mode": "READ_WRITE"}
    logger.go:146: 2024-11-18T12:57:29.112+0300 INFO    trying to restore read-write mode
    logger.go:146: 2024-11-18T12:57:29.113+0300 INFO    setting shard mode      {"old_mode": "READ_WRITE", "new_mode": "READ_WRITE"}
    logger.go:146: 2024-11-18T12:57:29.113+0300 INFO    shard mode set successfully     {"mode": "READ_WRITE"}
    logger.go:146: 2024-11-18T12:57:29.132+0300 INFO    local object storage operation  {"component": "BlobStor", "address": "5Hu41RGaong7w6x8FMmdR9roYcdrbR7Wy1m3aJ2pt34J/13tNqp8evbT3ZZUJ23pgp2H8avWZYEThvNW1k1xn9sPS", "op": "PUT", "type": "fstree", "storage_id": ""}
    logger.go:146: 2024-11-18T12:57:29.168+0300 WARN    could not unmarshal object      {"address": "5Hu41RGaong7w6x8FMmdR9roYcdrbR7Wy1m3aJ2pt34J/13tNqp8evbT3ZZUJ23pgp2H8avWZYEThvNW1k1xn9sPS", "err": "invalid header: invalid session token: missing session issuer"}
    logger.go:146: 2024-11-18T12:57:29.169+0300 WARN    could not unmarshal object      {"address": "7QRMDrx1kFBaq7R2pwu1b1LncrF2XbktRBZipvWRbPkq/3V9Kmqk36yuRmtYAAxTbwh4LbQsStwL2zhGs7QmSq1zc", "err": "invalid header: invalid session token: missing session issuer"}
    logger.go:146: 2024-11-18T12:57:29.169+0300 WARN    could not unmarshal object      {"address": "29k7ZTwNGAU6aUMnfGivecwRqFDhEx2KhaXGST5ycQe5/5UUzS5Hk3omMmVHCudnwn2FZ5ZeuAASg3vSFa1VNSVE", "err": "invalid header: invalid session token: missing session issuer"}
    logger.go:146: 2024-11-18T12:57:29.169+0300 WARN    could not unmarshal object      {"address": "2ZL5irCwvkdksinQHbjrD4A7NvhSbr36R5qDbtLZXmXv/9f8rcbsQYCUHckcHcj3WprsQ4eJdyph15xPV75tDrvWR", "err": "invalid header: invalid session token: missing session issuer"}
    logger.go:146: 2024-11-18T12:57:29.169+0300 WARN    could not unmarshal object      {"address": "3q1D7ykZnBWGnAQtt5T3RRiSpAqWAGFiLPdmXJCWRG6L/CZQuQVBCTmjtDuB6n42Pm7RKB4ebzdwTFRgoEbpddTmz", "err": "invalid header: invalid session token: missing session issuer"}
    logger.go:146: 2024-11-18T12:57:29.170+0300 WARN    could not unmarshal object      {"address": "6EDxrhcCuXDMDo59zW6hVwK1MaNea8CacM6rmhuBXuuX/GUUEtSt4VGun2TXwb5kP6vm6zF27g2V2Qc3vVbPfsiC4", "err": "invalid header: invalid session token: missing session issuer"}
    logger.go:146: 2024-11-18T12:57:29.173+0300 INFO    trying to restore read-write mode
    logger.go:146: 2024-11-18T12:57:29.173+0300 INFO    setting shard mode      {"old_mode": "READ_WRITE", "new_mode": "READ_WRITE"}
    logger.go:146: 2024-11-18T12:57:29.173+0300 INFO    shard mode set successfully     {"mode": "READ_WRITE"}
    --- FAIL: TestShardReload/open_meta_at_new_path (0.08s)
        reload_test.go:69: 
                Error Trace:    /home/endrey/neo/neofs-node/pkg/local_object_storage/shard/reload_test.go:69
                                                        /home/endrey/neo/neofs-node/pkg/local_object_storage/shard/reload_test.go:100
                Error:          Not equal: 
                                expected: true
                                actual  : false
                Test:           TestShardReload/open_meta_at_new_path
                Messages:       object #0 is missing
FAIL
FAIL    github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard  29.652s

I think here there are some differences that break the test.
Don't know why session token owner ID is nil.

--- FAIL: TestHeadRequest (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xb3d9db]

goroutine 9 [running]:
testing.tRunner.func1.2({0xbfc840, 0x1492760})
        /snap/go/current/src/testing/testing.go:1632 +0x230
testing.tRunner.func1()
        /snap/go/current/src/testing/testing.go:1635 +0x35e
panic({0xbfc840?, 0x1492760?})
        /snap/go/current/src/runtime/panic.go:785 +0x132
github.com/nspcc-dev/neofs-node/pkg/services/object/acl/eacl/v2.headersFromObject(0xc000328660, {0x88, 0x3c, 0xd5, 0xb2, 0xe1, 0x24, 0xa1, 0xa5, 0x49, ...}, ...)
        /home/endrey/neo/neofs-node/pkg/services/object/acl/eacl/v2/object.go:52 +0x3fb
github.com/nspcc-dev/neofs-node/pkg/services/object/acl/eacl/v2.(*cfg).localObjectHeaders(0xc2583cdf8424537b?, {0x88, 0x3c, 0xd5, 0xb2, 0xe1, 0x24, 0xa1, 0xa5, 0x49, ...}, ...)
        /home/endrey/neo/neofs-node/pkg/services/object/acl/eacl/v2/headers.go:273 +0x12e
github.com/nspcc-dev/neofs-node/pkg/services/object/acl/eacl/v2.(*cfg).readObjectHeaders(0xc0000ba8a0, 0xc0001af9e0)
        /home/endrey/neo/neofs-node/pkg/services/object/acl/eacl/v2/headers.go:124 +0x6aa
github.com/nspcc-dev/neofs-node/pkg/services/object/acl/eacl/v2.NewMessageHeaderSource({0xc0001afb40, 0x4, 0xc0001afa98?})
        /home/endrey/neo/neofs-node/pkg/services/object/acl/eacl/v2/headers.go:74 +0xc5
github.com/nspcc-dev/neofs-node/pkg/services/object/acl/eacl/v2.TestHeadRequest.func1(0xc000189380)
        /home/endrey/neo/neofs-node/pkg/services/object/acl/eacl/v2/eacl_test.go:115 +0x16e
github.com/nspcc-dev/neofs-node/pkg/services/object/acl/eacl/v2.TestHeadRequest(0xc000189380)
        /home/endrey/neo/neofs-node/pkg/services/object/acl/eacl/v2/eacl_test.go:134 +0xd78
testing.tRunner(0xc000189380, 0xdf8488)
        /snap/go/current/src/testing/testing.go:1690 +0xf4
created by testing.(*T).Run in goroutine 1
        /snap/go/current/src/testing/testing.go:1743 +0x390
FAIL    github.com/nspcc-dev/neofs-node/pkg/services/object/acl/eacl/v2 0.013s

This commit make Version nil.
Maybe fix this by checking the version before.

cmd/neofs-cli/modules/util/acl.go

pkg/services/object/tombstone/verify_test.go

cmd/neofs-node/attributes.go

cmd/neofs-node/netmap.go

cthulhu-rider · 2024-11-19T12:24:31Z

@End-rey bout the tests:

public key absence must be pre-checked by the node. But no reason to panic at the lib, i'll add the check there
definitely, prms must be correctly filled. Lets both a) add the check and return error from engine b) fix input in the test
yep, latest decoder complains about missing mandatory token fields, although it shouldn't. This is a lib flaw, i'll fix it
nil version must be treated specifically now. As i can see u've already fixed this right?

End-rey · 2024-11-19T20:28:12Z

@cthulhu-rider I fixed everything that needed to be fixed. And what are we going to do with the 3rd point? Wait for the problem to close in the SDK?

cthulhu-rider · 2024-11-20T08:24:54Z

Wait for the problem to close in the SDK?

yeah but in the background, there is nothing to do here for now

https://github.com/nspcc-dev/neofs-node/actions/runs/11913864433/job/33200509043?pr=3005 bothers me more

roman-khimov · 2024-11-20T08:49:38Z

What's up with tests here?

roman-khimov · 2024-11-20T15:52:13Z

Well, the node is broken with the new SDK. We either fix something or revert to the old patch here that used api-go (pushing SDK update into a separate PR). I suggest the latter since I have no idea how long will it take to fix SDKish problems.

pkg/local_object_storage/blobstor/put.go

pkg/core/object/fmt.go

There was incorrect behavior in the test that was fixed. Signed-off-by: Andrey Butusov <[email protected]>

Signed-off-by: Andrey Butusov <[email protected]>

Add a new function `cfg.reloadNodeAttributes` that updates the list of node attributes. Add RWMutex for `cfg.cfgNodeInfo.localInfo`. Add docs. Closes #1870. Signed-off-by: Andrey Butusov <[email protected]>

End-rey self-assigned this Nov 8, 2024

End-rey requested review from roman-khimov, carpawell and cthulhu-rider as code owners November 8, 2024 12:17

roman-khimov reviewed Nov 8, 2024

View reviewed changes

End-rey force-pushed the 1870-support-reloading-node-attributes-with-sighup branch from b323897 to 59a00ed Compare November 11, 2024 17:05

End-rey requested a review from roman-khimov November 13, 2024 08:43

roman-khimov reviewed Nov 13, 2024

View reviewed changes

cmd/neofs-node/config.go Outdated Show resolved Hide resolved

cmd/neofs-node/attributes.go Outdated Show resolved Hide resolved

cmd/neofs-node/netmap.go Show resolved Hide resolved

cmd/neofs-node/netmap.go Show resolved Hide resolved

End-rey force-pushed the 1870-support-reloading-node-attributes-with-sighup branch from 59a00ed to 7a367b0 Compare November 13, 2024 15:20

roman-khimov approved these changes Nov 13, 2024

View reviewed changes

cthulhu-rider reviewed Nov 14, 2024

View reviewed changes

End-rey mentioned this pull request Nov 14, 2024

netmap: add GetAttributes and SetAttributes for NodeInfo nspcc-dev/neofs-sdk-go#635

Merged

End-rey force-pushed the 1870-support-reloading-node-attributes-with-sighup branch from 7a367b0 to 1a73e73 Compare November 18, 2024 13:14

morph: fix reloading morph endpoints with SIGHUP

a1bd703

Incorrect expression to check for reconnection. Before that, it worked the other way around, now is fixed. Signed-off-by: Andrey Butusov <[email protected]>

End-rey force-pushed the 1870-support-reloading-node-attributes-with-sighup branch 2 times, most recently from 52de9be to 47149c2 Compare November 18, 2024 15:26

cthulhu-rider reviewed Nov 19, 2024

View reviewed changes

This was referenced Nov 19, 2024

Latest upgrade fixes/improvements nspcc-dev/neofs-sdk-go#637

Draft

Checking for the fields' presence should not be done by unmarshalers nspcc-dev/neofs-sdk-go#638

Open

End-rey force-pushed the 1870-support-reloading-node-attributes-with-sighup branch from 47149c2 to 21d37b7 Compare November 19, 2024 13:16

cthulhu-rider approved these changes Nov 20, 2024

View reviewed changes

End-rey force-pushed the 1870-support-reloading-node-attributes-with-sighup branch 2 times, most recently from 48aa919 to 2815c4f Compare November 21, 2024 09:45

roman-khimov reviewed Nov 21, 2024

View reviewed changes

pkg/local_object_storage/blobstor/put.go Outdated Show resolved Hide resolved

pkg/core/object/fmt.go Show resolved Hide resolved

End-rey added 3 commits November 21, 2024 17:07

tests: fix tests

1c0012c

There was incorrect behavior in the test that was fixed. Signed-off-by: Andrey Butusov <[email protected]>

object: check for missing public key in validateSignatureKey

0462acc

Signed-off-by: Andrey Butusov <[email protected]>

node: support reloading node attributes with SIGHUP

d430daa

Add a new function `cfg.reloadNodeAttributes` that updates the list of node attributes. Add RWMutex for `cfg.cfgNodeInfo.localInfo`. Add docs. Closes #1870. Signed-off-by: Andrey Butusov <[email protected]>

End-rey force-pushed the 1870-support-reloading-node-attributes-with-sighup branch from 2815c4f to d430daa Compare November 21, 2024 14:08

roman-khimov approved these changes Nov 21, 2024

View reviewed changes

End-rey mentioned this pull request Nov 21, 2024

Update to the latest SDK #3023

Draft

carpawell approved these changes Nov 21, 2024

View reviewed changes

roman-khimov merged commit 1092989 into master Nov 21, 2024
20 of 22 checks passed

roman-khimov deleted the 1870-support-reloading-node-attributes-with-sighup branch November 21, 2024 15:58

roman-khimov mentioned this pull request Dec 2, 2024

Add SIGHUP handler tests nspcc-dev/neofs-testcases#901

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

node: support reloading node attributes with SIGHUP #3005

node: support reloading node attributes with SIGHUP #3005

End-rey commented Nov 8, 2024

codecov bot commented Nov 8, 2024 •

edited

Loading

roman-khimov left a comment

roman-khimov left a comment

End-rey commented Nov 18, 2024

cthulhu-rider commented Nov 19, 2024

End-rey commented Nov 19, 2024

cthulhu-rider commented Nov 20, 2024 •

edited

Loading

roman-khimov commented Nov 20, 2024

roman-khimov commented Nov 20, 2024

node: support reloading node attributes with SIGHUP #3005

node: support reloading node attributes with SIGHUP #3005

Conversation

End-rey commented Nov 8, 2024

codecov bot commented Nov 8, 2024 • edited Loading

Codecov Report

roman-khimov left a comment

Choose a reason for hiding this comment

roman-khimov left a comment

Choose a reason for hiding this comment

End-rey commented Nov 18, 2024

cthulhu-rider commented Nov 19, 2024

End-rey commented Nov 19, 2024

cthulhu-rider commented Nov 20, 2024 • edited Loading

roman-khimov commented Nov 20, 2024

roman-khimov commented Nov 20, 2024

codecov bot commented Nov 8, 2024 •

edited

Loading

cthulhu-rider commented Nov 20, 2024 •

edited

Loading