Changes to binary distribution #6542

michalpristas · 2025-01-17T16:13:53Z

Ready for review but DO NOT MERGE before spec files of dependencies are updated in other repos

What does this PR do?

This PR changes logic of installation and ugprades in following manner.

Default install installs only:

agentbeat
endpoint-security
pf-host-agent

additional flag is added that includes components above and:

cloudbeat
apm-server
fleet-server
pf-elastic-symbolizer
pf-elastic-collector

Upon installation user choice is backed into .flavor file.

Part of elastic agent package is .flavors file defining flavor for this particular version.
Upon upgrade or install .flavors is read and only components specified in desired flavor (using flag with install or content of .flavor in case of upgrade) are copied.

In case of any issues with parsing any of the above files we fall back to previous way and copy everything.

Each components needs to define files in addition to binary, spec and config in a form

component_files:
 - file1
 - dir1/*

in their spec file.

During upgrade target version spec file is parsed. not the current installation.

Why is it important?

Reducing space on disk by omitting not needed components.

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation TODO: @michalpristas
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

How to test this PR locally

Scenario 1 and 2 (installation)

Build package and install without and with flag --install-server
check components present

Scenario 3 - upgrade to future version

Prepare same agent but with different commit hash (do small change and commit locally)
Run upgrade 9.0.0-SNAPSHOT --source-uri="file://....zip" --skip-verify and check only desired components are present

Scenario 4 - upgrade from stale version

Upgrade from previous version to this one and check ALL components are present

…ary-distr

cmacknz · 2025-01-17T20:39:30Z

You will need to add integration tests to make sure upgrades respect the flavor or lack of one.

For the existing upgrade tests you could probably add a check to ensure every component exists in both versions easily. For flavors you'll need new tests for those case most likely.

internal/pkg/agent/application/upgrade/step_unpack_test.go

cmacknz · 2025-01-17T20:53:57Z

internal/pkg/agent/application/upgrade/upgrade.go

-	unpackRes, err := u.unpack(version, archivePath, paths.Data())
+	// default to extended flavor to avoid breaking behavior
+
+	// no default flavor, keep everything in case flavor is not


keep everything in case flavor is not

Is not what?

cmacknz · 2025-01-17T21:27:39Z

internal/pkg/agent/install/.flavors

@@ -0,0 +1,13 @@
+basic:


What value are we getting out of making this a file and having to deal with opening+parsing it vs just embedded this list into the agent as code?

upgrade scenario.
if you want to change content of flavor in between version you need somehow know what should be present in a new version. upgrade code is executed from stale version not the new one so embedded content would reflect state of things of version-1. if you add a new component or change layout of existing one in a new version it would be stripped and break

OK, so if we add a new binary in the future we to communicate to the old agent what it needs to extract?

This feels very similar to what the manifest does with metadata to let us pull determine the extraction path.

elastic-agent/internal/pkg/agent/application/upgrade/step_unpack.go

Line 92 in 7d1aaae

metadata, err := getPackageMetadataFromZipReader(r, fileNamePrefix)

Can we make the definition of the flavors part of the manifest file and then pull the list out before extracting the rest of the zip/tar? That would make this work in an already proven way and make the flavours a property of a specific agent version which is easy to reason about.

it should be ok to merge if you're ok with mixing concerns. the only difference is that manifest is in versioned home after installation, but it should not matter in this case.

Keeping it in versioned home is a benefit IMO so that it is a property of an individual agent version. I'd rather not introduce something new for this if we can handle this with an existing mechanism.

CC @pchila for opinions on use of the manifest file for defining the flavors per agent version since he implemented it originally.

updated to use manifest in a meanwhile,
if @pchila objects i can revert. but i agree this is cleaner and removes one traversal through archive before unpacking during upgrade

michalpristas · 2025-01-20T09:30:44Z

Spec file updated for agentbeat here: elastic/beats#42348

…ary-distr

michalpristas · 2025-01-20T10:24:21Z

Fixing windows

…ary-distr

elasticmachine · 2025-01-21T08:59:07Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

…ary-distr

elastic-sonarqube · 2025-01-21T10:06:14Z

Quality Gate passed

Issues
8 New issues
5 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
68.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

internal/pkg/agent/application/upgrade/upgrade.go

andrzej-stencel · 2025-01-22T12:19:16Z

internal/pkg/agent/install/flavors.go

+}
+
+// SpecsForFlavor returns spec files associated with specific flavor
+func SpecsForFlavor(flavor FlavorDefinition) ([]string, error) {


This function always returns nil as error, is the error needed?

leftover from previous design, thanks for pointing it out

andrzej-stencel · 2025-01-22T12:20:51Z

internal/pkg/agent/application/upgrade/step_unpack.go

+
+	// fix versionedHome
+	versionedHome = strings.ReplaceAll(versionedHome, "\\", "/")
+	log.Errorf("using %q", versionedHome)


Is this needed? It's not an error, right?

it's not an error, this is because zip on windows are using unix paths 🤦

Co-authored-by: Andrzej Stencel <[email protected]>

…agent into feat/binary-distr

…ary-distr

michalpristas added 23 commits November 22, 2024 12:03

mark and skip flaky TestOTelManager_Run

b937799

Merge branch 'main' of github.com:elastic/elastic-agent

ac0cce2

Merge branch 'main' of github.com:elastic/elastic-agent

42c4611

Merge branch 'main' of github.com:elastic/elastic-agent

326add2

Merge branch 'main' of github.com:elastic/elastic-agent

fa83554

Merge branch 'main' of github.com:elastic/elastic-agent

a6dc99b

basic works

d23268c

zip works

9b7e08f

cleanup

1919511

ut for componentsForFLavor and subpathsForComponent

a99e9e6

ut for componentsForFLavor and subpathsForComponent

a119e5a

TestAllowedSubpathsForFlavor

33352ea

TestSkipComponentsPathWithSubpathsFn

ffd57b1

TestSkipComponentsPathFn

4016c44

TestSkipComponentsPathFn

350b087

TestParseComponentFiles

e0f3aef

TestFlavor

cd417c8

TestSpecsForFlavor

41ee06a

fix unpack and tests

b32f0c1

imports

6055adb

integ tests

863adc5

flavours in file

b44902e

Merge branch 'main' of github.com:elastic/elastic-agent into feat/bin…

822e308

…ary-distr

michalpristas added enhancement New feature or request backport-skip labels Jan 17, 2025

michalpristas self-assigned this Jan 17, 2025

michalpristas requested a review from a team as a code owner January 17, 2025 16:13

michalpristas requested review from andrzej-stencel and swiatekm January 17, 2025 16:13

lint

aadc6fe

changelog

51bd2c6

cmacknz reviewed Jan 17, 2025

View reviewed changes

internal/pkg/agent/application/upgrade/step_unpack_test.go Outdated Show resolved Hide resolved

cmacknz reviewed Jan 17, 2025

View reviewed changes

unpack UTs

67ed4a7

michalpristas mentioned this pull request Jan 20, 2025

Updated spec file for agentbeat to reflect new binary distribution elastic/beats#42348

Open

michalpristas added 2 commits January 20, 2025 10:31

spec files updated

0f50031

Merge branch 'main' of github.com:elastic/elastic-agent into feat/bin…

8a60a47

…ary-distr

michalpristas added 5 commits January 20, 2025 11:51

handle windows correctly in UTs

a095684

linter

7cf5147

install server for fleet server tests

01cef9c

Merge branch 'main' of github.com:elastic/elastic-agent into feat/bin…

0696c75

…ary-distr

install server for fleet server tests

6c39e92

swiatekm added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Jan 21, 2025

michalpristas added 2 commits January 21, 2025 10:12

flavors in manifest

724e852

Merge branch 'main' of github.com:elastic/elastic-agent into feat/bin…

255a8dc

…ary-distr

andrzej-stencel reviewed Jan 22, 2025

View reviewed changes

internal/pkg/agent/application/upgrade/upgrade.go Outdated Show resolved Hide resolved

andrzej-stencel reviewed Jan 22, 2025

View reviewed changes

michalpristas and others added 4 commits January 23, 2025 09:59

Update internal/pkg/agent/application/upgrade/upgrade.go

5117a62

Co-authored-by: Andrzej Stencel <[email protected]>

specs for flavor returning no error

81e24e5

Merge branch 'feat/binary-distr' of github.com:michalpristas/elastic-…

492bd35

…agent into feat/binary-distr

Merge branch 'main' of github.com:elastic/elastic-agent into feat/bin…

1bdff99

…ary-distr

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changes to binary distribution #6542

Changes to binary distribution #6542

michalpristas commented Jan 17, 2025

cmacknz commented Jan 17, 2025

cmacknz Jan 17, 2025

michalpristas Jan 20, 2025

cmacknz Jan 17, 2025 •

edited

Loading

michalpristas Jan 20, 2025 •

edited

Loading

cmacknz Jan 20, 2025

michalpristas Jan 21, 2025

cmacknz Jan 21, 2025

michalpristas Jan 22, 2025 •

edited

Loading

michalpristas commented Jan 20, 2025

michalpristas commented Jan 20, 2025

elasticmachine commented Jan 21, 2025

elastic-sonarqube bot commented Jan 21, 2025

andrzej-stencel Jan 22, 2025

michalpristas Jan 23, 2025

andrzej-stencel Jan 22, 2025

michalpristas Jan 23, 2025 •

edited

Loading

Changes to binary distribution #6542

Are you sure you want to change the base?

Changes to binary distribution #6542

Conversation

michalpristas commented Jan 17, 2025

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Scenario 1 and 2 (installation)

Scenario 3 - upgrade to future version

Scenario 4 - upgrade from stale version

cmacknz commented Jan 17, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmacknz Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

michalpristas Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michalpristas Jan 22, 2025 • edited Loading

Choose a reason for hiding this comment

michalpristas commented Jan 20, 2025

michalpristas commented Jan 20, 2025

elasticmachine commented Jan 21, 2025

elastic-sonarqube bot commented Jan 21, 2025

Quality Gate passed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michalpristas Jan 23, 2025 • edited Loading

Choose a reason for hiding this comment

cmacknz Jan 17, 2025 •

edited

Loading

michalpristas Jan 20, 2025 •

edited

Loading

michalpristas Jan 22, 2025 •

edited

Loading

michalpristas Jan 23, 2025 •

edited

Loading