Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Profile removal fails if host never received the profile #21891

Closed
ddribeiro opened this issue Sep 6, 2024 · 12 comments
Closed

Profile removal fails if host never received the profile #21891

ddribeiro opened this issue Sep 6, 2024 · 12 comments
Assignees
@jahzielv
Labels
bug Something isn't working as documented customer-eponym customer-preston customer-reedtimmer customer-rosner #g-mdm MDM product group P2 Prioritize as urgent :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. ~released bug This bug was found in a stable release.
Milestone
4.61.0

Comments

@ddribeiro
Copy link
Member

Fleet version: 4.55.0 and in Dogfood SNAPSHOT-1874915

💥  Actual behavior

When a macOS configuration profile is added to a team and then removed from that team before a host comes online, Fleet will send an MDM command to remove the profile from that host. However, since the profile was never delivered, the removal command fails:

Failed to remove: MDMClientError (89): Profile with identifier '<profile_identifier_here>' not found.

🧑‍💻  Steps to reproduce

  1. With your test host powered off, add a .mobileconfig profile to the team that host belongs to.
  2. Remove the host from the team.
  3. Power your host back on. Go to that host’s details page and click Failed under OS settings. You’ll see that Fleet attempted to send a RemoveProfile MDM command that failed because the profile with that identifier never existed on the host.

🕯️ More info (optional)
@ddribeiro ddribeiro added bug Something isn't working as documented :reproduce Involves documenting reproduction steps in the issue :incoming New issue in triage process. customer-reedtimmer labels Sep 6, 2024
@georgekarrv georgekarrv added :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. #g-mdm MDM product group and removed :reproduce Involves documenting reproduction steps in the issue :incoming New issue in triage process. labels Sep 9, 2024
@JoStableford
Copy link
Contributor

Related to a Slack conversation

@jahzielv jahzielv self-assigned this Sep 12, 2024
@lukeheath lukeheath added the ~released bug This bug was found in a stable release. label Sep 13, 2024
@jahzielv jahzielv mentioned this issue Sep 17, 2024
Merged
3 tasks
jahzielv added a commit that referenced this issue Sep 19, 2024
@jahzielv
fix: stop profiles that never reached the host from showing up as fai…
90c04ee 
…led (#22186)

> Related issue: #21891

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
@jahzielv jahzielv added this to the 4.58.0-tentative milestone Sep 20, 2024
@PezHub PezHub mentioned this issue Sep 24, 2024
Closed
@PezHub
Copy link
Contributor

PezHub commented Sep 24, 2024

Hey @jahzielv I'm still seeing this issue and can reproduce by uploading a profile to a team then quickly deleting it before it gets installed on the host. Fleet sends the remove command to the host without the profile installed so the uninstall fails.

Here's a loom video walk thru (including a workaround to fix at the end)

I believe it may be related to this issue as well #22121

@jahzielv
Copy link
Contributor

can reproduce by uploading a profile to a team then quickly deleting it before it gets installed on the host

Ok, I think I've figured out why that's happening. We're currently removing profiles that never made it to the host in a cron job, but adding and quickly deleting will most likely happen in between cron job runs, so the logic that removes the profile doesn't happen and the zombie host profile hangs around.

Now I have to figure out a way to catch profiles in that state without the cron job; I'll report back when I have something.

@jahzielv jahzielv mentioned this issue Sep 24, 2024
Merged
3 tasks
jahzielv added a commit that referenced this issue Oct 1, 2024
@jahzielv
fix: attempt to prevent race in profile adding and deleting (#22338)
24c84ed 
> Follow up on: #21891

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
@jahzielv
Copy link
Contributor

jahzielv commented Oct 1, 2024

@PezHub howdy! I merged a fix for the behavior you were seeing here: #22338

I'm moving this back to "awaiting QA"!

@PezHub
Copy link
Contributor

PezHub commented Oct 7, 2024

QA notes:

  1. uploaded a config profile and deleted it within 5 secs to create the orphaned stated
  2. confirmed it was stuck in pending removal soon after
  3. upon sending the refetch command the profile was cleared and removed from the queue

QA Approved!
Screenshot 2024-10-07 at 2 59 50 PM
Screenshot 2024-10-07 at 3 02 09 PM
Screenshot 2024-10-07 at 3 00 10 PM

This was referenced Oct 17, 2024
Closed
Closed
@fleet-release
Copy link
Contributor

Profile removal glitch,
Like clouds clearing, now all's fixed,
Fleet sails without hitch.

@georgekarrv georgekarrv removed the :demo label Oct 25, 2024
@mna mna mentioned this issue Oct 28, 2024
Closed
@jahzielv jahzielv mentioned this issue Nov 13, 2024
Closed
@lashomb
Copy link

lashomb commented Nov 20, 2024

We are still seeing this after upgrading to 4.58.

@Patagonia121
Copy link
Member

customer-eponymis reporting this issue is still ongoing despite being on 4.58, so I'm going to reopen this issue so we can start to investigate this further.

@Patagonia121 Patagonia121 reopened this Nov 20, 2024
@lukeheath lukeheath modified the milestones: 4.58.0, 4.60.0 Nov 21, 2024
@lukeheath
Copy link
Member

@jahzielv @georgekarrv Heads up this was re-opened. Please take a look and try to get a fix into v4.60.0 if possible. Thanks!

@lukeheath lukeheath modified the milestones: 4.60.0, 4.61.0-tentative Nov 21, 2024
@lukeheath
Copy link
Member

@jahzielv @georgekarrv I see v4.60.0 won't work since we're in deep freeze, but I'm adding P2 so we can prioritize over other bugs next week.

@lukeheath lukeheath added the P2 Prioritize as urgent label Nov 21, 2024
@jahzielv jahzielv mentioned this issue Dec 2, 2024
Open
@jahzielv
Copy link
Contributor

jahzielv commented Dec 3, 2024

Re-closing this bug; a feature request for product consideration has been filed here: #24264

@jahzielv jahzielv closed this as completed Dec 3, 2024
@fleet-release
Copy link
Contributor

Profile removal fail,
Fix lets hosts rise like the sun,
Clear skies, no detail stale.

@marko-lisica marko-lisica mentioned this issue Dec 20, 2024
Open
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jahzielv jahzielv

Labels
bug Something isn't working as documented customer-eponym customer-preston customer-reedtimmer customer-rosner #g-mdm MDM product group P2 Prioritize as urgent :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. ~released bug This bug was found in a stable release.
Projects
None yet
Milestone
4.61.0
Development

No branches or pull requests
9 participants
@lashomb @georgekarrv @lukeheath @ddribeiro @jahzielv @Patagonia121 @JoStableford @fleet-release @PezHub