Request for elevated permissions

[maclover7]

(opening this now, so I don't forget, to continue discussion with @jbergstroem and @mhdawson from the last wg meeting)

Lately I have been one of the more active participants in the WG, and one of the only people keeping an eye on the node-build IRC channel. As a result of this, I am doing more and more of the WG's core tasks, however I often do not have sufficient permissions to actually perform them. Some recent situations I have personally encountered are:

• Getting ci.nodejs.org to successfully build and test security releases
• The web server has not rebuilt the site many times recently, and I have had to guide TSC members through the GitHub UI on how to get the site to update
• Release team members assume access to ci-release.nodejs.org, and need me to perform certain admin tasks
• Simplifying and extracting Jenkins job scripts

The WG projects I have been working on include trying to bring the Ansible refactor to a conclusion, triaging the issue tracker, Docker usage in test CI, and just generally keeping the wheels turning.

I'd like to request higher permissions, to complement the increased responsibility I have been taking on for the WG's business over the past several weeks/months.

[refack]

I would like to join this request.

As I see it there are several access points that could be considered independently:

1. access to www infra
2. access to github-bot infra
3. access to ci.nodejs.org infra
4. access to ci-release.nodejs.org infra
5. access to release machines
6. admin access to nodejs/node and nodejs/nodejs.org GitHub Repos

Also /CC @nodejs/security

[joyeecheung]

If I understand correctly, the requests translate to:

1. Adding your public keys to github-bot, release and infra secrets folder
2. Giving you admin access to nodejs/node and nodejs/nodejs.org

For 1 I think we'll need consensus from @nodejs/build and for 2 we will need consensus from @nodejs/tsc (maybe @nodejs/community-committee as well?)

I am +1 to the requests FWIW, but it would be good if the procedures can be documented as much as possible to make sure they can be passed on.

[phillipj]

+1 FWIW

More people knowing the github-bot with access to its server would be valuable. It usually means being an infra member, I'm just an exception that has been provided access to that server explicitly.

[maclover7]

@joyeecheung

If I understand correctly, the requests translate to:

For release, it would be running dotgpg add in the build/release directory of nodejs-private/secrets and adding me to the nodejs/jenkins-release-admins team

For infra, it would be again running dotgpg add in the build/ and build/infra directories of nodejs-private/secrets, and then creating individual accounts (I'd have to be added to the Node.js "group") for each of the hosting providers

consensus from @nodejs/tsc

Probably too late for today's TSC meeting, but I'm happy to attend next week's as an observer, or to answer questions about my request

[mcollina]

+1

[mhdawson]

Thanks for opening and starting the discussion with examples. I like @refack's list of what could be considered independently. @refack @maclover7 do you believe you need access to all of those or a subset?

[refack]

I believe that we need more active collaborators with access to all these point for the benefit the org. All of those points needed attention in the last couple of months.

If any item is considered "too risky" then the decision on that item can be deferred, this way we don't have an all or nothing situation.

For example I did not list access to the infra providers, since IMHO it is low priority.
Actually the list is roughly prioritized according to my memory of incidents.

[mhdawson]

+1 to greater coverage across the board.

[maclover7]

@mhdawson

do you believe you need access to all of those or a subset?

Part of the issue is that all of the machines within the various access groups (ex: test, release, infra) all use the same SSH key within that group. Without reconfiguring a lot of our existing stuff, this would make it hard to give out access in small bits, and also make it hard to keep track of who has access to what services.

The issues I have encountered seem to point to being added to the infra and release access groups, in order to be able to resolve them in the future.

+1 to greater coverage across the board.

I've been trying to get the Ansible playbooks in the best state possible, and adding new docs about our Jenkins test cluster as much as I can. My hope is that this will lead to a less stress when trying to deal with issues.

[Trott]

I don't know exactly what's involved, but I am +100 to upgrading @maclover7 and @refack to more trusted levels of access in Build WG if it helps them be more effective. Both of them have been more active than anyone as far as I know in Build WG lately. No disrespect to awesome folks like @joaocgreis, @jbergstroem, @rvagg, and @mhdawson (and others--not a comprehensive list), but @maclover7 especially and @refack too probably pay more attention and do more than everyone else combined every single day to keep stuff from grinding to a halt. I cannot overstate how important their Build WG work is to the project.

In the past, I know one question asked before elevating privileges was "Does the person have an employer such that there will be consequences if they misbehave?" I wonder if recent experiences suggest that we might want to reconsider that criteria. I understand the motivation. But it's also true that Build WG hasn't been able to scale up with the rest of the project and it's causing problems for both the project at large and the Build WG membership in particular. I don't have a good counter-proposal, but I welcome thoughts/suggestions. I do think that @maclover7 and @refack have earned sufficient trust that this should be a no-brainer.

[mhdawson]

Adding @rvagg @joaocgreis and @gibfahn as well.

[jbergstroem]

I agree to adding more people to infra (and release; they were designed to be different for similar reasons, being able to help with releases versus core infra. Perhaps a side track could be to revisit what architecture/software should live where?) and after going through the commit, issue and pr logs I can only chime in with what others seem to agree on here.

While doing so it would also be great if we could get some kind of outlines going in terms of how we evaluate this criteria. As it was said in the last build meeting: "There are people watching who has access to release machines".

[rvagg]

I want to make sure that while we're discussing this that we're clear on some of the challenges we have in opening up this stuff to a broader group and why there's friction in doing so.

We're talking about critical infrastructure that impacts on all aspects of security of our build pipeline and impacts on the relationships that we have carefully managed with infrastructure donors. We've always focused discussions about this on "trust" and "accountability". The unfortunate fact is that it's much easier to establish these two things with employees of companies that have things to loose, particularly the big guys. IBM makes that easy, as does Microsoft (and by proxy through Janea Systems). Individuals who work for those companies also have their employer to answer to and have legal contracts in place that hold them accountable so it's much more difficult to imagine a scenario where we have sever leakage or malicious action where there's isn't some kind of serious legal ramifications. So that covers Michael, Gibson (now at Apple so just as easy), and Joao.

I have NodeSource, although we're a smaller company, but I also basically started this whole infra thing (it was the key deliverable out of the node-forward effort and pre-dates even io.js and talk of a Foundation) and have built a large majority of what we have now and have been maintaining a lot of the relationships we have with sponsors. Johan is unaffiliated but also has a very long history here and a significant investment in doing the actual work.

So, when adding new people, we have to look very seriously at mitigating the risk of sharing core secrets (certs, etc.) of the technical project and also the relationships with sponsors that tend to need special care, or walking within not very well defined boundaries. I think we would very much like to add both @refack and @maclover7 and it's crystal clear that they are both dedicated to our infra and are technically very capable. But they're also unaffiliated and have no chain of accountability that would provide assurance for the project against malicious action beyond trust gained through action--not that that trust isn't significant. The connection of the trust placed in you by the project with your very employment serves as a very important piece of the puzzle. If I screw up, if I have some kind of flip-out and take down our infra or inject some malicious code into releases (trivially possible to do transparently with the kind of access we're talking about here, even with all our other checks in place and not even with full access to key resources), then I suffer, not just from project blow-back but because it'd fall on my employer so I'd probably be out of a job and also be in for some legal action due to reputational damage.

So, please don't think we're dragging our heels on this because we want to protect some kind of special inner sanctum (it's not that glamorous and also kind of annoying having a small group with the knowledge & access), there's critical questions to be answered here that impact us all and those questions can't be simply answered by the kind of feels that we use across the rest of the project.

There may be a role here for the Foundation to help provide us with legal help in establishing a formal trust relationship, perhaps something we can sign. But given our history with being able to extract any legal help from the Foundation (for even trivial things like can we leave a date off our copyright notices???) I'm not really hopeful that they're going to be useful here in the slightest.

[refack]

So, please don't think we're dragging our heels on this because we want to protect some kind of special inner sanctum...

I think that is clear, and I tried to reiterate that during the last WG meeting.
An added point is that access to the release machines will add new members' identities to the audit trail of the node binaries.

There may be a role here for the Foundation to help provide us with legal help in establishing a formal trust relationship, perhaps something we can sign.

IMHO that's a great idea 👍

[bnb]

cc @MylesBorins - I think you'd be interested in the last bit of @rvagg's comment above, RE: Foundation work.

[maclover7]

But they're also unaffiliated and have no chain of accountability that would provide assurance for the project against malicious action beyond trust gained through action--not that that trust isn't significant.

I agree. As you mentioned with Foundation legal resources, I am happy to assist and try to setup mechanisms that create more accountability that helps to protect Node as a project. The ultimate issue we're dealing with is how to expand access to the essential assets of the project -- nodejs.org, release servers -- in a responsible way that properly safeguards them.

[maclover7]

This is squeezing us currently, there's an issue with the github-bot (in the "infra" machine group) that requires SSH access

[joyeecheung]

I am tagging the issue tsc-agenda to discuss at the next TSC meeting.

[rvagg]

Next TSC meeting is one of the time slots I won't be able to attend, I'd appreciate my comments above (#1337 (comment)) being represented clearly. I suspect the instinct from others who don't normally have to think about these kind of trust & security issues will be to be more liberal about handing out access. That's the TSC's choice, ultimately, but as a group that's entrusted with managing the integrity of most of our infrastructure it's important that these concerns are heard and the reasoning behind our processes are properly represented.

On the question of involving the Foundation to get legals sorted out, I'd suggest we not hold our breath to see any action on this, history isn't on our side in getting material help on that kind of thing. Let's try and move forward with the assumption that we're not going to get much help any time soon. One way to do that might be to identify all of the different kinds of access that "infra" has and see if we can atomise it a bit more so it's not all-or-nothing. That way we might be able to identify ways we can delegate access without having to have a complete trust framework in place.

[mhdawson]

Based on the discussion I'll post the list from above:

• access to www infra
• access to github-bot infra
• access to ci.nodejs.org infra
• access to ci-release.nodejs.org infra
• access to release machines
• admin access to nodejs/node and nodejs/nodejs.org GitHub Repos

@refack, @maclover7 if you had to prioritize that list based on what is holding you back the most from
being able to complete day to day activities. This might help in the "atomise" it discussion.

[maclover7]

The two main things holding me back right now are access to the www server when website pushes aren't working and being a Jenkins admin on ci-release.nodejs.org

[refack]

IMHO this could be pushed to a week when @rvagg is available to present the reasoning behind the status quo.

[mhdawson]

@maclover7 are there issues for the work related to ci-release.nodejs.org? I think I understand the www side just want to better understand where you've been pulled in on the ci-release.nodejs.org side.

[maclover7]

@maclover7 are there issues for the work related to ci-release.nodejs.org?

@mhdawson During the security release issues, Refael was the only one around on IRC that is part of nodejs/jenkins-release-admins, so would be good to have more coverage there. Also, I'd like to work on a new job in the release CI that can let people retrigger deploys to nodejs.org, so doesn't have to escalate to pinging build folks.

[joyeecheung]

@Trott At this week's TSC meeting it was proposed that we take the idea in nodejs/TSC#553 (comment) and schedule a special meeting to chat about alternatives (e.g. scripted tasks that can be triggered without direct access to machines) and workarounds for the accountability issues before coming back to the TSC for specific requests to approve.

I'll set up a doodle later.

[joyeecheung]

I opened #1373 with a doodle and an agenda

[sam-github]

Why is this on the build WG agenda still? It looks like @refack labelled it in June 2018, can we take it off, or figure out what needs discussion?

[github-actions]

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.