Monorepos (greenkeeperio#140)

feat: support for monorepos and dependencies from monorepos

BREAKING CHANGE: multiple breaking changes, see list below.

- needs testing with CI services other than TravisCI
- CI Environment Variable needed for default branches other than `master`

README.md

@@ -1,4 +1,4 @@
-# greenkeeper-lockfile
+# Greenkeeper Lockfile
 
 After [enabling Greenkeeper for your repository](https://github.com/integration/greenkeeper) you can use this package to make it work with lockfiles, such as `npm-shrinkwrap.json`, `package-lock.json` or `yarn.lock`.
 
@@ -11,6 +11,21 @@ After [enabling Greenkeeper for your repository](https://github.com/integration/
 [![js-standard-style](https://img.shields.io/badge/code%20style-standard-brightgreen.svg?style=flat)](https://github.com/feross/standard)
 [![semantic-release](https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg)](https://github.com/semantic-release/semantic-release)
 
+- [Greenkeeper Lockfile](#greenkeeper-lockfile)
+	- [Package Managers](#package-managers)
+	- [CI Services](#ci-services)
+	- [How does it work](#how-does-it-work)
+	- [Setup](#setup)
+	- [Using Greenkeeper with Monorepos](#using-greenkeeper-with-monorepos)
+	- [Testing multiple node versions](#testing-multiple-node-versions)
+	- [CircleCI workflows](#circleci-workflows)
+	- [TeamCity Setup](#teamcity-setup)
+	- [Configuration options](#configuration-options)
+	- [Contributing a CI Service](#contributing-a-ci-service)
+		- [Environment information](#environment-information)
+		- [Detecting your service](#detecting-your-service)
+		- [Testing your service](#testing-your-service)
+
 ## Package Managers
 
 * ✅ npm _(including npm5)_
@@ -46,6 +61,10 @@ After [enabling Greenkeeper for your repository](https://github.com/integration/
 
 3. Configure your CI to run `greenkeeper-lockfile-update` right before it executes your tests and `greenkeeper-lockfile-upload` right after it executed your tests.
 
+_The next Step is only applicable greenkeeper-lockfile version 2 (with monorepo support)_
+
+4. If you use a default branch that is **not** `master` then you have to add the environment variable `GK_LOCK_DEFAULT_BRANCH` with the name of your default branch to your CI.
+
 
 ### Example Travis CI configurations
 
@@ -80,6 +99,12 @@ after_script: greenkeeper-lockfile-upload
 
 To run the lockfile-update script with custom command line arguments, set the `GK_LOCK_YARN_OPTS` environment variable to your needs (set it to `--ignore-engines`, for example). They will be appended to the `yarn add` command.
 
+## Using Greenkeeper with Monorepos
+
+greenkeeper-lockfile 2.0.0 offers support for monorepos. To use it make sure you install `greenkeeper-lockfile@2` explicitly.
+
+If you are using a default branch on Github that is **not** called `master`, please set an Environment Variable `GK_LOCK_DEFAULT_BRANCH` with the name of your default branch in your CI.
+
 ## Testing multiple node versions
 
 It is common to test multiple node versions and therefor have multiple test jobs for one build. In this case the lockfile will automatically be updated for every job, but only uploaded for the first one.
@@ -96,7 +121,7 @@ before_script: greenkeeper-lockfile-update
 after_script: greenkeeper-lockfile-upload
 ```
 
-### CircleCI workflows
+## CircleCI workflows
 
 In order to use `greenkeeper-lockfile` with CircleCI workflows, it must be in the first job run. Use [sequential job execution](https://circleci.com/docs/2.0/workflows/#sequential-job-execution-example) to ensure the job that runs `greenkeeper-lockfile` is always executed first. For example, if `greenkeeper-lockfile` is run in the `lockfile` job, all other jobs in the workflow must require the `lockfile` job to finish before running:
 
@@ -111,14 +136,23 @@ workflows:
             - lockfile
 ```
 
-### TeamCity Setup
+## TeamCity Setup
 
 In order for this to work with TeamCity, the build configuration needs to set
 the following environment variables:
 
 - VCS_ROOT_URL from the vcsroot.<vcsrootid>.url parameter
 - VCS_ROOT_BRANCH from the teamcity.build.branch parameter
 
+## Configuration options
+
+| Environment Variable  | default value | what is it for? |
+| ------------- | ------------- | ------------- |
+| GK_LOCK_YARN_OPTS  | ''  | Add yarn options that greenkeeper should use e.g. `--ignore-engines`  |
+| GK_LOCK_DEFAULT_BRANCH  | 'master'  | Set your default github branch name |
+| GK_LOCK_COMMIT_AMEND  | false  | Lockfile commit should be amended to the regular Greenkeeper commit  |
+| GK_LOCK_COMMIT_NAME  | 'greenkeeperio-bot'  | Set your prefered git commit name  |
+| GK_LOCK_COMMIT_EMAIL  | '[email protected]'  | Set your prefered git commit email  |
 
 ## Contributing a CI Service
 

ci-services/bitrise.js

@@ -1,5 +1,3 @@
-const gitHelpers = require('../lib/git-helpers')
-
 const env = process.env
 
 // http://devcenter.bitrise.io/faq/available-environment-variables/
@@ -19,9 +17,6 @@ module.exports = {
   repoSlug: parseRepoSlug(env.GIT_REPOSITORY_URL),
   // The name of the current branch
   branchName: env.BITRISE_GIT_BRANCH,
-  // Is this the first push on this branch
-  // i.e. the Greenkeeper commit
-  firstPush: gitHelpers.getNumberOfCommitsOnBranch(env.BITRISE_GIT_BRANCH) === 1,
   // Is this a regular build
   correctBuild: env.PR === 'false',
   // Should the lockfile be uploaded from this build

ci-services/buildkite.js

@@ -7,7 +7,6 @@ const env = process.env
 module.exports = {
   repoSlug: gitHelpers.getRepoSlug(env.BUILDKITE_REPO),
   branchName: env.BUILDKITE_BRANCH,
-  firstPush: gitHelpers.getNumberOfCommitsOnBranch(env.BUILDKITE_BRANCH) === 1,
   correctBuild: env.BUILDKITE_PULL_REQUEST === 'false',
   uploadBuild: true
 }

ci-services/circleci.js

@@ -7,7 +7,6 @@ const env = process.env
 module.exports = {
   repoSlug: `${env.CIRCLE_PROJECT_USERNAME}/${env.CIRCLE_PROJECT_REPONAME}`,
   branchName: env.CIRCLE_BRANCH,
-  firstPush: !env.CIRCLE_PREVIOUS_BUILD_NUM,
   correctBuild: _.isEmpty(env.CI_PULL_REQUEST),
   uploadBuild: env.CIRCLE_NODE_INDEX === `${env.BUILD_LEADER_ID || 0}`
 }

ci-services/codeship.js

@@ -40,9 +40,6 @@ module.exports = {
   repoSlug: getRepoSlug(),
   // The name of the current branch
   branchName: env.CI_BRANCH,
-  // Is this the first push on this branch
-  // i.e. the Greenkeeper commit
-  firstPush: shouldUpdate(),
   // Is this a regular build (use tag: ^greenkeeper/)
   correctBuild: shouldUpdate(),
   // Should the lockfile be uploaded from this build (use tag: ^greenkeeper/)

ci-services/jenkins.js

@@ -2,8 +2,6 @@
 
 const env = process.env
 
-const gitHelpers = require('../lib/git-helpers')
-
 // Jenkins reports the branch name and Git URL in a couple of different places depending on use of the new
 // pipeline vs. older job types.
 const gitUrl = env.CHANGE_URL || env.GIT_URL
@@ -16,7 +14,6 @@ const branchName = matchesGreenkeeper ? matchesGreenkeeper[0] : origBranch
 module.exports = {
   gitUrl,
   branchName,
-  firstPush: env.BUILD_NUMBER === '1' || gitHelpers.getNumberOfCommitsOnBranch(env.GIT_BRANCH) === 1,
   correctBuild: true, // assuming this is always the correct build to update the lockfile
   uploadBuild: true // assuming 1 build per branch/PR
 }

ci-services/semaphoreci.js

@@ -7,7 +7,6 @@ const env = process.env
 module.exports = {
   repoSlug: env.SEMAPHORE_REPO_SLUG,
   branchName: env.BRANCH_NAME,
-  firstPush: env.SEMAPHORE_BUILD_NUMBER === '1',
   correctBuild: _.isEmpty(env.PULL_REQUEST_NUMBER),
   uploadBuild: env.SEMAPHORE_CURRENT_JOB === '1'
 }

ci-services/teamcity.js

@@ -29,7 +29,6 @@ function shouldUpload () {
 module.exports = {
   repoSlug: gitHelpers.getRepoSlug(env.VCS_ROOT_URL),
   branchName: env.VCS_ROOT_BRANCH,
-  firstPush: shouldUpload(),
   correctBuild: !isPullRequest(),
   uploadBuild: shouldUpload()
 }

ci-services/travis.js

@@ -7,9 +7,6 @@ module.exports = {
   repoSlug: env.TRAVIS_REPO_SLUG,
   // The name of the current branch
   branchName: env.TRAVIS_BRANCH,
-  // Is this the first push on this branch
-  // i.e. the Greenkeeper commit
-  firstPush: !env.TRAVIS_COMMIT_RANGE,
   // Is this a regular build
   correctBuild: env.TRAVIS_PULL_REQUEST === 'false',
   // Should the lockfile be uploaded from this build

ci-services/wercker.js

@@ -1,13 +1,10 @@
 'use strict'
 
-const gitHelpers = require('../lib/git-helpers')
-
 const env = process.env
 
 module.exports = {
   repoSlug: `${env.WERCKER_GIT_OWNER}/${env.WERCKER_GIT_REPOSITORY}`,
   branchName: env.WERCKER_GIT_BRANCH,
-  firstPush: gitHelpers.getNumberOfCommitsOnBranch(env.WERCKER_GIT_BRANCH) === 1,
   correctBuild: env.WERCKER_GIT_DOMAIN === 'github.com',
 
   // In wercker, only add the upload step to the pipeline you'd want to upload from

lib/extract-dependency.js

@@ -17,7 +17,34 @@ module.exports = function extractDependency (pkg, branchPrefix, branch) {
     })
   }))
 
-  return _.find(allDependencies, (dependency) => {
-    return branch === branchPrefix + dependency.name + '-' + dependency.version
+  // nicked from https://github.com/greenkeeperio/greenkeeper/blob/master/lib/validate-greenkeeper-json.js#L9
+  const groupRex = '([a-zA-Z0-9_-]+/)?'
+
+  const monorepoReleaseRex = RegExp(`${branchPrefix}${groupRex}monorepo.([a-zA-Z0-9_-]+)-([.0-9-]+)`)
+
+  if (monorepoReleaseRex.test(branch)) {
+    const monorepoDefinintions = require('greenkeeper-monorepo-definitions')
+    const monorepoDefinitionGroupName = branch.match(monorepoReleaseRex)[2]
+
+    if (!monorepoDefinitionGroupName) {
+      return console.error('Could not extract the dependency group name from the branch name.')
+    }
+
+    const monorepo = monorepoDefinintions[monorepoDefinitionGroupName]
+    if (!monorepo) {
+      return console.error(`${monorepoDefinitionGroupName} is missing in Greenkeeper's monorepo definitions`)
+    }
+
+    const dependencies = allDependencies.filter(dependency => {
+      return monorepo.includes(dependency.name)
+    })
+
+    return _.compact(dependencies)
+  }
+
+  const dependency = _.find(allDependencies, (dependency) => {
+    const rex = RegExp(`${branchPrefix}${groupRex}${dependency.name}-${dependency.version}`)
+    return rex.test(branch)
   })
+  return _.compact([dependency])
 }

lib/git-helpers.js

@@ -23,5 +23,42 @@ module.exports = {
     return (
       `${parsed[1]}/${parsed[2]}`
     )
+  },
+  hasLockfileCommit: function hasLockfileCommit (info) {
+    const defaultBranch = process.env.GK_LOCK_DEFAULT_BRANCH || 'master'
+    // CI clones are often shallow clones. Let’s make sure we have enough to work with
+    // https://stackoverflow.com/a/44036486/242298
+    // console.log(`git config --replace-all remote.origin.fetch +refs/heads/*:refs/remotes/origin/*`)
+    exec(`git config --replace-all remote.origin.fetch +refs/heads/*:refs/remotes/origin/*`)
+    // console.log(`git fetch`)
+    exec(`git fetch`)
+    // console.log(`git checkout master`)
+    // Sometimes weird things happen with Git, so let's make sure that we have a clean
+    // working tree before we go in!
+    exec(`git stash`)
+    exec(`git checkout ${defaultBranch}`)
+
+    const reset = () => exec(`git checkout -`)
+
+    try {
+      exec(`git log --oneline origin/${info.branchName}...${defaultBranch} | grep 'chore(package): update lockfile'`)
+    } catch (e) {
+      if (e.status === 1 &&
+        e.stdout.toString() === '' &&
+        e.stderr.toString() === '') { // grep didn’t find anything, and we are fine with that
+        reset()
+        return false // no commits
+      } else if (e.status > 1) { // git or grep errored
+        reset()
+        throw e
+      } else {
+        // git succeeded, grep failed to match anything, but no error occured, e.g.: no commit yet
+        reset()
+        return false
+      }
+    }
+    // git succeeded, grep found a match, we have a commit already
+    reset()
+    return true
   }
 }

lib/ignores.js

@@ -0,0 +1,10 @@
+const fs = require('fs')
+module.exports = function ignores () {
+  const ignoreFile = '.gitignore'
+  if (!fs.existsSync(ignoreFile)) {
+    return []
+  }
+
+  const ignoreContents = fs.readFileSync(ignoreFile).toString()
+  return (ignoreContents.length && ignoreContents.split('\n')) || []
+}

lib/update-lockfile.js

@@ -16,14 +16,10 @@ const yarnFlags = {
   'optionalDependencies': ' -O'
 }
 
-module.exports = function updateLockfile (dependency, options) {
+module.exports.updateLockfile = function updateLockfile (dependency, options) {
   if (!options.yarn && semver.lt(exec('npm --version').toString().trim(), '3.0.0')) {
     exec('npm shrinkwrap')
   } else {
-    // revert and unstage the changes done by greenkeeper
-    exec('git revert -n HEAD')
-    exec('git reset HEAD')
-
     if (options.yarn) {
       const flag = yarnFlags[dependency.type]
       const envArgs = process.env.GK_LOCK_YARN_OPTS ? ` ${process.env.GK_LOCK_YARN_OPTS.trim()}` : ''
@@ -40,29 +36,40 @@ module.exports = function updateLockfile (dependency, options) {
         exec('npm5 -v')
         npmBin = 'npm5'
       } catch (err) {}
-
       exec(`${npmBin} install${args}`)
     }
   }
+}
 
-  const commitEmail = process.env.GK_LOCK_COMMIT_EMAIL ? process.env.GK_LOCK_COMMIT_EMAIL.trim() : '[email protected]'
-  const commitName = process.env.GK_LOCK_COMMIT_NAME ? process.env.GK_LOCK_COMMIT_NAME.trim() : 'greenkeeperio-bot'
-  const shouldAmend = !_.includes([undefined, `0`, 'false', 'null', 'undefined'], process.env.GK_LOCK_COMMIT_AMEND)
-
+module.exports.stageLockfile = function stageLockfile () {
+  // make sure that we have changes to add
   if (exec('git status --porcelain').toString() === '') return
 
   // stage the updated lockfile
   exec('git add npm-shrinkwrap.json 2>/dev/null || true')
   exec('git add package-lock.json 2>/dev/null || true')
   exec('git add yarn.lock 2>/dev/null || true')
+}
+
+module.exports.commitLockfiles = function commitLockfiles () {
+  const commitEmail = process.env.GK_LOCK_COMMIT_EMAIL ? process.env.GK_LOCK_COMMIT_EMAIL.trim() : '[email protected]'
+  const commitName = process.env.GK_LOCK_COMMIT_NAME ? process.env.GK_LOCK_COMMIT_NAME.trim() : 'greenkeeperio-bot'
+  const shouldAmend = !_.includes([undefined, `0`, 'false', 'null', 'undefined'], process.env.GK_LOCK_COMMIT_AMEND)
 
   exec(`git config user.email "${commitEmail}"`)
   exec(`git config user.name "${commitName}"`)
 
   if (shouldAmend) {
     exec(`git commit --amend --author="${commitName} <${commitEmail}>" --no-edit`)
   } else {
-    const updateMessage = 'chore(package): update lockfile\n\nhttps://npm.im/greenkeeper-lockfile'
+    let lockfileWording
+    // either say "lockfile" or "lockfiles" depending on how many files are changed
+    if (exec('git status --porcelain').toString().split('\n').length > 2) {
+      lockfileWording = 'lockfiles'
+    } else {
+      lockfileWording = 'lockfile'
+    }
+    const updateMessage = `chore(package): update ${lockfileWording}\n\nhttps://npm.im/greenkeeper-lockfile`
     exec(`git commit -m "${updateMessage}"`)
   }
 }