Version selection refactor, logging, schema previews, and more

[christophermancini]

The initial scope of this PR was to not use CommittedDate for filtering PRs that will not be included as new versions. This has clearly been identified as a source for missing PRs in Issue #26. This modified approach looks at the PR UpdatedAt field instead. This field is updated every time there is a new commit and shifts the filtering from the code to the GraphQL engine, improving performance even further.

How it works, is that the version information pushed back to concourse contains the PR #, the head ref sha, and the PR UpdatedAt (UpdatedDate). Every resource check operation will take the updated value from the most recent version in Concourse to filter the list of PRs queried. We end up with only PRs that are open and have been updated since the last time we checked. We also use the updated value from Concourse to filter the TimelineItems connection, only receiving events & objects attached to the PR since we last checked. This allows us to expand our methods of filtering down the PR versions we return to Concourse.

You can read more about the filtering options in the changes to the README of this PR.

Other noteworthy changes include:

• added support for some schema previews
• added support to trigger a build via a PR comment containing [build ci | ci build]
• updated the files API request to use the V4 GraphQL interface instead of V3 Rest
• added logging to /tmp/github-pr-resource.log on the containers which can be accessed via hijacking the check, get, or put steps for a pipeline
• updated path filtering to use a Glob library instead of filepath.Match which doesn't adhere to glob standards and is not deterministic (e.g. acts differently on different operating systems)
• decoupled internal PullRequest struct from the GitHub PullRequestObject
• added environment variable interpolation to the context parameter on status put

We used a GraphQL query like the following to experiment and come up with this solution:

query ($c: String, $q: String!, $s: DateTime!) {
  search(query: $q, type: ISSUE, first: 100, after: $c) {
    edges {
      node {
        ... on PullRequest {
          id
          number
          title
          url
          baseRefName
          headRefName
          isCrossRepository
          createdAt
          updatedAt
          headRef {
            id
            name
            target {
              __typename
              ... on Commit {
                oid
                abbreviatedOid
                authoredDate
                committedDate
                pushedDate
                message
                author {
                  date
                  email
                  name
                  user {
                    login
                  }
                }
              }
            }
          }
          files(first: 100) {
            edges {
              cursor
              node {
                path
              }
            }
          }
          repository {
            url
          }
          timelineItems(last: 100, since: $s) {
            edges {
              cursor
              node {
                __typename
                ... on BaseRefChangedEvent {
                  id
                  createdAt
                }
                ... on HeadRefForcePushedEvent {
                  id
                  createdAt
                }
                ... on ReopenedEvent {
                  id
                  createdAt
                }
                ... on IssueComment {
                  id
                  createdAt
                  bodyText
                }
                ... on PullRequestCommit {
                  id
                  commit {
                    oid
                    abbreviatedOid
                    authoredDate
                    committedDate
                    pushedDate
                    message
                    author {
                      date
                      email
                      name
                      user {
                        login
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
    pageInfo {
      endCursor
      hasNextPage
    }
  }
}

Sample variables:

{"c":null,"q":"is:pr is:open repo:digitalocean/concourse updated:\u003e2019-07-22T17:18:11-04:00","s":"2019-07-22T17:18:11-04:00"}

[aliculPix4D]

Hi, can you please give us some more information when are you planning to merge this PR or the current status? Thank you.

[christophermancini]

Hi @aliculPix4D, thanks for reaching out. I am getting close to having something that is worth merging. Just need to do a bit more testing and polish it up. Then I will squash, have some other folks test it, then merge to our fork. We will give it a little while running internally (maybe two weeks) before submitting it upstream in a pull request.

[bhcleek]

Will this end up triggering builds of a PR every time there's any action on the PR (e.g. comments, reviews, title changes, etc.)?

[christophermancini]

@bhcleek nah, I recently added a section to the README covering how the search and filtering works, you can see it here:
https://github.com/digitalocean/github-pr-resource/pull/1/files#diff-04c6e90faac2675aa89e2176d2eec7d8R55

[nelsam]

Rather than include this list, can we just link to some godoc? This type of documentation is a prime example of "a lie waiting to happen" - it's too far from the code to expect anyone to remember to update it when they change the code.

[christophermancini]

I agree, however, I wanted it documented on the README for now because all of this work is in a feature branch and I am not sure it will ever get merged back upstream. Once we merge it to the master branch, I will replace it to a link to GoDoc.

[nelsam]

You know else statements are bad go 😜

This is another case where the var pulls []pullrequest.PullRequest followed by assignment inside branching logic suggests that what you really want is a function call:

func findPulls(since time.Time) ([]pullrequest.PullRequest, error) {
    if since.IsZero() {
        return manager.GetLatestOpenPullRequest()
    }
    return manager.ListOpenPullRequests(since)
}

func Check(request CheckRequest, manager Github) (CheckResponse, error) {
    var response CheckResponse
    pulls, err := findPulls(request.Version.UpdatedDate)

[christophermancini]

You are right, when I wrote it and looked at it, I could hear you in the back of my head... Thanks for the sample fix. Will update it. :)

[bhcleek]

@christophermancini this PR is huge. Are there particular concerns that you'd like to me focus on?

[christophermancini]

@bhcleek The logic for the Check operation.