Migrate Buildkite CI from AWS to GKE agent queues

[mstifflin]

What changed?

• Update Buildkite pipeline yaml to work with the newly provisioned queues in Google Kubernetes Engine
• Use agent-stack-k8s v0.8.0 helm chart which has its own expected pipeline yaml syntax in order to successfully onboard.
• Install buildkite-agent in Dockerfile for use in the code coverage step
  • The mount that previously worked in AWS's VM based infra doesn't work in Kubernete's container based set up. Install the cli directly for simplicity.
• Increase timeout for a test that was flakier on GKE
• Leave the unit-test step on AWS while debugging timeout issues.

Why?

• Migrate from AWS > Google Cloud. Buildkite Enterprise recommends GKE (as opposed to GCP) as the way to have a queue with autoscaling compute.

How did you test it?

• https://buildkite.com/uberopensource/cadence-server/builds/18881

Potential risks

• CI builds will be broken or flaky
• Can be mitigated by a git revert

Release notes
n/a

Documentation Changes
n/a

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 62.56%. Comparing base (9524eca) to head (14ae582).

❗ Current head 14ae582 differs from pull request most recent head e90efb0. Consider uploading reports for the commit e90efb0 to get more accurate results

Additional details and impacted files

see 6 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9524eca...e90efb0. Read the comment docs.

[coveralls]

Pull Request Test Coverage Report for Build 018f1666-ddc2-4a01-895c-4add087713ec

Details

• 0 of 1 (0.0%) changed or added relevant line in 1 file are covered.
• 32 unchanged lines in 12 files lost coverage.
• Overall coverage increased (+0.02%) to 67.744%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
common/persistence/persistence-tests/historyV2PersistenceTest.go	0	1	0.0%

Files with Coverage Reduction	New Missed Lines	%
common/task/weighted_round_robin_task_scheduler.go	1	89.05%
service/history/task/transfer_standby_task_executor.go	2	86.83%
common/persistence/sql/sqlplugin/postgres/task.go	2	73.4%
common/persistence/sql/sqlplugin/postgres/db.go	2	80.0%
service/matching/db.go	2	73.23%
common/util.go	2	91.78%
service/matching/taskListManager.go	2	81.16%
common/log/tag/tags.go	3	50.46%
service/history/queue/timer_queue_processor_base.go	3	77.82%
common/task/fifo_task_scheduler.go	4	83.51%

Totals
Change from base Build 018f127b-cd0f-43c3-b47a-35b58b7623db:	0.02%
Covered Lines:	99367
Relevant Lines:	146680

---

💛 - Coveralls

[davidporter-id-au]

thank you for doing this

[taylanisikdemir]

if this job is failing on K8s for pull-request pipeline it will also probably fail in master pipeline.

[mstifflin]

I did set pipeline-master's unit test step back to AWS, but I realized I added an extra "make just build" command to it that wasn't there previously. Will update.