This repository has been archived by the owner on May 12, 2021. It is now read-only.
kata-runtime hung after container creation failure #1088
Labels
bug
Incorrect behaviour
Comments
|
I somehow got the reason why both kata-runtime and kubelet hung, will provide more information later |
|
@peter-wangxu - thanks for the initial level of detail -- I'm interested in your further assessment here. @sboeuf - heads up. |
|
@bergwolf - heads up. |
|
I experience this too. |
|
Based on the lock on the socket it looks like kata-proxy is not being cleaned up by the runtime. |
|
Not able to find the root cause: but find a reliable way to reproduce in K8S cluster: apiVersion: v1
kind: Pod
metadata:
name: kata-stress
labels:
app: myapp
runtime: kata
annotations:
io.kubernetes.cri.untrusted-workload: "true"
spec:
containers:
- name: myapp-container
image: docker.io/progrium/stress:latest
imagePullPolicy: IfNotPresent
command: ["stress"]
args: ["--cpu","2","--io","1","--vm","2","--vm-bytes","128M","--timeout","10s"]
resources:
limits:
cpu: 1
memory: 300M
requests:
cpu: 500m
memory: 100Mcreate pod in k8s kubectl apply -f kata.yamlafter 10 seconds, the pod completed with root 40290 40281 0 Jan03 ? 00:00:00 /usr/bin/kata-runtime --root /run/containerd/runc/k8s.io --log /run/containerd/io.containerd.runtime.v1.linux/k8s.io/17a15cb99d2b29a724f1669b616c5d799947782034f123f4dc4868f81340b020/log.json --log-format json create --bundle /run/containerd/io.containerd.runtime.v1.linux/k8s.io/17a15cb99d2b29a724f1669b616c5d799947782034f123f4dc4868f81340b020 --pid-file /run/containerd/io.containerd.runtime.v1.linux/k8s.io/17a15cb99d2b29a724f1669b616c5d799947782034f123f4dc4868f81340b020/init.pid 17a15cb99d2b29a724f1669b616c5d799947782034f123f4dc4868f81340b020
the kubelet hung because it's trying to fetch the containers status via kata-runtime state via containerd, which was also blocked by the sbs lock file Thanks |
|
I got the trace by using kill -ABRT of the kata-runtime create Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.871874846-08:00" level=error msg="--- contention:" arch=amd64 name=kata-runtime pid=4 0290 source=runtime
384 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.871980969-08:00" level=error msg="cycles/second=2194961408" arch=amd64 name=kata-runtime pid=40290 source=runtime
385 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872007845-08:00" level=error msg="goroutine 4 [running]:" arch=amd64 name=kata-runtime pid=40290 source=runtime
386 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872030984-08:00" level=error msg="runtime/pprof.writeGoroutineStacks(0x5555a4a6a3e0, 0xc42033ad20, 0xc4203fa000, 0x1000)" arch=amd64 name=kata-runtime pid=40290 source=runtime
387 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872149701-08:00" level=error msg="\t/usr/src/packages/BUILD/local/go/src/runtime/pprof/pprof.go:650 +0xa9" arch=amd64 name=kata-runtime pid=40290 source=runtime
388 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872191919-08:00" level=error msg="runtime/pprof.writeGoroutine(0x5555a4a6a3e0, 0xc42033ad20, 0x2, 0x0, 0x0)" arch=amd64 name=kata-runtime pid=40290 source=runtime
389 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872330363-08:00" level=error msg="\t/usr/src/packages/BUILD/local/go/src/runtime/pprof/pprof.go:639 +0x46" arch=amd64 name=kata-runtime pid=40290 source=runtime
390 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872360573-08:00" level=error msg="runtime/pprof.(*Profile).WriteTo(0x5555a4e457a0, 0x5555a4a6a3e0, 0xc42033ad20, 0x2, 0x0, 0x0)" arch=amd64 name=kata-runtime pid=40290 source=runtime
391 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872378637-08:00" level=error msg="\t/usr/src/packages/BUILD/local/go/src/runtime/pprof/pprof.go:310 +0x3e6" arch=amd64 name=kata-runtime pid=40290 source=runtime
392 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.87239525-08:00" level=error msg="github.com/kata-containers/runtime/pkg/signals.Backtrace()" arch=amd64 name=kata-runtime pid=40290 source=runtime
393 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872422833-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/pkg/signals/signals.go:73 +0xb7" arch=amd64 name=kata-runtime pid=40290 source=runtime
394 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872640489-08:00" level=error msg="github.com/kata-containers/runtime/pkg/signals.Die(0xc42000c040)" arch=amd64 name=kata-runtime pid=40290 source=runtime
394 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872640489-08:00" level=error msg="github.com/kata-containers/runtime/pkg/signals.Die(0xc42000c040)" arch=amd64 name=kata-runtime pid=40290 source=runtime
395 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872688216-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/pkg/signals/signals.go:119 +0x32" arch=amd64 name=kata-runtime pid=40290 source=runtime
396 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872729794-08:00" level=error msg="main.setupSignalHandler.func2(0xc42009c420, 0xc42000c040)" arch=amd64 name=kata-runtime pid=40290 source=runtime
397 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872760648-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/cli/main.go:212 +0x251" arch=amd64 name=kata-runtime pid=40290 source=runtime
398 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872793036-08:00" level=error msg="created by main.setupSignalHandler" arch=amd64 name=kata-runtime pid=40290 source=runtime
399 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872830528-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/cli/main.go:199 +0x152" arch=amd64 name=kata-runtime pid=40290 source=runtime
400 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872851798-08:00" level=error arch=amd64 name=kata-runtime pid=40290 source=runtime
401 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.87290111-08:00" level=error msg="goroutine 1 [select, locked to thread]:" arch=amd64 name=kata-runtime pid=40290 source=runtime
402 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872927065-08:00" level=error msg="github.com/kata-containers/runtime/vendor/github.com/intel/govmm/qemu.(*QMP).executeCommandWithResponse(0xc420201f40, 0x5555a4a754a0, 0xc4202a06f0, 0x5555a453d869, 0xa, 0xc4203a2840, 0x0, 0x0, 0x0, 0xc4203a 2870, ...)" aamd64 name=kata-runtime pid=40290 source=runtime
403 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872950948-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/vendor/github.com/intel/govmm/qemu/qmp.go:599 +0x2c5" arch=amd64 name=kata-runtime pid=40290 source=runtime
404 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.872969598-08:00" level=error msg="github.com/kata-containers/runtime/vendor/github.com/intel/govmm/qemu.(*QMP).executeCommand(0xc420201f40, 0x5555a4a754a0, 0xc4202a06f0, 0x5555a453d869, 0xa, 0xc4203a2840, 0xc4203a2870, 0x5555a4eb1dc8, 0x555 5a4a26860)" aamd64 name=kata-runtime pid=40290 source=runtime
405 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.873003145-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/vendor/github.com/intel/govmm/qemu/qmp.go:613 +0x93" arch=amd64 name=kata-runtime pid=40290 source=runtime
406 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.873081826-08:00" level=error msg="github.com/kata-containers/runtime/vendor/github.com/intel/govmm/qemu.(*QMP).ExecuteDeviceDel(0xc420201f40, 0x5555a4a754a0, 0xc4202a06f0, 0xc42032e760, 0x5, 0x5555a4eb1dc8, 0x5555a4a79480)" arch=amd64 name= kata-runtime 40290 source=runtime
407 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.873130558-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/vendor/github.com/intel/govmm/qemu/qmp.go:960 +0x19f" arch=amd64 name=kata-runtime pid=40290 source=runtime
408 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.873158662-08:00" level=error msg="github.com/kata-containers/runtime/virtcontainers.(*qemu).hotplugRemoveCPUs(0xc4202cc400, 0x1, 0x0, 0xc4202ab9d0, 0x5555a4446700)" arch=amd64 name=kata-runtime pid=40290 source=runtime
409 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.873194895-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/virtcontainers/qemu.go:1060 +0xce" arch=amd64 name=kata-runtime pid=40290 source=runtime
410 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.873363445-08:00" level=error msg="github.com/kata-containers/runtime/virtcontainers.(*qemu).hotplugCPUs(0xc4202cc400, 0x555500000001, 0x1, 0x5555a4eb1dc8, 0x5555a4a79480, 0x5555a4eb1dc8)" arch=amd64 name=kata-runtime pid=40290 source=runtim
411 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.873417426-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/virtcontainers/qemu.go:986 +0xac" arch=amd64 name=kata-runtime pid=40290 source=runtime
412 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.873442504-08:00" level=error msg="github.com/kata-containers/runtime/virtcontainers.(*qemu).hotplugDevice(0xc4202cc400, 0x5555a4921800, 0xc4200a7354, 0xa, 0x1, 0x5555a4a754a0, 0xc4203a2810, 0x7ffd41633ec2, 0x40)" arch=amd64 name=kata-runtim d=40290 source=runtime
413 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.873461105-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/virtcontainers/qemu.go:932 +0x2a7" arch=amd64 name=kata-runtime pid=40290 source=runtime
414 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.8734769-08:00" level=error msg="github.com/kata-containers/runtime/virtcontainers.(*qemu).hotplugRemoveDevice(0xc4202cc400, 0x5555a4921800, 0xc4200a7354, 0xa, 0x0, 0x0, 0x0, 0x0)" arch=amd64 name=kata-runtime pid=40290 source=runtime
415 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.873532185-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/virtcontainers/qemu.go:963 +0xce" arch=amd64 name=kata-runtime pid=40290 source=runtime
416 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.87357688-08:00" level=error msg="github.com/kata-containers/runtime/virtcontainers.(*Container).removeResources(0xc4201eaf00, 0x3, 0xc420040570)" arch=amd64 name=kata-runtime pid=40290 source=runtime
417 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.873660354-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/virtcontainers/container.go:1256 +0x296" arch=amd64 name=kata-runtime pid=40290 source=runtime
418 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.873697362-08:00" level=error msg="github.com/kata-containers/runtime/virtcontainers.(*Container).rollbackFailingContainerCreation(0xc4201eaf00)" arch=amd64 name=kata-runtime pid=40290 source=runtime
419 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.873905784-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/virtcontainers/container.go:661 +0x42" arch=amd64 name=kata-runtime pid=40290 source=runtime
420 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.873958802-08:00" level=error msg="github.com/kata-containers/runtime/virtcontainers.createContainer.func1(0xc4202ac340, 0xc4202ac338)" arch=amd64 name=kata-runtime pid=40290 source=runtime
421 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.874006958-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/virtcontainers/container.go:705 +0x47" arch=amd64 name=kata-runtime pid=40290 source=runtime
422 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.87405282-08:00" level=error msg="github.com/kata-containers/runtime/virtcontainers.createContainer(0xc420001980, 0x7ffd41633ec2, 0x40, 0xc420202780, 0x7d, 0x0, 0xc42023eea0, 0xa, 0xd, 0xc420222400, ...)" arch=amd64 name=kata-runtime pid=402 90 source=run
423 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.874090485-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/virtcontainers/container.go:721 +0x896" arch=amd64 name=kata-runtime pid=40290 source=runtime
424 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.874123696-08:00" level=error msg="github.com/kata-containers/runtime/virtcontainers.(*Sandbox).CreateContainer(0xc420001980, 0x7ffd41633ec2, 0x40, 0xc420202780, 0x7d, 0x0, 0xc42023eea0, 0xa, 0xd, 0xc420222400, ...)" arch=amd64 name=kata-run pid=40290 source=runtime
425 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.874164788-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/virtcontainers/sandbox.go:1276 +0x72" arch=amd64 name=kata-runtime pid=40290 source=runtime
426 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.874202116-08:00" level=error msg="github.com/kata-containers/runtime/virtcontainers.CreateContainer(0x5555a4a754a0, 0xc420263e90, 0xc42020c4c0, 0x40, 0x7ffd41633ec2, 0x40, 0xc420202780, 0x7d, 0x0, 0xc42023eea0, ...)" arch=amd64 name=kata-ru e pid=40290 source=runtime
427 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.874237197-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/virtcontainers/api.go:398 +0x1df" arch=amd64 name=kata-runtime pid=40290 source=runtime
428 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.874268619-08:00" level=error msg="github.com/kata-containers/runtime/virtcontainers.(*VCImpl).CreateContainer(0x5555a4e92820, 0x5555a4a754a0, 0xc4201e7e60, 0xc42020c4c0, 0x40, 0x7ffd41633ec2, 0x40, 0xc420202780, 0x7d, 0x0, ...)" arch=amd64 name=kata-runtime pid=40290 source=runtime
429 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.874300707-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/virtcontainers/implementation.go:91 +0x99" arch=amd64 name=kata-runtime pid=40290 source=runtime
430 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.874342977-08:00" level=error msg="main.createContainer(0x5555a4a754a0, 0xc4201e7e60, 0xc4200349e0, 0x9, 0x0, 0xc42000d7a0, 0x0, 0x0, 0xc420055800, 0xb, ...)" arch=amd64 name=kata-runtime pid=40290 source=runtime
431 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.874465668-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/cli/create.go:340 +0x45d" arch=amd64 name=kata-runtime pid=40290 source=runtime
432 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.874507078-08:00" level=error msg="main.create(0x5555a4a754a0, 0xc4201e7560, 0x7ffd41633ec2, 0x40, 0xc420202580, 0x76, 0x0, 0x0, 0x7ffd41633e42, 0x7f, ...)" arch=amd64 name=kata-runtime pid=40290 source=runtime
433 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.874555871-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/cli/create.go:172 +0x590" arch=amd64 name=kata-runtime pid=40290 source=runtime
434 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.874586682-08:00" level=error msg="main.glob..func1(0xc4201fe580, 0x100, 0xc4201fe580)" arch=amd64 name=kata-runtime pid=40290 source=runtime
435 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.874615911-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/cli/create.go:79 +0x405" arch=amd64 name=kata-runtime pid=40290 source=runtime
436 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.874718555-08:00" level=error msg="github.com/kata-containers/runtime/vendor/github.com/urfave/cli.HandleAction(0x5555a4935e00, 0x5555a4a65d68, 0xc4201fe580, 0xc420032400, 0x0)" arch=amd64 name=kata-runtime pid=40290 source=runtime
437 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.874932861-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/vendor/github.com/urfave/cli/app.go:490 +0xca" arch=amd64 name=kata-runtime pid=40290 source=runtime
438 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.87498331-08:00" level=error msg="github.com/kata-containers/runtime/vendor/github.com/urfave/cli.Command.Run(0x5555a45392cc, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5555a4543ba6, 0x12, 0x0, ...)" arch=amd64 name=kata-runtime pid=40290 source=runtim e
439 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.87512003-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/vendor/github.com/urfave/cli/command.go:210 +0xa38" arch=amd64 name=kata-runtime pid=40290 source=runtime
440 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875180959-08:00" level=error msg="github.com/kata-containers/runtime/vendor/github.com/urfave/cli.(*App).Run(0xc4201e2000, 0xc4200a00d0, 0xd, 0xd, 0x0, 0x0)" arch=amd64 name=kata-runtime pid=40290 source=runtime
441 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875211209-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/vendor/github.com/urfave/cli/app.go:255 +0x6a2" arch=amd64 name=kata-runtime pid=40290 source=runtime
442 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875257939-08:00" level=error msg="main.createRuntimeApp(0x5555a4a75420, 0xc4200a6010, 0xc4200a00d0, 0xd, 0xd, 0xc420073f78, 0xc420073f48)" arch=amd64 name=kata-runtime pid=40290 source=runtime
443 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875291677-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/cli/main.go:479 +0x271" arch=amd64 name=kata-runtime pid=40290 source=runtime
444 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875410592-08:00" level=error msg="main.createRuntime(0x5555a4a75420, 0xc4200a6010)" arch=amd64 name=kata-runtime pid=40290 source=runtime
445 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875451133-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/cli/main.go:522 +0x7c" arch=amd64 name=kata-runtime pid=40290 source=runtime
446 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875573506-08:00" level=error msg="main.main()" arch=amd64 name=kata-runtime pid=40290 source=runtime
447 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.8756008-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/cli/main.go:554 +0x95" arch=amd64 name=kata-runtime pid=40290 source=runtime
448 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875617369-08:00" level=error arch=amd64 name=kata-runtime pid=40290 source=runtime
449 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.87564533-08:00" level=error msg="goroutine 19 [syscall]:" arch=amd64 name=kata-runtime pid=40290 source=runtime
450 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875671599-08:00" level=error msg="os/signal.signal_recv(0x5555a4a70ba0)" arch=amd64 name=kata-runtime pid=40290 source=runtime
451 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875687036-08:00" level=error msg="\t/usr/src/packages/BUILD/local/go/src/runtime/sigqueue.go:139 +0xa8" arch=amd64 name=kata-runtime pid=40290 source=runtime
452 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875705436-08:00" level=error msg="os/signal.loop()" arch=amd64 name=kata-runtime pid=40290 source=runtime
453 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875724525-08:00" level=error msg="\t/usr/src/packages/BUILD/local/go/src/os/signal/signal_unix.go:22 +0x24" arch=amd64 name=kata-runtime pid=40290 source=runtime
454 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875739945-08:00" level=error msg="created by os/signal.init.0" arch=amd64 name=kata-runtime pid=40290 source=runtime
455 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875753832-08:00" level=error msg="\t/usr/src/packages/BUILD/local/go/src/os/signal/signal_unix.go:28 +0x43" arch=amd64 name=kata-runtime pid=40290 source=runtime
456 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875861182-08:00" level=error arch=amd64 name=kata-runtime pid=40290 source=runtime
457 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875875782-08:00" level=error msg="goroutine 20 [select, locked to thread]:" arch=amd64 name=kata-runtime pid=40290 source=runtime
458 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875890221-08:00" level=error msg="runtime.gopark(0x5555a4a66638, 0x0, 0x5555a4539ad0, 0x6, 0x18, 0x1)" arch=amd64 name=kata-runtime pid=40290 source=runtime
459 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875905313-08:00" level=error msg="\t/usr/src/packages/BUILD/local/go/src/runtime/proc.go:291 +0x120" arch=amd64 name=kata-runtime pid=40290 source=runtime
460 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.87591965-08:00" level=error msg="runtime.selectgo(0xc42004d750, 0xc42008a480)" arch=amd64 name=kata-runtime pid=40290 source=runtime
461 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875937017-08:00" level=error msg="\t/usr/src/packages/BUILD/local/go/src/runtime/select.go:392 +0xe56" arch=amd64 name=kata-runtime pid=40290 source=runtime
462 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875952576-08:00" level=error msg="runtime.ensureSigM.func1()" arch=amd64 name=kata-runtime pid=40290 source=runtime
463 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875966467-08:00" level=error msg="\t/usr/src/packages/BUILD/local/go/src/runtime/signal_unix.go:549 +0x1f6" arch=amd64 name=kata-runtime pid=40290 source=runtime
464 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875981067-08:00" level=error msg="runtime.goexit()" arch=amd64 name=kata-runtime pid=40290 source=runtime
465 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.875997599-08:00" level=error msg="\t/usr/src/packages/BUILD/local/go/src/runtime/asm_amd64.s:2361 +0x1" arch=amd64 name=kata-runtime pid=40290 source=runtime
466 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876012193-08:00" level=error arch=amd64 name=kata-runtime pid=40290 source=runtime
467 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876032696-08:00" level=error msg="goroutine 5 [select]:" arch=amd64 name=kata-runtime pid=40290 source=runtime
468 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876047846-08:00" level=error msg="github.com/kata-containers/runtime/vendor/github.com/intel/govmm/qemu.(*QMP).mainLoop(0xc420201f40)" arch=amd64 name=kata-runtime pid=40290 source=runtime
469 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876065984-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/vendor/github.com/intel/govmm/qemu/qmp.go:535 +0x49d" arch=amd64 name=kata-runtime pid=40290 source=runtime
470 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876095078-08:00" level=error msg="created by github.com/kata-containers/runtime/vendor/github.com/intel/govmm/qemu.startQMPLoop" arch=amd64 name=kata-runtime pid=40290 source=runtime
471 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876122755-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/vendor/github.com/intel/govmm/qemu/qmp.go:573 +0xfe" arch=amd64 name=kata-runtime pid=40290 source=runtime
472 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876209284-08:00" level=error arch=amd64 name=kata-runtime pid=40290 source=runtime
473 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.87622366-08:00" level=error msg="goroutine 21 [IO wait]:" arch=amd64 name=kata-runtime pid=40290 source=runtime
474 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876238421-08:00" level=error msg="internal/poll.runtime_pollWait(0x7fa6c150ce30, 0x72, 0xc42005fc90)" arch=amd64 name=kata-runtime pid=40290 source=runtime
475 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876253596-08:00" level=error msg="\t/usr/src/packages/BUILD/local/go/src/runtime/netpoll.go:173 +0x59" arch=amd64 name=kata-runtime pid=40290 source=runtime
476 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876272881-08:00" level=error msg="internal/poll.(*pollDesc).wait(0xc420217318, 0x72, 0xffffffffffffff00, 0x5555a4a6c8c0, 0x5555a4e58ae8)" arch=amd64 name=kata-runtime pid=40290 source=runtime
477 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876288713-08:00" level=error msg="\t/usr/src/packages/BUILD/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9d" arch=amd64 name=kata-runtime pid=40290 source=runtime
478 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.87630375-08:00" level=error msg="internal/poll.(*pollDesc).waitRead(0xc420217318, 0xc4201bb400, 0xb30, 0xb30)" arch=amd64 name=kata-runtime pid=40290 source=runtime
479 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876318935-08:00" level=error msg="\t/usr/src/packages/BUILD/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f" arch=amd64 name=kata-runtime pid=40290 source=runtime
480 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876339289-08:00" level=error msg="internal/poll.(*FD).Read(0xc420217300, 0xc4201bb4d0, 0xb30, 0xb30, 0x0, 0x0, 0x0)" arch=amd64 name=kata-runtime pid=40290 source=runtime
481 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876488455-08:00" level=error msg="\t/usr/src/packages/BUILD/local/go/src/internal/poll/fd_unix.go:157 +0x17f" arch=amd64 name=kata-runtime pid=40290 source=runtime
482 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876505123-08:00" level=error msg="net.(*netFD).Read(0xc420217300, 0xc4201bb4d0, 0xb30, 0xb30, 0x400, 0xc4200a73a0, 0xe)" arch=amd64 name=kata-runtime pid=40290 source=runtime
483 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876581818-08:00" level=error msg="\t/usr/src/packages/BUILD/local/go/src/net/fd_unix.go:202 +0x51" arch=amd64 name=kata-runtime pid=40290 source=runtime
484 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876661231-08:00" level=error msg="net.(*conn).Read(0xc42000f080, 0xc4201bb4d0, 0xb30, 0xb30, 0x0, 0x0, 0x0)" arch=amd64 name=kata-runtime pid=40290 source=runtime
485 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876678374-08:00" level=error msg="\t/usr/src/packages/BUILD/local/go/src/net/net.go:176 +0x6c" arch=amd64 name=kata-runtime pid=40290 source=runtime
486 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876702503-08:00" level=error msg="bufio.(*Scanner).Scan(0xc42005ff40, 0xc42005ff28)" arch=amd64 name=kata-runtime pid=40290 source=runtime
487 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876726626-08:00" level=error msg="\t/usr/src/packages/BUILD/local/go/src/bufio/scan.go:208 +0xc1" arch=amd64 name=kata-runtime pid=40290 source=runtime
488 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876747259-08:00" level=error msg="github.com/kata-containers/runtime/vendor/github.com/intel/govmm/qemu.(*QMP).readLoop(0xc420201f40, 0xc42008a540)" arch=amd64 name=kata-runtime pid=40290 source=runtime
489 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876819649-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/vendor/github.com/intel/govmm/qemu/qmp.go:245 +0xf1" arch=amd64 name=kata-runtime pid=40290 source=runtime
490 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876845764-08:00" level=error msg="created by github.com/kata-containers/runtime/vendor/github.com/intel/govmm/qemu.(*QMP).mainLoop" arch=amd64 name=kata-runtime pid=40290 source=runtime
491 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.876862254-08:00" level=error msg="\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/vendor/github.com/intel/govmm/qemu/qmp.go:495 +0xbc" arch=amd64 name=kata-runtime pid=40290 source=runtime
492 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877054668-08:00" level=error msg="heap profile: 1: 1048576 [1: 1048576] @ heap/1048576" arch=amd64 name=kata-runtime pid=40290 source=runtime
493 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877076458-08:00" level=error msg="1: 1048576 [1: 1048576] @ 0x5555a44b5b15 0x5555a44b5a46 0x5555a44b25e6 0x5555a44bbf47 0x5555a44bc252 0x5555a45315f1 0x5555a3e920f1" arch=amd64 name=kata-runtime pid=40290 source=runtime
494 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877099991-08:00" level=error msg="#\t0x5555a44b5b14\truntime/pprof.writeGoroutineStacks+0x44\t\t\t\t/usr/src/packages/BUILD/local/go/src/runtime/pprof/pprof.go:648" arch=amd64 name=kata-runtime pid=40290 source=runtime
495 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877121107-08:00" level=error msg="#\t0x5555a44b5a45\truntime/pprof.writeGoroutine+0x45\t\t\t\t/usr/src/packages/BUILD/local/go/src/runtime/pprof/pprof.go:639" arch=amd64 name=kata-runtime pid=40290 source=runtime
496 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877137106-08:00" level=error msg="#\t0x5555a44b25e5\truntime/pprof.(*Profile).WriteTo+0x3e5\t\t\t\t/usr/src/packages/BUILD/local/go/src/runtime/pprof/pprof.go:310" arch=amd64 name=kata-runtime pid=40290 source=runtime
497 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877247872-08:00" level=error msg="#\t0x5555a44bbf46\tgithub.com/kata-containers/runtime/pkg/signals.Backtrace+0xb6\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/pkg/signals/signals.go:73" arch=amd64 name=kata-runtime pi d=40290rce=runtime
498 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877271545-08:00" level=error msg="#\t0x5555a44bc251\tgithub.com/kata-containers/runtime/pkg/signals.Die+0x31\t\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/pkg/signals/signals.go:119" arch=amd64 name=kata-runtime pid=4 0290 so=runtime
499 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877298172-08:00" level=error msg="#\t0x5555a45315f0\tmain.setupSignalHandler.func2+0x250\t\t\t\t/usr/src/packages/BUILD/go/src/github.com/kata-containers/runtime/cli/main.go:212" arch=amd64 name=kata-runtime pid=40290 source=runtime
500 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877329744-08:00" level=error arch=amd64 name=kata-runtime pid=40290 source=runtime
501 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877345404-08:00" level=error arch=amd64 name=kata-runtime pid=40290 source=runtime
502 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877360138-08:00" level=error msg="# runtime.MemStats" arch=amd64 name=kata-runtime pid=40290 source=runtime
503 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877374272-08:00" level=error msg="# Alloc = 2035376" arch=amd64 name=kata-runtime pid=40290 source=runtime
504 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877389874-08:00" level=error msg="# TotalAlloc = 3469808" arch=amd64 name=kata-runtime pid=40290 source=runtime
505 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877456393-08:00" level=error msg="# Sys = 8919288" arch=amd64 name=kata-runtime pid=40290 source=runtime
506 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877478722-08:00" level=error msg="# Lookups = 140" arch=amd64 name=kata-runtime pid=40290 source=runtime
507 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877492895-08:00" level=error msg="# Mallocs = 21020" arch=amd64 name=kata-runtime pid=40290 source=runtime
508 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877506405-08:00" level=error msg="# Frees = 9602" arch=amd64 name=kata-runtime pid=40290 source=runtime
509 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877594495-08:00" level=error msg="# HeapAlloc = 2035376" arch=amd64 name=kata-runtime pid=40290 source=runtime
510 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.87761425-08:00" level=error msg="# HeapSys = 5734400" arch=amd64 name=kata-runtime pid=40290 source=runtime
511 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877682692-08:00" level=error msg="# HeapIdle = 2433024" arch=amd64 name=kata-runtime pid=40290 source=runtime
512 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877709103-08:00" level=error msg="# HeapInuse = 3301376" arch=amd64 name=kata-runtime pid=40290 source=runtime
513 Jan 04 04:49:57 ubuntu kata-runtime[40290]: time="2019-01-04T04:49:57.877722764-08:00" level=error msg="# HeapReleased = 0" arch=amd64 name=kata-runtime pid=40290 source=runtime
|
|
/cc @devimc - can you take a look? |
|
similar to: #1074 |
|
I can take a look |
|
@peter-wangxu I can't reproduce it $ kubectl get pods
NAME READY STATUS RESTARTS AGE
kata-stress 1/1 Running 3 2m19s
$ sudo kata-runtime list
ID PID STATUS BUNDLE CREATED OWNER
8294c5c89d5cf11e3f293e2ab1d98ed83d7f78300daf3852187dc1dd6b3d136b 3113 running /run/containers/storage/overlay-containers/8294c5c89d5cf11e3f293e2ab1d98ed83d7f78300daf3852187dc1dd6b3d136b/userdata 2019-01-07T18:00:57.748434185Z #0
da45c492e806d651bc0221a9e665db72c584ccd689e433ad80d517a19f7d258d 3157 running /run/containers/storage/overlay-containers/da45c492e806d651bc0221a9e665db72c584ccd689e433ad80d517a19f7d258d/userdata 2019-01-07T18:00:58.425220655Z #0
5c7c1796552f8ce8c78bdc58ce8a1f26e94cf2f84b6c97d327e27807e5f4680e 3204 running /run/containers/storage/overlay-containers/5c7c1796552f8ce8c78bdc58ce8a1f26e94cf2f84b6c97d327e27807e5f4680e/userdata 2019-01-07T18:00:59.379912218Z #0
1d6dd664e87c7d92a2a789b7e9a2ca41665b193fd82670671a3698289db152fe 3250 running /run/containers/storage/overlay-containers/1d6dd664e87c7d92a2a789b7e9a2ca41665b193fd82670671a3698289db152fe/userdata 2019-01-07T18:00:59.77104282Z #0
f68de771e12bb98011494d7ff10c762b3da96f222e36ae5f85ab50aef07361ae 6003 running /run/containers/storage/overlay-containers/f68de771e12bb98011494d7ff10c762b3da96f222e36ae5f85ab50aef07361ae/userdata 2019-01-07T18:04:36.425354385Z #0
5d3e1c3eb20f255b7ce8bd3b437013c7cff7d0540900f1fcbd43d2a3dfec1ac3 7361 stopped /run/containers/storage/overlay-containers/5d3e1c3eb20f255b7ce8bd3b437013c7cff7d0540900f1fcbd43d2a3dfec1ac3/userdata 2019-01-07T18:06:13.845279431Z #0
f041333bee0ff986a50594aba99aa3ddda672a64e67c8069003c52f6bf0131da 7774 running /run/containers/storage/overlay-containers/f041333bee0ff986a50594aba99aa3ddda672a64e67c8069003c52f6bf0131da/userdata 2019-01-07T18:06:47.811317602Z #0
...........
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kata-stress 0/1 Completed 7 13m
$ sudo kata-runtime list
ID PID STATUS BUNDLE CREATED OWNER
8294c5c89d5cf11e3f293e2ab1d98ed83d7f78300daf3852187dc1dd6b3d136b 3113 running /run/containers/storage/overlay-containers/8294c5c89d5cf11e3f293e2ab1d98ed83d7f78300daf3852187dc1dd6b3d136b/userdata 2019-01-07T18:00:57.748434185Z #0
da45c492e806d651bc0221a9e665db72c584ccd689e433ad80d517a19f7d258d 3157 running /run/containers/storage/overlay-containers/da45c492e806d651bc0221a9e665db72c584ccd689e433ad80d517a19f7d258d/userdata 2019-01-07T18:00:58.425220655Z #0
5c7c1796552f8ce8c78bdc58ce8a1f26e94cf2f84b6c97d327e27807e5f4680e 3204 running /run/containers/storage/overlay-containers/5c7c1796552f8ce8c78bdc58ce8a1f26e94cf2f84b6c97d327e27807e5f4680e/userdata 2019-01-07T18:00:59.379912218Z #0
1d6dd664e87c7d92a2a789b7e9a2ca41665b193fd82670671a3698289db152fe 3250 running /run/containers/storage/overlay-containers/1d6dd664e87c7d92a2a789b7e9a2ca41665b193fd82670671a3698289db152fe/userdata 2019-01-07T18:00:59.77104282Z #0
f68de771e12bb98011494d7ff10c762b3da96f222e36ae5f85ab50aef07361ae 6003 running /run/containers/storage/overlay-containers/f68de771e12bb98011494d7ff10c762b3da96f222e36ae5f85ab50aef07361ae/userdata 2019-01-07T18:04:36.425354385Z #0
4363609145f0192b8925a7d8d116703d943564e3155feb6d644afdd5e96ab1d4 13336 stopped /run/containers/storage/overlay-containers/4363609145f0192b8925a7d8d116703d943564e3155feb6d644afdd5e96ab1d4/userdata 2019-01-07T18:17:29.808114439Z #0might be something is broken in your environment? can you try with last release? |
|
ohh wait, I was using crio not containerd as runtime, now I can reproduce it 😅 |
|
to reproduce this issue I had to change between runtimes (crio and containerd),
I don't know if these are the right steps to reproduce this issue, but I wrote a patch to fix it diff --git a/virtcontainers/container.go b/virtcontainers/container.go
index 545700b..d73a6e4 100644
--- a/virtcontainers/container.go
+++ b/virtcontainers/container.go
@@ -927,19 +927,29 @@ func (c *Container) stop() error {
// get failed if the process hasn't exited.
c.sandbox.agent.waitProcess(c, c.id)
- if err := c.sandbox.agent.stopContainer(c.sandbox, *c); err != nil {
+ var err error
+ defer func() {
+ if e := c.setContainerState(types.StateStopped); e != nil {
+ c.Logger().WithError(e).Errorf("Could not change container state to stopped")
+ if err == nil {
+ err = e
+ }
+ }
+ }()
+
+ if err = c.sandbox.agent.stopContainer(c.sandbox, *c); err != nil {
return err
}
- if err := c.detachDevices(); err != nil {
+ if err = c.detachDevices(); err != nil {
return err
}
- if err := c.removeDrive(); err != nil {
+ if err = c.removeDrive(); err != nil {
return err
}
- return c.setContainerState(types.StateStopped)
+ return err
}
func (c *Container) enter(cmd types.Cmd) (*Process, error) {
diff --git a/virtcontainers/kata_agent.go b/virtcontainers/kata_agent.go
index 64b40a7..f7ec6cd 100644
--- a/virtcontainers/kata_agent.go
+++ b/virtcontainers/kata_agent.go
@@ -1568,6 +1568,17 @@ func (k *kataAgent) sendReq(request interface{}) (interface{}, error) {
span.SetTag("request", request)
defer span.Finish()
+ if k.state.ProxyPid > 0 {
+ proxyProcess, err := os.FindProcess(k.state.ProxyPid)
+ if err != nil {
+ return nil, fmt.Errorf("Could not find a proxy with process id %d: %v", k.state.ProxyPid, err)
+ }
+
+ if err = proxyProcess.Signal(syscall.Signal(0)); err != nil {
+ return nil, fmt.Errorf("Proxy is not running: %v", err)
+ }
+ }
+
if err := k.connect(); err != nil {
return nil, err
}@peter-wangxu @ccollicutt can you confirm if above patch works for you? |
|
@peter-wangxu @ccollicutt (nudge) 😄 |
devimc
pushed a commit
to devimc/kata-runtime
that referenced
this issue
Feb 18, 2019
container is killed forcely, container's state MUST change its state to stop immediately to avoid leaving it in a bad state. fixes kata-containers#1088 Signed-off-by: Julio Montes <[email protected]>
devimc
pushed a commit
to devimc/kata-runtime
that referenced
this issue
Feb 18, 2019
container is killed by force, container's state MUST change its state to stop immediately to avoid leaving it in a bad state. fixes kata-containers#1088 Signed-off-by: Julio Montes <[email protected]>
devimc
pushed a commit
to devimc/kata-runtime
that referenced
this issue
Feb 18, 2019
container is killed by force, container's state MUST change its state to stop immediately to avoid leaving it in a bad state. fixes kata-containers#1088 Signed-off-by: Julio Montes <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Description of problem
I am using the kata-runtime with containerd and kubernetes:
after a failure in pod creation, the kata-runtime hung after pod creation failure due to invalid command in pod spec
peter@ubuntu:~/k8s$ sudo kata-runtime list ^Cthe
kata-runtime listwill never returnExpected result
The kata-runtime should remove any left-over if failed and the should not block the further pod creation
Actual result
further more the kata-runtime never free the file lock
the
flockwill never return as the file was locked by the kata-runtime process(replace this text with the output of the
kata-collect-data.shscript, afteryou have reviewed its content to ensure it does not contain any private
information).
peter@ubuntu:~/k8s$ sudo kata-collect-data.sh
Meta details
Running
kata-collect-data.shversion1.4.1 (commit 45ea48e)at2019-01-04.03:24:20.486242806-0800.Runtime is
/usr/bin/kata-runtime.kata-envOutput of "
/usr/bin/kata-runtime kata-env":Runtime config files
Runtime default config files
Runtime config file contents
Config file
/etc/kata-containers/configuration.tomlnot foundOutput of "
cat "/usr/share/defaults/kata-containers/configuration.toml"":KSM throttler
version
Output of "
/usr/libexec/kata-ksm-throttler/kata-ksm-throttler --version":systemd service
Image details
Initrd details
No initrd
Logfiles
Runtime logs
Recent runtime problems found in system journal:
Proxy logs
Recent proxy problems found in system journal:
Shim logs
No recent shim problems found in system journal.
Throttler logs
No recent throttler problems found in system journal.
Container manager details
Have
dockerDocker
Output of "
docker version":Output of "
docker info":Output of "
systemctl show docker":Have
kubectlKubernetes
Output of "
kubectl version":Output of "
kubectl config view":Output of "
systemctl show kubelet":No
crioPackages
Have
dpkgOutput of "
dpkg -l|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"":No
rpmThe text was updated successfully, but these errors were encountered: