Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(kernel): remove slub_debug kernel param #157

Merged
merged 1 commit into from
Oct 14, 2018

Conversation

andrewrynhard
Copy link
Member

@andrewrynhard andrewrynhard commented Oct 14, 2018

With slub_debug=P the following occurred when spinning up an EC2 instance with an AMI built from the Dianemo docker container:

2018/10/14 15:07:33.111170 main.go:44: initializing mount points
[   31.798655] XFS (xvda2): Mounting V5 Filesystem
[   31.831006] XFS (xvda2): Metadata CRC error detected at xfs_agi_read_verify+0xc6/0xf0, xfs_agi block 0x2 
[   31.839450] XFS (xvda2): Unmount and run xfs_repair
[   31.844043] XFS (xvda2): First 128 bytes of corrupted metadata buffer:
[   31.849937] 00000000fe7a10f8: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[   31.857665] 00000000763e02ae: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[   31.864996] 00000000885bf2eb: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[   31.872928] 000000008abf1fb1: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[   31.881218] 00000000829bd485: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[   31.889018] 00000000238a97a2: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[   31.897222] 000000006582036b: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[   31.905296] 0000000076fa4f16: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[   31.913197] XFS (xvda2): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0x2 len 1 error 74
[   31.921894] XFS (xvda2): xfs_imap_lookup: xfs_ialloc_read_agi() returned error -117, agno 0
[   31.929543] XFS (xvda2): Failed to read root inode 0x60, error 117
2018/10/14 15:07:33.261386 main.go:28: recovered from: mount /root: structure needs cleaning
[   31.941627] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
[   31.941627] 
[   31.950250] CPU: 0 PID: 1479 Comm: init Tainted: G                T 4.18.13-dianemo #1
[   31.957325] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[   31.962916] Call Trace:
[   31.965740]  dump_stack+0x5c/0x7b
[   31.969228]  panic+0xdf/0x224
[   31.972427]  do_exit+0xb39/0xb40
[   31.975847]  ? __switch_to_asm+0x40/0x70
[   31.979797]  ? __switch_to_asm+0x34/0x70
[   31.983616]  ? __switch_to_asm+0x40/0x70
[   31.987463]  do_group_exit+0x34/0xb0
[   31.991068]  get_signal+0x280/0x5d0
[   31.994643]  do_signal+0x31/0x570
[   31.998515]  ? do_nanosleep+0xb7/0x190
[   32.006570]  exit_to_usermode_loop+0x5f/0xb0
[   32.015649]  do_syscall_64+0x233/0x301
[   32.019827]  ? __switch_to_asm+0x34/0x70
[   32.023645]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   32.028108] RIP: 0033:0x844a8d
[   32.031153] Code: Bad RIP value.
[   32.034416] RSP: 002b:00007f1cd1ffed80 EFLAGS: 00000202 ORIG_RAX: 0000000000000023
[   32.040990] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000844a8d
[   32.047085] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00007f1cd1ffed80
[   32.052881] RBP: 00007f1cd1ffed90 R08: 0000000000000000 R09: 0000000000000000
[   32.058460] R10: 001be51d00000000 R11: 0000000000000202 R12: 00007ffe40eff81e
[   32.063900] R13: 00007ffe40eff81f R14: 00007f1cd1fff700 R15: 0000000000000000
[   32.069291] Kernel Offset: 0x15400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

This PR removes the parameter for now until we can figure out the root cause. To reproduce the issue, use the autonomy/dianemo:5ccc909 image to build an AMI.

@andrewrynhard andrewrynhard added this to the v0.1.0 milestone Oct 14, 2018
@andrewrynhard andrewrynhard self-assigned this Oct 14, 2018
@andrewrynhard andrewrynhard merged commit bbc3097 into siderolabs:master Oct 14, 2018
@andrewrynhard andrewrynhard deleted the fix-ami branch October 14, 2018 18:17
@andrewrynhard andrewrynhard modified the milestones: v0.1.0-alpha.16, v0.1 Jun 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant