Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Issue]: Cannot register Static Global Var on inline variable #3463

Open
tpadioleau opened this issue Apr 29, 2024 · 11 comments
Open

[Issue]: Cannot register Static Global Var on inline variable #3463

tpadioleau opened this issue Apr 29, 2024 · 11 comments
Assignees
@kjayapra-amd
Labels
Under Investigation

Comments

@tpadioleau
Copy link

tpadioleau commented Apr 29, 2024 •
edited
Loading

Problem Description

If I call hipMemcpyToSymbol/hipMemcpyFromSymbol on the same inline and __constant__ variable from different source files I get the error Cannot register Static Global Var, error:13.

Find attached inline-variable.zip a small reproducer. It can be compiled and executed with the following commands.

cmake -DCMAKE_CXX_COMPILER=hipcc -DCMAKE_CXX_STANDARD=17 -DGPU_TARGETS=gfx90a -B build
cmake --build build
./build/inline-variable

I only have an easy access to rocm 6.0.0, I could not try the latest rocm 6.1.0 version.

Operating System

Red Hat Enterprise Linux version 8.8

CPU

AMD EPYC 9654 96-Core

GPU

AMD Instinct MI250X

ROCm Version

ROCm 6.0.0

ROCm Component

HIP, HIPCC, rdc

Steps to Reproduce

No response

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

ROCk module is loaded

HSA System Attributes

Runtime Version: 1.1
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model: LARGE
System Endianness: LITTLE
Mwaitx: DISABLED
DMAbuf Support: YES

==========
HSA Agents

Agent 1

Name: AMD EPYC 7A53 64-Core Processor
Uuid: CPU-XX
Marketing Name: AMD EPYC 7A53 64-Core Processor
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3541
BDFID: 0
Internal Node ID: 0
Compute Unit: 32
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 65577568(0x3e8a260) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 65577568(0x3e8a260) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 65577568(0x3e8a260) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:

Agent 2

Name: AMD EPYC 7A53 64-Core Processor
Uuid: CPU-XX
Marketing Name: AMD EPYC 7A53 64-Core Processor
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 1
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3541
BDFID: 0
Internal Node ID: 1
Compute Unit: 32
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 66050988(0x3efdbac) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 66050988(0x3efdbac) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 66050988(0x3efdbac) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:

Agent 3

Name: AMD EPYC 7A53 64-Core Processor
Uuid: CPU-XX
Marketing Name: AMD EPYC 7A53 64-Core Processor
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 2
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3541
BDFID: 0
Internal Node ID: 2
Compute Unit: 32
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 66050988(0x3efdbac) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 66050988(0x3efdbac) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 66050988(0x3efdbac) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:

Agent 4

Name: AMD EPYC 7A53 64-Core Processor
Uuid: CPU-XX
Marketing Name: AMD EPYC 7A53 64-Core Processor
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 3
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3541
BDFID: 0
Internal Node ID: 3
Compute Unit: 32
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 64940256(0x3dee8e0) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 64940256(0x3dee8e0) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 64940256(0x3dee8e0) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:

Agent 5

Name: gfx90a
Uuid: GPU-aeefa4db4734109e
Marketing Name: AMD Instinct MI250X
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 4
Device Type: GPU
Cache Info:
L1: 16(0x10) KB
L2: 8192(0x2000) KB
Chip ID: 29704(0x7408)
ASIC Revision: 1(0x1)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 1700
BDFID: 49408
Internal Node ID: 4
Compute Unit: 110
SIMDs per CU: 4
Shader Engines: 8
Shader Arrs. per Eng.: 1
WatchPts on Addr. Ranges:4
Features: KERNEL_DISPATCH
Fast F16 Operation: TRUE
Wavefront Size: 64(0x40)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 2048(0x800)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 78
SDMA engine uCode:: 8
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 67092480(0x3ffc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS:
Size: 67092480(0x3ffc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 67092480(0x3ffc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 4
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32
*** Done ***

Additional Information

No response
@yxsamliu yxsamliu mentioned this issue May 1, 2024
Closed
@yxsamliu
Copy link
Contributor

yxsamliu commented May 1, 2024

At first I thought it was clang issue, but further investigation showed that clang generated correct code.

The issue seems to be a HIP runtime bug caused by https://github.com/ROCm/clr/blob/feaef7f54eb2d517b00a63f157780a84c7ecda31/hipamd/src/hip_code_object.cpp#L874

Basically, runtime expects each shadow variable to be only registered once, otherwise it returns hipErrorInvalidSymbol.

However, this is not true when different translation units defines an inline variable with the same name. These variables are merged by linker as one variable and they have the same address. Therefore runtime could encounter registering the same shadow variable multiple times. As long as they map to the same device variable name, it should be allowed.

@tpadioleau
Copy link
Author

Thank you very much for the quick investigation!

Basically, runtime expects each shadow variable to be only registered once, otherwise it returns hipErrorInvalidSymbol.

To understand correctly, there is a call to this registerStatGlobalVar function in each translation unit right ?

Can you tell what HIP versions will get the fix ?

@kjayapra-amd kjayapra-amd self-assigned this May 2, 2024
@yxsamliu
Copy link
Contributor

yxsamliu commented May 2, 2024 •
edited
Loading

Thank you very much for the quick investigation!

Basically, runtime expects each shadow variable to be only registered once, otherwise it returns hipErrorInvalidSymbol.

To understand correctly, there is a call to this registerStatGlobalVar function in each translation unit right ?

In each translation unit, registerStatGlobalVar is called for each device variable defined by that translation unit.

Can you tell what HIP versions will get the fix ?

I will leave your question for HIP runtime team. @mangupta

@kjayapra-amd
Copy link
Contributor

kjayapra-amd commented May 2, 2024 •
edited
Loading

Fix is in review (#460091). Will update here once I know the landing version.

@tpadioleau
Copy link
Author

Has the fix been accepted ? If so, what versions have this fix ?

@tpadioleau
Copy link
Author

Any update ?

@schung-amd
Copy link

Hi @tpadioleau, sorry for the delayed response. This fix isn't in a release yet as far as I can tell, but I can keep tabs on this and update you once the fix lands in a future release. Your reproducer zip file seems to be broken; can you reupload it so I can verify?

@tpadioleau
Copy link
Author

@schung-amd Thanks for the update, what do you mean by broken ? You could download it but not open it ? On my side it seems to work fine.

@schung-amd
Copy link

schung-amd commented Oct 11, 2024 •
edited
Loading

Never mind, I was passing the wrong options to tar, the file is fine. Can confirm that this is not fixed as of ROCm 6.2.2, I'll update you when the fix lands in a release.

@erayinanc
Copy link

Hi, I am also blocked by this issue. Tried ROCm 6.2.4 without any luck. Is a fix expected in ROCm 6.3?

@schung-amd
Copy link

Sorry you're blocked by this issue. I can't make any promises, but we're looking into getting this fix into the next major release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kjayapra-amd kjayapra-amd

Labels
Under Investigation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@yxsamliu @erayinanc @tpadioleau @kjayapra-amd @schung-amd