Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unroll String.Equals for constant input [0..16] length #64821

Closed
wants to merge 21 commits into from

Conversation

EgorBo
Copy link
Member

@EgorBo EgorBo commented Feb 4, 2022

This PR unrolls String.Equals when the 2nd argument is a constant string of [0..16] length (can be extended to e.g. 32 via AVX2 easily but needs some work with inliner's budget)

Example

bool Test1(string s) => s == "Hi"; // 2 chars
bool Test2(string s) => s == "Content-Length"; // 14 chars

New codegen:

; Method Tests:Test1(System.String):bool:this
G_M52821_IG01:              
G_M52821_IG02:              
       4885D2               test     rdx, rdx
       7416                 je       SHORT G_M52821_IG04
G_M52821_IG03:              
       48B80200000048006900 mov      rax, 0x69004800000002 ;; it contains 'H', 'i' and 4 bytes of Length
       48394208             cmp      qword ptr [rdx+8], rax
       0F94C0               sete     al
       0FB6C0               movzx    rax, al
       EB02                 jmp      SHORT G_M52821_IG05
G_M52821_IG04:              
       33C0                 xor      eax, eax
G_M52821_IG05:              
       C3                   ret      
; Total bytes of code: 30


; Method Tests:Test2(System.String):bool:this
G_M31606_IG01:              
       C5F877               vzeroupper 
G_M31606_IG02:              
       4885D2               test     rdx, rdx  ;; <-- is s null?
       7438                 je       SHORT G_M31606_IG04
G_M31606_IG03:              
       837A080E             cmp      dword ptr [rdx+8], 14 ;; <-- is s.Length == 14 ?
       7530                 jne      SHORT G_M51463_IG04
       C5FA6F4218           vmovdqu  xmm0, xmmword ptr [rdx+24]
       C5FA6F4A0C           vmovdqu  xmm1, xmmword ptr [rdx+12]
       C5F1EF0D30000000     vpxor    xmm1, xmm1, xmmword ptr [reloc @RWD00]
       C5F9EF0538000000     vpxor    xmm0, xmm0, xmmword ptr [reloc @RWD16]
       C5F1EBC0             vpor     xmm0, xmm1, xmm0
       C4E27917C0           vptest   xmm0, xmm0
       0F94C0               sete     al
       0FB6C0               movzx    rax, al
       EB02                 jmp      SHORT G_M51463_IG05
G_M31606_IG04:              
       33C0                 xor      eax, eax
G_M31606_IG05:              
       C3                   return

RWD00  	dq	0074006E006F0043h, 002D0074006E0065h ;; 'C' 'o' 'n' 't' 'e' 'n' 't' '-'
RWD16  	dq	0065004C002D0074h, 006800740067006Eh ;; 't' '-' 'L' 'e' 'n' 'g' 't' 'h'
; Total bytes of code: 60

@dotnet/jit-contrib @stephentoub @jkotas @GrabYourPitchforks

@ghost ghost assigned EgorBo Feb 4, 2022
@dotnet-issue-labeler dotnet-issue-labeler bot added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Feb 4, 2022
@ghost
Copy link

ghost commented Feb 4, 2022

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

I decided to push my old prototype for review 🙂
This PR does two things:

  1. Inliner now doesn't early give up on large methods with [Intrinsic] attribute on them - it's a sort of temp workaround until we implement "partial prescan" - e.g. read first 30 bytes of IL and try to find something motivational enough to continue prescanning, e.g. IsKnownConsant call. - but it doesn't mean it always inline such methods - they use the standard procedure, in this case the presence of IsKnownConsant + constant argument at the callsite helps
  2. It unrolls String.StartsWith as a showcase - we can now do whatever we want with it in plain C# 🙂 It worth noting that it doesn't hurt throughput much as JIT is able to skip unreachable blocks early in the importer.

We can unroll StartsWith further for any length via overlapping and via SIMD if needed e.g. https://gist.github.com/EgorBo/8a4e4cda14eac0e605dd7bac68c56314#file-span_startswith_benchmark-cs-L80-L89
(Also will add tests if it's ok)

Other potential candidates: Equals, EndsWith, OrdinalIgnoreCase mode. The Span overloads need more thinking..

Demo:

bool Test1(string s) => s.StartsWith("t", StringComparison.Ordinal);
bool Test2(string s) => s.StartsWith("te", StringComparison.Ordinal);
bool Test3(string s) => s.StartsWith("tes", StringComparison.Ordinal);
bool Test4(string s) => s.StartsWith("test", StringComparison.Ordinal);
bool Test4(string s, string var) => s.StartsWith(var, StringComparison.Ordinal);

Codegen diff: https://www.diffchecker.com/Y7yBNGt4

@dotnet/jit-contrib @stephentoub @jkotas @GrabYourPitchforks

Author: EgorBo
Assignees: EgorBo
Labels:

area-CodeGen-coreclr

Milestone: -

@omariom
Copy link
Contributor

omariom commented Feb 4, 2022

What does it do with this?

s.StartsWith("test5", StringComparison.Ordinal)

@EgorBo
Copy link
Member Author

EgorBo commented Feb 4, 2022

What does it do with this?

s.StartsWith("test5", StringComparison.Ordinal)

Current impl will use the defaul impl, but can be easily extended

src/coreclr/jit/inline.def Outdated Show resolved Hide resolved
@EgorBo
Copy link
Member Author

EgorBo commented Feb 5, 2022

Added Equals.

bool Test(string? s) => s == "Hi";

codegen:

; Method Foo2:Test(System.String):bool:this
G_M50453_IG01:              ;; offset=0000H
						;; bbWeight=1    PerfScore 0.00

G_M50453_IG02:              ;; offset=0000H
       4885D2               test     rdx, rdx
       7416                 je       SHORT G_M50453_IG04
						;; bbWeight=1    PerfScore 1.25

G_M50453_IG03:              ;; offset=0005H
       48B80200000048006900 mov      rax, 0x69004800000002
       48394208             cmp      qword ptr [rdx+8], rax
       0F94C0               sete     al
       0FB6C0               movzx    rax, al
       EB02                 jmp      SHORT G_M50453_IG05
						;; bbWeight=0.50 PerfScore 3.25

G_M50453_IG04:              ;; offset=001BH
       33C0                 xor      eax, eax
						;; bbWeight=0.50 PerfScore 0.12

G_M50453_IG05:              ;; offset=001DH
       C3                   ret      
						;; bbWeight=1    PerfScore 1.00
; Total bytes of code: 30

Length and firstChar are merged into a single cmp (the trick I learned from Levi in Discord)

PS: codegen is still suboptimal, but it's a completely different issue

@EgorBo EgorBo changed the title Unroll String.StartsWith for constant input and ordinal Unroll String.StartsWith and String.Equals for constant input Feb 5, 2022
@EgorBo
Copy link
Member Author

EgorBo commented Feb 5, 2022

Pushed a demo for vectorized unrolling (len >= 8 && len <= 16):

bool Test1(string s) => s == "ProxyHeader";

new codegen:

; Method Tests:Test1(System.String):bool:this
G_M52821_IG01:              ;; offset=0000H
       C5F877               vzeroupper 
G_M52821_IG02:              ;; offset=0003H
       4885D2               test     rdx, rdx
       7504                 jne      SHORT G_M52821_IG04
G_M52821_IG03:              ;; offset=0008H
       33C0                 xor      eax, eax
       EB32                 jmp      SHORT G_M52821_IG05
G_M52821_IG04:              ;; offset=000CH
       C5F910052C000000     vmovupd  xmm0, xmmword ptr [reloc @RWD00]
       C5FA6F4A0C           vmovdqu  xmm1, xmmword ptr [rdx+12]
       C5F1EF0D2F000000     vpxor    xmm1, xmm1, xmmword ptr [reloc @RWD16]
       8B4208               mov      eax, dword ptr [rdx+8]
       C5F9EF4402FC         vpxor    xmm0, xmm0, xmmword ptr [rdx+rax-4]
       C5F1EBC0             vpor     xmm0, xmm1, xmm0
       C4E27917C0           vptest   xmm0, xmm0
       0F94C0               sete     al
       0FB6C0               movzx    rax, al
G_M52821_IG05:              ;; offset=0039H
       C3                   ret      
RWD00  	dq	0065004800790078h, 0072006500640061h
RWD16  	dq	0078006F00720050h, 0061006500480079h
; Total bytes of code: 58

Unfortunately inliner goes out of time budget if I enable SIMD path

@AndyAyersMS
Copy link
Member

Unfortunately inliner goes out of time budget if I enable SIMD path

You can always up the limit, if it's close. say from 10x to12x would be ok. But 10x to 100x maybe not so much.

@EgorBo EgorBo changed the title Unroll String.StartsWith and String.Equals for constant input Unroll String.Equals for constant input [0..16] length Feb 5, 2022
@EgorBo
Copy link
Member Author

EgorBo commented Feb 5, 2022

Unfortunately inliner goes out of time budget if I enable SIMD path

You can always up the limit, if it's close. say from 10x to12x would be ok. But 10x to 100x maybe not so much.

Good point, thanks.

So overall I decided to only keep Equals and more like a showcase, the SIMD version is quite small and handles [9..16] inputs.
I updated the description with codegen examples. It's ready for review, I guess I only need to add more tests.
All inputs in [0..16] range are executed in less than 0.5ns

@jkotas
Copy link
Member

jkotas commented Feb 5, 2022

inliner's budget

Consider a method that does 100 string comparisons against constant strings. Are all these 100 string comparisons going to be optimized, without hitting inliner's budget?

@jkotas
Copy link
Member

jkotas commented Feb 5, 2022

Consider a method that does 100 string comparisons against constant strings

Also, it would be interesting to measure JIT time of a method like this (before/after).

@am11
Copy link
Member

am11 commented Feb 5, 2022

simple change, big codegen diff O_o

Indeed 😁

…startswith

# Conflicts:
#	src/libraries/System.Private.CoreLib/src/System/String.Comparison.cs
@EgorBo
Copy link
Member Author

EgorBo commented Feb 10, 2022

Consider a method that does 100 string comparisons against constant strings. Are all these 100 string comparisons going to be optimized, without hitting inliner's budget?

@jkotas So the budget is common for all the callees for a specific root and it gets eaten very quickly with string.Equals, e.g.:

static bool Foo(string s)
{
    if (s == "11111111" || 
        s == "22222222" || 
        s == "33333333" || 
        s == "44444444" || 
        s == "55555555")
    {
        return true;
    }
    return false;
}

codegen:

; Method UnrolledStringEquals:Foo(System.String):bool
G_M33664_IG01:              ;; offset=0000H
       56                   push     rsi
       4883EC20             sub      rsp, 32
       C5F877               vzeroupper 
       488BF1               mov      rsi, rcx
						;; bbWeight=1    PerfScore 2.50

G_M33664_IG02:              ;; offset=000BH
       4885F6               test     rsi, rsi
       7436                 je       SHORT G_M33664_IG04
						;; bbWeight=1    PerfScore 1.25

G_M33664_IG03:              ;; offset=0010H
       837E0808             cmp      dword ptr [rsi+8], 8
       7530                 jne      SHORT G_M33664_IG04
       C5FA6F460C           vmovdqu  xmm0, xmmword ptr [rsi+12]
       C5FA6F4E0C           vmovdqu  xmm1, xmmword ptr [rsi+12]
       C5F1EF0DF8000000     vpxor    xmm1, xmm1, xmmword ptr [reloc @RWD00]
       C5F9EF05F0000000     vpxor    xmm0, xmm0, xmmword ptr [reloc @RWD00]
       C5F1EBC0             vpor     xmm0, xmm1, xmm0
       C4E27917C0           vptest   xmm0, xmm0
       0F94C2               sete     dl
       0FB6D2               movzx    rdx, dl
       EB02                 jmp      SHORT G_M33664_IG05
						;; bbWeight=0.50 PerfScore 11.29

G_M33664_IG04:              ;; offset=0041H
       33D2                 xor      edx, edx
						;; bbWeight=0.50 PerfScore 0.12

G_M33664_IG05:              ;; offset=0043H
       85D2                 test     edx, edx
       0F85B4000000         jne      G_M33664_IG11
						;; bbWeight=1    PerfScore 1.25

G_M33664_IG06:              ;; offset=004BH
       4885F6               test     rsi, rsi
       7436                 je       SHORT G_M33664_IG07
       837E0808             cmp      dword ptr [rsi+8], 8
       7530                 jne      SHORT G_M33664_IG07
       C5FA6F460C           vmovdqu  xmm0, xmmword ptr [rsi+12]
       C5FA6F4E0C           vmovdqu  xmm1, xmmword ptr [rsi+12]
       C5F1EF0DC8000000     vpxor    xmm1, xmm1, xmmword ptr [reloc @RWD16]
       C5F9EF05C0000000     vpxor    xmm0, xmm0, xmmword ptr [reloc @RWD16]
       C5F1EBC0             vpor     xmm0, xmm1, xmm0
       C4E27917C0           vptest   xmm0, xmm0
       0F94C2               sete     dl
       0FB6D2               movzx    rdx, dl
       EB02                 jmp      SHORT G_M33664_IG08
						;; bbWeight=0.50 PerfScore 11.92

G_M33664_IG07:              ;; offset=0081H
       33D2                 xor      edx, edx
						;; bbWeight=0.50 PerfScore 0.12

G_M33664_IG08:              ;; offset=0083H
       85D2                 test     edx, edx
       7573                 jne      SHORT G_M33664_IG11
       4885F6               test     rsi, rsi
       7436                 je       SHORT G_M33664_IG09
       837E0808             cmp      dword ptr [rsi+8], 8
       7530                 jne      SHORT G_M33664_IG09
       C5FA6F460C           vmovdqu  xmm0, xmmword ptr [rsi+12]
       C5FA6F4E0C           vmovdqu  xmm1, xmmword ptr [rsi+12]
       C5F1EF0D9C000000     vpxor    xmm1, xmm1, xmmword ptr [reloc @RWD32]
       C5F9EF0594000000     vpxor    xmm0, xmm0, xmmword ptr [reloc @RWD32]
       C5F1EBC0             vpor     xmm0, xmm1, xmm0
       C4E27917C0           vptest   xmm0, xmm0
       0F94C2               sete     dl
       0FB6D2               movzx    rdx, dl
       EB02                 jmp      SHORT G_M33664_IG10
						;; bbWeight=0.50 PerfScore 12.54

G_M33664_IG09:              ;; offset=00BDH
       33D2                 xor      edx, edx
						;; bbWeight=0.50 PerfScore 0.12

G_M33664_IG10:              ;; offset=00BFH
       85D2                 test     edx, edx
       7532                 jne      SHORT G_M33664_IG11
       48BA805565D474010000 mov      rdx, 0x174D4655580      ; "44444444"
       488B12               mov      rdx, gword ptr [rdx]
       488BCE               mov      rcx, rsi
       E8503EFEFF           call     System.String:<Equals>g__EqualsUnrolled_9_to_16|38_1(System.String,System.String):bool
       85C0                 test     eax, eax
       7519                 jne      SHORT G_M33664_IG11
       48BA885565D474010000 mov      rdx, 0x174D4655588      ; "55555555"
       488B12               mov      rdx, gword ptr [rdx]
       488BCE               mov      rcx, rsi
       E8BF43FEFF           call     System.String:Equals(System.String,System.String):bool
       85C0                 test     eax, eax
       740B                 je       SHORT G_M33664_IG13
						;; bbWeight=0.50 PerfScore 5.38

G_M33664_IG11:              ;; offset=00F5H
       B801000000           mov      eax, 1
						;; bbWeight=0.50 PerfScore 0.12

G_M33664_IG12:              ;; offset=00FAH
       4883C420             add      rsp, 32
       5E                   pop      rsi
       C3                   ret      
						;; bbWeight=0.50 PerfScore 0.88

G_M33664_IG13:              ;; offset=0100H
       33C0                 xor      eax, eax
						;; bbWeight=0.50 PerfScore 0.12

G_M33664_IG14:              ;; offset=0102H
       4883C420             add      rsp, 32
       5E                   pop      rsi
       C3                   ret      
						;; bbWeight=0.50 PerfScore 0.88
RWD00  	dq	0031003100310031h, 0031003100310031h
RWD16  	dq	0032003200320032h, 0032003200320032h
RWD32  	dq	0033003300330033h, 0033003300330033h
; Total bytes of code: 264

so no budget for `s == "55555555"

Also, it would be interesting to measure JIT time of a method like this (before/after).

since it can't do 100 Equals I don't see anything terrible and in general I guess there are not so many, I also tested the test I added - no visible effects.

What is nice is that for non-unrollable cases jit ignores those calls as it is able to remove simple dead branches during import.

Any opinion on this? It feels great to be able to declare rules to unroll stuff in pure C# but yeah, several long equals can eat inliner's budget and we won't be able to inline other stuff in the current root.

@EgorBo
Copy link
Member Author

EgorBo commented Feb 10, 2022

hm... g__EqualsUnrolled_9_to_16 was not inlined and it's terrible if executed without it... 😢

@EgorBo
Copy link
Member Author

EgorBo commented Feb 10, 2022

So the only thing I can do here is:

  1. Remove budget limits for methods with IsKnownConstants(const arg)
  2. Always return false for IsKnownConstants if method is too big already (or has too many locals)

@EgorBo
Copy link
Member Author

EgorBo commented Feb 10, 2022

So it was an interesting challenge but I feel like it is not worth the trouble. Maybe it will be more relevant for Spans, especially for Spans of bytes (UTF8).

Also, as an alternative good solution similar to this we can use UnrollHelpers (similar to CastHelpers) and replace calls in JIT - it should be able to inline helper calls (it currently does not but it's a simple change) - we will avoid the budget problem and will be able to do it only for hot blocks for instance and still have "rules for unrolling" defined in C#.

@AndyAyersMS
Copy link
Member

How much do you lose in perf by not force-inlining the EqualsUnrolled_0_to_7 and EqualsUnrolled_9_to_16 calls?

@EgorBo
Copy link
Member Author

EgorBo commented Feb 10, 2022

How much do you lose in perf by not force-inlining the EqualsUnrolled_0_to_7 and EqualsUnrolled_9_to_16 calls?

it's pretty bad, e.g. this is what happens when the masks are not folded to constants:

static Vector128<ushort> Foo(string b)
{
    return 
        Sse2.Xor(
            Vector128.Create(
                b[0], b[1], b[2], b[3],
                b[4], b[5], b[6], b[7]),
            Vector128.Create(
                b[b.Length - 8], b[b.Length - 7],
                b[b.Length - 6], b[b.Length - 5],
                b[b.Length - 4], b[b.Length - 3],
                b[b.Length - 2], b[b.Length - 1]));
}
; Assembly listing for method XX:Foo(System.String):System.Runtime.Intrinsics.Vector128`1[UInt16]

G_M28844_IG01:              ;; offset=0000H
       55           push     ebp
       8BEC         mov      ebp, esp
       56           push     esi
       C5F877       vzeroupper 
						;; bbWeight=1    PerfScore 3.25
G_M28844_IG02:              ;; offset=0007H
       8B4204       mov      eax, dword ptr [edx+4]
       85C0         test     eax, eax
       0F8403010000 je       G_M28844_IG05
       0FB77208     movzx    esi, word  ptr [edx+8]
       C5F96EC6     vmovd    xmm0, esi
       83F801       cmp      eax, 1
       0F86F1000000 jbe      G_M28844_IG05
       C5F9C4420A01 vpinsrw  xmm0, xmm0, word  ptr [edx+10], 1
       83F802       cmp      eax, 2
       0F86E1000000 jbe      G_M28844_IG05
       C5F9C4420C02 vpinsrw  xmm0, xmm0, word  ptr [edx+12], 2
       83F803       cmp      eax, 3
       0F86D1000000 jbe      G_M28844_IG05
       C5F9C4420E03 vpinsrw  xmm0, xmm0, word  ptr [edx+14], 3
       83F804       cmp      eax, 4
       0F86C1000000 jbe      G_M28844_IG05
       C5F9C4421004 vpinsrw  xmm0, xmm0, word  ptr [edx+16], 4
       83F805       cmp      eax, 5
       0F86B1000000 jbe      G_M28844_IG05
       C5F9C4421205 vpinsrw  xmm0, xmm0, word  ptr [edx+18], 5
       83F806       cmp      eax, 6
       0F86A1000000 jbe      G_M28844_IG05
       C5F9C4421406 vpinsrw  xmm0, xmm0, word  ptr [edx+20], 6
       83F807       cmp      eax, 7
       0F8691000000 jbe      G_M28844_IG05
       C5F9C4421607 vpinsrw  xmm0, xmm0, word  ptr [edx+22], 7
       8D70F8       lea      esi, [eax-8]
       3BF0         cmp      esi, eax
       0F837F000000 jae      G_M28844_IG05
       0FB77442F8   movzx    esi, word  ptr [edx+2*eax-8]
       C5F96ECE     vmovd    xmm1, esi
       8D70F9       lea      esi, [eax-7]
       3BF0         cmp      esi, eax
       736E         jae      SHORT G_M28844_IG05
       C5F1C44C42FA01 vpinsrw  xmm1, xmm1, word  ptr [edx+2*eax-6], 1
       8D70FA       lea      esi, [eax-6]
       3BF0         cmp      esi, eax
       735F         jae      SHORT G_M28844_IG05
       C5F1C44C42FC02 vpinsrw  xmm1, xmm1, word  ptr [edx+2*eax-4], 2
       8D70FB       lea      esi, [eax-5]
       3BF0         cmp      esi, eax
       7350         jae      SHORT G_M28844_IG05
       C5F1C44C42FE03 vpinsrw  xmm1, xmm1, word  ptr [edx+2*eax-2], 3
       8D70FC       lea      esi, [eax-4]
       3BF0         cmp      esi, eax
       7341         jae      SHORT G_M28844_IG05
						;; bbWeight=1    PerfScore 78.75
G_M28844_IG03:              ;; offset=00C8H
       C5F1C40C4204 vpinsrw  xmm1, xmm1, word  ptr [edx+2*eax], 4
       8D70FD       lea      esi, [eax-3]
       3BF0         cmp      esi, eax
       7333         jae      SHORT G_M28844_IG05
       C5F1C44C420205 vpinsrw  xmm1, xmm1, word  ptr [edx+2*eax+2], 5
       8D70FE       lea      esi, [eax-2]
       3BF0         cmp      esi, eax
       7324         jae      SHORT G_M28844_IG05
       C5F1C44C420406 vpinsrw  xmm1, xmm1, word  ptr [edx+2*eax+4], 6
       8D70FF       lea      esi, [eax-1]
       3BF0         cmp      esi, eax
       7315         jae      SHORT G_M28844_IG05
       C5F1C44C420607 vpinsrw  xmm1, xmm1, word  ptr [edx+2*eax+6], 7
       C5F9EFC1     vpxor    xmm0, xmm0, xmm1
       C5F91101     vmovupd  xmmword ptr [ecx], xmm0
						;; bbWeight=1    PerfScore 27.58
G_M28844_IG04:              ;; offset=0100H
       5E           pop      esi
       5D           pop      ebp
       C3           ret      
						;; bbWeight=1    PerfScore 2.00
G_M28844_IG05:              ;; offset=0103H
       E809891520   call     CORINFO_HELP_RNGCHKFAIL
       CC           int3     
						;; bbWeight=0    PerfScore 0.00

; Total bytes of code 265, prolog size 7, PerfScore 139.88, instruction count 70, allocated bytes for code 283 (MethodHash=985a8f53) for method XX:Foo(System.String):System.Runtime.Intrinsics.Vector128`1[UInt16]
; ============================================================

@EgorBo
Copy link
Member Author

EgorBo commented Feb 10, 2022

At this point it sounds like it's actually simpler to do everything in JIT directly - I'll return with a prototype in JIT this weekend 🙂 Going to close this one, just to keep as an example of doing things in pure C# only.

in jit I'll be able to optimize corner special cases (e.g. 8 chars), StartsWith, no budget problems, no throughput problems, AVX2 path (for up to 32byte), access to block weights, Spans...

@EgorBo EgorBo closed this Feb 10, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Mar 12, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants