Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove a bunch of 'skip GC threads' checks since they now have tasks #55160

Merged
merged 1 commit into from
Jul 24, 2024

Conversation

d-netto
Copy link
Member

@d-netto d-netto commented Jul 18, 2024

GC threads now have tasks (since #53815).

@d-netto d-netto added the GC Garbage collector label Jul 18, 2024
@d-netto d-netto requested a review from kpamnany July 18, 2024 02:40
@d-netto d-netto force-pushed the dcn-gc-threads-have-tasks branch 2 times, most recently from 8babca2 to 758b3ad Compare July 23, 2024 22:37
@d-netto d-netto force-pushed the dcn-gc-threads-have-tasks branch from 758b3ad to 57e33d5 Compare July 23, 2024 22:37
Copy link
Contributor

@kpamnany kpamnany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think CI tests either of these so we need to manually check. I suggest running one of the multithreaded GC benchmarks with the concurrent sweep thread enabled, and insert ccalls to jl_live_tasks() and jl_print_task_backtraces(1) somewhere in the middle.

@d-netto
Copy link
Member Author

d-netto commented Jul 23, 2024

Test

From GCBenchmarks:

include(joinpath("..", "..", "..", "util", "utils.jl"))

module BinaryTreeImmutable

# Adopted from
# https://benchmarksgame-team.pages.debian.net/benchmarksgame/description/binarytrees.html#binarytrees

using Base.Threads
using Printf

struct Node
    l::Union{Nothing, Node}
    r::Union{Nothing, Node}
end

function make(n::Int)
    return n === 0 ? Node(nothing, nothing) : Node(make(n-1), make(n-1))
end

function check(node::Node)
    return  1 + (node.l === nothing ? 0 : check(node.l) + check(node.r))
end

function dummy_task()
    while true
        tasks = @ccall jl_live_tasks()::Any
        @show tasks
        @ccall jl_print_task_backtraces(1::Cint)::Cvoid
        sleep(100)
    end
end 

function binary_trees(io, n::Int)
    @printf io "stretch tree of depth %jd\t check: %jd\n" n+1 check(make(n+1))
    Threads.@spawn dummy_task()
    long_tree = make(n)
    minDepth = 4
    resultSize = div((n - minDepth), 2) + 1
    results = Vector{String}(undef, resultSize)
    Threads.@threads for depth in minDepth:2:n
        c = 0
        niter = 1 << (n - depth + minDepth)
        for _ in 1:niter
            c += check(make(depth))
        end
        index = div((depth - minDepth),2) + 1
        results[index] = @sprintf "%jd\t trees of depth %jd\t check: %jd\n" niter depth c
    end

    for i in results
        write(io, i)
    end

    @printf io "long lived tree of depth %jd\t check: %jd\n" n check(long_tree)
end

end #module

using .BinaryTreeImmutable

@gctime BinaryTreeImmutable.binary_trees(devnull, 21)

Command

../julia-master/julia -t8 --gcthreads=8,1 --project=. benches/multithreaded/binary_tree/tree_immutable.jl

Output

[ Info: Setting GC memory pressure callback
tasks = Any[Task (runnable, started) @0x0000000108fb4010, Task (done) @0x000000010cfa4010, Task (runnable, started) @0x0000000108fb41a0, Task (done) @0x000000010cfb0010, Task (runnable, started) @0x0000000333970010, Task (done) @0x000000010cfac010, Task (done) @0x000000010cfa8010, Task (done) @0x000000010cf98010, Task (done) @0x000000010cf9c010, Task (done) @0x000000010cf8c010, Task (runnable, started) @0x000000010cf90010, Task (runnable, started) @0x000000010cf94010, Task (runnable, started) @0x000000010cf7c010, Task (runnable, started) @0x000000010cf78010, Task (runnable, started) @0x000000010cfa0010, Task (runnable, started) @0x000000010cf80010, Task (runnable, started) @0x000000010cf84010, Task (runnable, started) @0x000000010cf88010]
==== Thread 1 created 2 live tasks
     ---- Root task (0x108fb4010)
          (sticky: 1, started: 1, state: 0, tid: 1)
jl_start_fiber_swap at /Users/dnetto/RAI/julia-master/src/task.c:1435
ctx_switch at /Users/dnetto/RAI/julia-master/src/task.c:616
ijl_switch at /Users/dnetto/RAI/julia-master/src/task.c:649
try_yieldto at ./task.jl:1104
wait at ./task.jl:1178
#wait#731 at ./condition.jl:139
wait at ./condition.jl:134 [inlined]
_wait at ./task.jl:328
threading_run at ./threadingconstructs.jl:186
macro expansion at ./threadingconstructs.jl:208 [inlined]
binary_trees at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:40
macro expansion at /Users/dnetto/RAI/GCBenchmarks/util/utils.jl:37 [inlined]
top-level scope at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:36
jl_toplevel_eval_flex at /Users/dnetto/RAI/julia-master/src/toplevel.c:996
jl_toplevel_eval_flex at /Users/dnetto/RAI/julia-master/src/toplevel.c:945
ijl_toplevel_eval at /Users/dnetto/RAI/julia-master/src/toplevel.c:1016 [inlined]
ijl_toplevel_eval_in at /Users/dnetto/RAI/julia-master/src/toplevel.c:1058
eval at ./boot.jl:438 [inlined]
include_string at ./loading.jl:2618
_include at ./loading.jl:2678
include at ./Base.jl:580 [inlined]
exec_options at ./client.jl:324
_start at ./client.jl:532
jfptr__start_71580 at /Users/dnetto/RAI/julia-master/usr/lib/julia/sys.dylib (unknown line)
jl_apply at /Users/dnetto/RAI/julia-master/src/./julia.h:2183 [inlined]
true_main at /Users/dnetto/RAI/julia-master/src/jlapi.c:900
jl_repl_entrypoint at /Users/dnetto/RAI/julia-master/src/jlapi.c:1059
     ---- End root task
     ---- Task 1 (0x333d28330)
          (sticky: 0, started: 1, state: 0, tid: 1)
make at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:16
make at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:17
make at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:17
make at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:17
make at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:17
make at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:17
make at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:17
make at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:17
make at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:17
make at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:17
make at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:17 [inlined]
macro expansion at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:44 [inlined]
#28#threadsfor_fun#2 at ./threadingconstructs.jl:271
#28#threadsfor_fun at ./threadingconstructs.jl:238 [inlined]
#1 at ./threadingconstructs.jl:172
unknown function (ip: 0x11e60c053)
jl_apply at /Users/dnetto/RAI/julia-master/src/./julia.h:2183 [inlined]
start_task at /Users/dnetto/RAI/julia-master/src/task.c:1240
     ---- End task 1
==== End thread 1
==== Thread 2 created 2 live tasks
     ---- Root task (0x10cfa4010)
          (sticky: 1, started: 1, state: 1, tid: 2)
unknown function (ip: 0xffffcb8d5bc8bb7f)
     ---- End root task
     ---- Task 1 (0x108fb41a0)
          (sticky: 0, started: 1, state: 0, tid: 0)
jl_start_fiber_swap at /Users/dnetto/RAI/julia-master/src/task.c:1435
ctx_switch at /Users/dnetto/RAI/julia-master/src/task.c:616
ijl_switch at /Users/dnetto/RAI/julia-master/src/task.c:649
try_yieldto at ./task.jl:1104
wait at ./task.jl:1178
#wait#731 at ./condition.jl:139
wait at ./condition.jl:134 [inlined]
_trywait at ./asyncevent.jl:145
profile_printing_listener at ./Base.jl:600
#1199 at ./Base.jl:620
jfptr_YY.1199_74010 at /Users/dnetto/RAI/julia-master/usr/lib/julia/sys.dylib (unknown line)
jl_apply at /Users/dnetto/RAI/julia-master/src/./julia.h:2183 [inlined]
start_task at /Users/dnetto/RAI/julia-master/src/task.c:1240
     ---- End task 1
     ---- Task 2 (0x333d28b00)
          (sticky: 0, started: 1, state: 0, tid: 2)
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:20
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
macro expansion at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:44 [inlined]
#28#threadsfor_fun#2 at ./threadingconstructs.jl:271
#28#threadsfor_fun at ./threadingconstructs.jl:238 [inlined]
#1 at ./threadingconstructs.jl:172
unknown function (ip: 0x11e60c053)
jl_apply at /Users/dnetto/RAI/julia-master/src/./julia.h:2183 [inlined]
start_task at /Users/dnetto/RAI/julia-master/src/task.c:1240
     ---- End task 2
==== End thread 2
==== Thread 3 created 2 live tasks
     ---- Root task (0x10cfb0010)
          (sticky: 1, started: 1, state: 1, tid: 3)
unknown function (ip: 0xffffcb8d5bc8bb7f)
     ---- End root task
     ---- Task 1 (0x333970010)
          (sticky: 0, started: 1, state: 0, tid: 5)
jl_rec_backtrace at /Users/dnetto/RAI/julia-master/src/stackwalk.c:930 [inlined]
jlbacktracet at /Users/dnetto/RAI/julia-master/src/stackwalk.c:1198
jl_print_task_backtraces at /Users/dnetto/RAI/julia-master/src/stackwalk.c:1268
dummy_task at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:28
#1 at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:35
unknown function (ip: 0x11e5ac037)
jl_apply at /Users/dnetto/RAI/julia-master/src/./julia.h:2183 [inlined]
start_task at /Users/dnetto/RAI/julia-master/src/task.c:1240
     ---- End task 1
     ---- Task 2 (0x333d28970)
          (sticky: 0, started: 1, state: 0, tid: 3)
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:20
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
check at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:21
macro expansion at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:44 [inlined]
#28#threadsfor_fun#2 at ./threadingconstructs.jl:271
#28#threadsfor_fun at ./threadingconstructs.jl:238 [inlined]
#1 at ./threadingconstructs.jl:172
unknown function (ip: 0x11e60c053)
jl_apply at /Users/dnetto/RAI/julia-master/src/./julia.h:2183 [inlined]
start_task at /Users/dnetto/RAI/julia-master/src/task.c:1240
     ---- End task 2
==== End thread 3
==== Thread 4 created 1 live tasks
     ---- Root task (0x10cfac010)
          (sticky: 1, started: 1, state: 1, tid: 4)
unknown function (ip: 0xffffcb8d5bc8bb7f)
     ---- End root task
     ---- Task 1 (0x333d28010)
          (sticky: 0, started: 1, state: 0, tid: 4)
make at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:16
make at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:17
make at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:17
make at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:17 [inlined]
macro expansion at /Users/dnetto/RAI/GCBenchmarks/benches/multithreaded/binary_tree/tree_immutable.jl:44 [inlined]
#28#threadsfor_fun#2 at ./threadingconstructs.jl:271
#28#threadsfor_fun at ./threadingconstructs.jl:238 [inlined]
#1 at ./threadingconstructs.jl:172
unknown function (ip: 0x11e60c053)
jl_apply at /Users/dnetto/RAI/julia-master/src/./julia.h:2183 [inlined]
start_task at /Users/dnetto/RAI/julia-master/src/task.c:1240
     ---- End task 1
==== End thread 4
==== Thread 5 created 1 live tasks
     ---- Root task (0x10cfa8010)
          (sticky: 1, started: 1, state: 1, tid: 5)
unknown function (ip: 0xffffcb8d5bc8bb7f)
     ---- End root task
     ---- Task 1 (0x333d28650)
          (sticky: 0, started: 1, state: 1, tid: 0)
      no stack
     ---- End task 1
==== End thread 5
==== Thread 6 created 1 live tasks
     ---- Root task (0x10cf98010)
          (sticky: 1, started: 1, state: 1, tid: 6)
unknown function (ip: 0xffffcb8d5bc8bb7f)
     ---- End root task
     ---- Task 1 (0x333d287e0)
          (sticky: 0, started: 1, state: 0, tid: 6)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
     ---- End task 1
==== End thread 6
==== Thread 7 created 1 live tasks
     ---- Root task (0x10cf9c010)
          (sticky: 1, started: 1, state: 1, tid: 7)
unknown function (ip: 0xffffcb8d5bc8bb7f)
     ---- End root task
     ---- Task 1 (0x333d281a0)
          (sticky: 0, started: 1, state: 0, tid: 7)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
     ---- End task 1
==== End thread 7
==== Thread 8 created 1 live tasks
     ---- Root task (0x10cf8c010)
          (sticky: 1, started: 1, state: 1, tid: 8)
unknown function (ip: 0xffffcb8d5bc8bb7f)
     ---- End root task
     ---- Task 1 (0x333d284c0)
          (sticky: 0, started: 1, state: 0, tid: 8)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
     ---- End task 1
==== End thread 8
==== Skipping backtrace for parallel GC thread 9
==== Skipping backtrace for parallel GC thread 10
==== Skipping backtrace for parallel GC thread 11
==== Skipping backtrace for parallel GC thread 12
==== Skipping backtrace for parallel GC thread 13
==== Skipping backtrace for parallel GC thread 14
==== Skipping backtrace for parallel GC thread 15
==== Skipping backtrace for concurrent GC thread 16
==== Done
(value = nothing, times = 0x00000000b4b1e16a, gc_diff = Base.GC_Diff(9790045760, 420, 0, 305743707, 10, 491, 1247176168, 124, 14), gc_start = Base.GC_Num(3183760, 0, 0, 6401, 0, 5767481, 1372, 6070, 75422667, 371606560, 0x0000000002bc0000, 9, 2, 18007959, 110965516, 7167, 9792, 65792, 1363834, 1133208, 14601541, 60820127, 6499255057349791, 6499255602280625), gc_end = Base.GC_Num(12326368, 0, 0, 6821, 0, 311511188, 1382, 6561, 1322598835, 10152509712, 0x0000000002bc0000, 133, 16, 45131375, 433673164, 141208, 212895958, 287594545, 294417, 28500, 375804499, 946774001, 6499258346439333, 6499259014800375))

Copy link
Contributor

@kpamnany kpamnany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@d-netto d-netto added the merge me PR is reviewed. Merge when all tests are passing label Jul 24, 2024
@d-netto d-netto merged commit 0055747 into master Jul 24, 2024
7 of 8 checks passed
@d-netto d-netto deleted the dcn-gc-threads-have-tasks branch July 24, 2024 03:13
@d-netto d-netto removed the merge me PR is reviewed. Merge when all tests are passing label Jul 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
GC Garbage collector
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants