Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scx_layered: Restrict preemption to layer cpumask #681

Merged
merged 2 commits into from
Sep 24, 2024

Conversation

hodgesds
Copy link
Contributor

When preempting restrict preemption to the current layer cpumask. This may reduce the amount of preemption, but cause better cache locality of preempted tasks.

When preempting restrict preemption to the current layer cpumask. This
may reduce the amount of preemption, but cause better cache locality
of preempted tasks.

Signed-off-by: Daniel Hodges <[email protected]>
/*
* First try preempting in the local LLC
*/
bpf_for(idx, 0, cachec->nr_cpus) {
s32 preempt_cpu = bpf_cpumask_any_distribute(cast_mask(topo_cpus));
trace("PREEMPT attempt on cpu %d from cpu %d",
preempt_cpu, bpf_get_smp_processor_id());
if (preempt_cpu > cachec->nr_cpus)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

drive by fix, this check is wrong and not needed

@likewhatevs
Copy link
Contributor

lgtm, a comment explaining the mask manipulation would be nice though

@hodgesds
Copy link
Contributor Author

Tested with stress-ng and I can see most no X-NUMA preemptions with a config to keep the stress-ng tasks on a single node:

###### Tue, 24 Sep 2024 07:16:56 -0700 ######
tot=  11971 local=44.30 open_idle= 0.00 affn_viol= 0.78 proc=6ms
busy= 15.6 util= 1114.5 load=   3867.2 fallback_cpu=  2
excl_coll=10.02 excl_preempt=0.13 excl_idle=0.11 excl_wakeup=0.00
  hodgesd  : util/frac=    4.4/  0.4 load/frac=      5.5/  0.1 tasks=   829
             tot=   4366 local=94.46 wake/exp/reenq= 5.52/ 0.02/ 0.00
             xlayer_wake= 4.63 xlayer_rewake= 0.53
             keep/max/busy= 0.89/ 0.00/ 0.09 kick= 0.00 yield/ign= 0.05/    0 
             open_idle= 0.00 mig= 7.47 xnuma_mig= 0.53 xllc_mig= 0.53 affn_viol= 0.00
             preempt/first/xllc/xnuma/idle/fail= 0.00/ 0.00/ 0.00/ 0.00/ 0.00/ 0.00 min_exec= 0.00/   0.00ms
             cpus=  2 [  2,  2] 00001000 00100000 00000000
             excl_coll= 0.00 excl_preempt= 0.00
  normal   : util/frac=  141.4/ 12.7 load/frac=   2861.7/ 74.0 tasks=  4549
             tot=   7589 local=15.54 wake/exp/reenq=83.70/ 0.76/ 0.00
             xlayer_wake= 6.26 xlayer_rewake= 1.25
             keep/max/busy=16.06/ 0.00/ 0.00 kick= 0.58 yield/ign= 0.42/  356 
             open_idle= 0.00 mig=77.23 xnuma_mig=12.37 xllc_mig=12.37 affn_viol= 1.23
             preempt/first/xllc/xnuma/idle/fail= 0.41/ 0.25/ 0.00/ 0.00/69.43/28.09 min_exec= 0.00/   0.00ms
             cpus=  4 [  2,  4] 00000003 00000300 00000000
             excl_coll=15.81 excl_preempt= 0.20
  random   : util/frac=    0.0/  0.0 load/frac=      0.0/  0.0 tasks=     0
             tot=      0 local= 0.00 wake/exp/reenq= 0.00/ 0.00/ 0.00
             xlayer_wake= 0.00 xlayer_rewake= 0.00
             keep/max/busy= 0.00/ 0.00/ 0.00 kick= 0.00 yield/ign= 0.00/    0 
             open_idle= 0.00 mig= 0.00 xnuma_mig= 0.00 xllc_mig= 0.00 affn_viol= 0.00
             preempt/first/xllc/xnuma/idle/fail= 0.00/ 0.00/ 0.00/ 0.00/ 0.00/ 0.00 min_exec= 0.00/   0.00ms
             cpus=  0 [  0,  0] 00000000 00000000 00000000
  stress-ng: util/frac=  968.7/ 86.9 load/frac=   1000.0/ 25.9 tasks=    11
             tot=     16 local= 0.00 wake/exp/reenq= 0.00/100.0/ 0.00
             xlayer_wake= 0.00 xlayer_rewake= 0.00
             keep/max/busy=66187.5/125.0/ 0.00 kick=100.0 yield/ign= 0.00/    0 
             open_idle= 0.00 mig=87.50 xnuma_mig=31.25 xllc_mig=31.25 affn_viol= 0.00
             preempt/first/xllc/xnuma/idle/fail= 0.00/ 0.00/ 0.00/ 0.00/ 0.00/ 0.00 min_exec= 0.00/   0.00ms
             cpus= 14 [ 14, 14] 07f00000 f0000000 00000007
             excl_coll= 0.00 excl_preempt= 0.00

Update comments on layer preemption to be more descriptive.

Signed-off-by: Daniel Hodges <[email protected]>
@hodgesds hodgesds merged commit 0b4a2af into sched-ext:main Sep 24, 2024
1 of 2 checks passed
@hodgesds hodgesds deleted the layered-preempt-fix branch September 26, 2024 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants