Eagle speculative decoding part 4: Add EAGLE2 worker#2150

Merged

merrymercy merged 94 commits intosgl-project:mainfrom yukavio:spec_infer

Jan 2, 2025

+1,212

Commits on Oct 13, 2024

temp
kavioyu
committed

Commits on Oct 14, 2024

migrated to new upstream, need implement evict memory
kavioyu
committed

Commits on Oct 15, 2024

prove single req
kavioyu
committed

Commits on Oct 16, 2024

fix bug for long generate due to eagle_verify_retrive kernel
kavioyu
committed
fix bug of eagle spec verify
kavioyu
committed

Commits on Oct 19, 2024

support cuda graph
kavioyu
committed

Commits on Oct 22, 2024

support batch inference
kavioyu
committed

Commits on Oct 23, 2024

temp
kavioyu
committed

Commits on Oct 24, 2024

fix memeory leak
kavioyu
committed
add sampling score
kavioyu
committed
support target model cuda graph
kavioyu
committed
disable target model cuda graph
kavioyu
committed
fix batch bug
kavioyu
committed

Commits on Oct 25, 2024

disable cuda graph pad in eagle
kavioyu
committed
fix server args
kavioyu
committed
fix cuda graph and split prefill
kavioyu
committed
optimize generate attn arg
kavioyu
committed
fix parent list dtype
kavioyu
committed

Commits on Oct 26, 2024

fix draft worker memory problem
kavioyu
committed
need to fix decode error when request retract happend
kavioyu
committed
remove debug info
kavioyu
committed

Commits on Oct 27, 2024

fix bug
kavioyu
committed

Commits on Oct 29, 2024

fix some bug and support target model use cuda graph
kavioyu
committed

Commits on Nov 1, 2024

fix conflict, should solve scheduler and cuda graph problem
kavioyu
committed
fix naive cuda graph
kavioyu
committed

Commits on Nov 2, 2024

fix cuda graph
kavioyu
committed
support split prefill batch
kavioyu
committed
fix cuda graph padding
kavioyu
committed

Commits on Nov 4, 2024

remove modification of target model and remove some redundant code
kavioyu
committed

Commits on Nov 5, 2024

fix cache management
kavioyu
committed

Commits on Nov 6, 2024

merge main and fix conflict
kavioyu
committed

Commits on Nov 7, 2024

add eagle example
kavioyu
committed
recover original example
kavioyu
committed

Commits on Nov 8, 2024

fix split prefill
kavioyu
committed

Commits on Nov 11, 2024

fix code style
kavioyu
committed

Commits on Nov 12, 2024

update to main and fix conflict
kavioyu
committed
add cutex to ci dependency
kavioyu
committed
add default value for speculative argument
kavioyu
committed
fix code style
kavioyu
committed

Commits on Nov 14, 2024

fix ci
kavioyu
committed

Commits on Nov 18, 2024

update and fix conflict
kavioyu
committed

Commits on Nov 22, 2024

update and fix conflict
kavioyu
committed
fix run with dp
kavioyu
committed
fix github comment
kavioyu
committed
remove flashinfer_utils
kavioyu
committed

Commits on Nov 23, 2024

update and fix conflict
kavioyu
committed
change forward_mode.target_verify to is_extend
kavioyu
committed
add eagle test to ci
kavioyu
committed
test ci
kavioyu
committed

Commits on Nov 24, 2024

add eagle test to run_suit
kavioyu
committed
Merge branch 'main' into spec_infer
merrymercy
authored
refine eagle ci and make eagle worker's input/output explicit
kavioyu
committed
fix ci
kavioyu
committed
remove performance test
kavioyu
committed

Commits on Nov 25, 2024

Commits on Dec 5, 2024

update to main
kavioyu
committed

Commits on Dec 6, 2024

fix comment
kavioyu
committed

Commits on Dec 29, 2024