WIP feat:Init commit for rust backend #1180

Aisuko · 2023-10-17T02:37:01Z

Description

This PR relates to #939

Notes for Reviewers

Signed commits

Yes, I signed my commits.

Signed-off-by: GitHub <[email protected]>

mudler · 2023-10-17T16:36:36Z

cc @lu-zero

.gitignore

backend/rust/Makefile

backend/rust/src/main.rs

Co-authored-by: Luca Barbato <[email protected]> Signed-off-by: Aisuko <[email protected]>

Signed-off-by: GitHub <[email protected]>

Signed-off-by: Aisuko <[email protected]>

backend/rust/bunker/build.rs

backend/rust/burn/src/main.rs

Signed-off-by: Aisuko <[email protected]>

backend/rust/bunker/src/service.rs

backend/rust/bunker/src/lib.rs

Co-authored-by: Luca Barbato <[email protected]> Signed-off-by: Aisuko <[email protected]>

backend/rust/burn/src/main.rs

Signed-off-by: Aisuko <[email protected]>

lu-zero · 2023-10-31T08:48:30Z

it seems to look for libtorch and fails to find it. if you use the ndarray backend does it work?

Aisuko · 2023-10-31T23:37:39Z

it seems to look for libtorch and fails to find it. if you use the ndarray backend does it work?

Will try it and give a feedback

Update

ndarary backend can be used to debug in IDE. And the torch backend has some issues on Mac M1. Here I am trying to set up LIBTORCH_USE_PYTORCH=1 as env with the conda env which is installed PyTorch. However, it is still hit other issues on M1 environment. So, I'm going to use ndarray to help me debug the conversion part code.

lu-zero · 2023-11-01T08:37:49Z

On the M1 probably the wgpu backend is the nicest to use, but ndarray is the one that does not depend on the host system.

Signed-off-by: Aisuko <[email protected]>

Aisuko · 2023-11-01T11:09:56Z

On the M1 probably the wgpu backend is the nicest to use, but ndarray is the one that does not depend on the host system.

Thanks a lot. I have made some change here. I have been migrated the code which is included Llama2 to fork repo, and I am working on the a more simpler model. Here are some reasons:

A simpler model can be more effecient to debug than Llama2, less parameters, and less memory used. (Only load half of Llama2 parameters to tensor can cost at least 13min in my local env now)
We can move faster on this PR. It is good for us to refractor the code, project structure and abstract some common traits.
Easy for code reviewing
Easy for adding some test cases(CI).

Here I hit an issue on reshaping of the Tensor. So, we can try to implement a simple one instead of getting stuck on the Llama2.

backend/rust/models/src/lib.rs

Signed-off-by: Aisuko <[email protected]>

lu-zero · 2023-11-16T06:34:15Z

backend/rust/models/src/whisper/utils.rs

+        // And now the nonlinear scale
+        let min_log_hz = 1000.0; // beginning of log region (Hz)
+        let min_log_mel = (min_log_hz - f_min) / f_sp;
+        let logstep = (6.4f64).ln() / 27.0; // step size for log region


those constants are repeated, being always f64 you can just keep them as consts

thank you, will do.

Signed-off-by: Aisuko <[email protected]>

backend/rust/backend/src/main.rs

Signed-off-by: Aisuko <[email protected]>

netlify · 2023-11-23T01:15:35Z

❌ Deploy Preview for localai failed.

Name	Link
🔨 Latest commit	`c990112`
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/655ea7b3d02aec0008ca4cdf

Aisuko · 2023-11-23T01:22:15Z

backend/rust/models/src/llama/llama.rs

+
+        let tensor3=tensor2.transpose();
+
+        let tensor41=tensor3.repeat(2, 2);


@lu-zero Here, I am going to use wgpu backend instead of tch. However, I the repeat function here only support 2 dimensions tensor, (Can only repeat dimension with dim=1) https://github.com/Tracel-AI/burn/blob/b86bc5876149bd73bc59cb5197fd3ee8b92509d4/burn-tensor/src/tensor/ops/tensor.rs#L222C7-L222C7.

I have been tried several solutions, like use swap_dims and flattern these internal function of Tensor, but here hard to say it is correct and also causes other issues. Is there a better example for this?

asking upstream probably it is the best route (sorry for the belated reply, I got very busy and the message got lost in the mailbox)

No worries, thanks for your support. I will continue to work on this one after I applied PhD successfully. Currently, sooo busy. But I still want to get this PR to merged.

Once you are more free please contact me, probably a good deal of the issues will be ironed out by upstream meanwhile :)

Aisuko marked this pull request as draft October 17, 2023 02:38

Aisuko self-assigned this Oct 17, 2023

Aisuko force-pushed the feat/rust_grpc branch 2 times, most recently from aacaf4e to afcd7bd Compare October 17, 2023 04:53

Init commit for rust backend

ef3fe9a

Signed-off-by: GitHub <[email protected]>

Aisuko force-pushed the feat/rust_grpc branch from afcd7bd to ef3fe9a Compare October 17, 2023 04:57

Aisuko added the new-backend label Oct 17, 2023

lu-zero reviewed Oct 17, 2023

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

lu-zero reviewed Oct 17, 2023

View reviewed changes

backend/rust/Makefile Outdated Show resolved Hide resolved

lu-zero reviewed Oct 17, 2023

View reviewed changes

backend/rust/src/main.rs Outdated Show resolved Hide resolved

lu-zero reviewed Oct 17, 2023

View reviewed changes

backend/rust/src/main.rs Outdated Show resolved Hide resolved

Aisuko and others added 3 commits October 18, 2023 10:47

Update backend/rust/Makefile

029a71f

Co-authored-by: Luca Barbato <[email protected]> Signed-off-by: Aisuko <[email protected]>

Add tracing

5c67aa6

Signed-off-by: GitHub <[email protected]>

Add workspace

1806dd7

Signed-off-by: Aisuko <[email protected]>

Aisuko requested a review from lu-zero October 18, 2023 08:00

lu-zero reviewed Oct 18, 2023

View reviewed changes

backend/rust/bunker/build.rs Outdated Show resolved Hide resolved

lu-zero reviewed Oct 18, 2023

View reviewed changes

backend/rust/burn/src/main.rs Outdated Show resolved Hide resolved

lu-zero reviewed Oct 18, 2023

View reviewed changes

backend/rust/burn/src/main.rs Outdated Show resolved Hide resolved

Aisuko requested a review from lu-zero October 19, 2023 04:51

Replace the generated file to the generated folder

61bd269

Signed-off-by: Aisuko <[email protected]>

Aisuko force-pushed the feat/rust_grpc branch from 9b74e4e to 61bd269 Compare October 19, 2023 08:43

lu-zero reviewed Oct 19, 2023

View reviewed changes

backend/rust/bunker/src/service.rs Outdated Show resolved Hide resolved

lu-zero reviewed Oct 19, 2023

View reviewed changes

backend/rust/bunker/src/lib.rs Outdated Show resolved Hide resolved

Update backend/rust/bunker/src/lib.rs

b92677b

Co-authored-by: Luca Barbato <[email protected]> Signed-off-by: Aisuko <[email protected]>

Aisuko commented Oct 20, 2023

View reviewed changes

backend/rust/burn/src/main.rs Outdated Show resolved Hide resolved

Remove services.rs

a2bb86f

Signed-off-by: Aisuko <[email protected]>

Aisuko force-pushed the feat/rust_grpc branch from ef8a86b to a2bb86f Compare October 20, 2023 00:36

Aisuko requested a review from lu-zero October 20, 2023 01:09

Add test health in Makefile

bc6c1fc

Signed-off-by: Aisuko <[email protected]>

Aisuko added 2 commits November 1, 2023 20:12

Add new model

c0dadcc

Signed-off-by: Aisuko <[email protected]>

Implement a new simple model

fb67c91

Signed-off-by: Aisuko <[email protected]>

Aisuko force-pushed the feat/rust_grpc branch from 4c7f5ca to fb67c91 Compare November 1, 2023 10:55

Aisuko force-pushed the feat/rust_grpc branch 3 times, most recently from d9f1f7d to da3a0d8 Compare November 3, 2023 11:50

Aisuko commented Nov 3, 2023

View reviewed changes

backend/rust/models/src/lib.rs Show resolved Hide resolved

Implement MNIST model and inference

1d2fd99

Signed-off-by: Aisuko <[email protected]>

Aisuko force-pushed the feat/rust_grpc branch 4 times, most recently from cb216fa to ed95d9c Compare November 4, 2023 02:26

Add check memory feature

660cc49

Signed-off-by: Aisuko <[email protected]>

Aisuko force-pushed the feat/rust_grpc branch from ed95d9c to 660cc49 Compare November 4, 2023 02:43

Aisuko mentioned this pull request Nov 5, 2023

[EPIC] Model support dashboard (v2) #1126

Open

lu-zero reviewed Nov 16, 2023

View reviewed changes

Aisuko force-pushed the feat/rust_grpc branch 3 times, most recently from a6ff963 to b91b79c Compare November 18, 2023 01:35

Trying to call mnist model in main

d62c701

Signed-off-by: Aisuko <[email protected]>

Aisuko force-pushed the feat/rust_grpc branch from b91b79c to d62c701 Compare November 18, 2023 07:29

Add test case for load model and import getusage

b210203

Signed-off-by: Aisuko <[email protected]>

Aisuko commented Nov 19, 2023

View reviewed changes

backend/rust/backend/src/main.rs Show resolved Hide resolved

Add llama for test

c990112

Signed-off-by: Aisuko <[email protected]>

Aisuko commented Nov 23, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP feat:Init commit for rust backend #1180

WIP feat:Init commit for rust backend #1180

Aisuko commented Oct 17, 2023 •

edited

Loading

mudler commented Oct 17, 2023

lu-zero commented Oct 31, 2023

Aisuko commented Oct 31, 2023 •

edited

Loading

lu-zero commented Nov 1, 2023

Aisuko commented Nov 1, 2023

lu-zero Nov 16, 2023

Aisuko Nov 17, 2023

netlify bot commented Nov 23, 2023 •

edited

Loading

Aisuko Nov 23, 2023

lu-zero Dec 21, 2023

Aisuko Jul 2, 2024

lu-zero Jul 2, 2024


		let tensor3=tensor2.transpose();

		let tensor41=tensor3.repeat(2, 2);

WIP feat:Init commit for rust backend #1180

Are you sure you want to change the base?

WIP feat:Init commit for rust backend #1180

Conversation

Aisuko commented Oct 17, 2023 • edited Loading

mudler commented Oct 17, 2023

lu-zero commented Oct 31, 2023

Aisuko commented Oct 31, 2023 • edited Loading

Update

lu-zero commented Nov 1, 2023

Aisuko commented Nov 1, 2023

lu-zero Nov 16, 2023

Choose a reason for hiding this comment

Aisuko Nov 17, 2023

Choose a reason for hiding this comment

netlify bot commented Nov 23, 2023 • edited Loading

❌ Deploy Preview for localai failed.

Aisuko Nov 23, 2023

Choose a reason for hiding this comment

lu-zero Dec 21, 2023

Choose a reason for hiding this comment

Aisuko Jul 2, 2024

Choose a reason for hiding this comment

lu-zero Jul 2, 2024

Choose a reason for hiding this comment

Aisuko commented Oct 17, 2023 •

edited

Loading

Aisuko commented Oct 31, 2023 •

edited

Loading

netlify bot commented Nov 23, 2023 •

edited

Loading