docs: add finetuner docs #771

ZiniuYu · 2022-07-18T07:47:39Z

This pr adds instructions on how to use fine-tuned model in clip-server trained by finetuner.
The doc is currently under user-guides. Will decide a more appropriate place later

codecov · 2022-07-18T07:51:10Z

Codecov Report

Merging #771 (ee7c62e) into main (2b78b12) will decrease coverage by 0.24%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #771      +/-   ##
==========================================
- Coverage   82.09%   81.85%   -0.25%     
==========================================
  Files          17       17              
  Lines        1240     1251      +11     
==========================================
+ Hits         1018     1024       +6     
- Misses        222      227       +5

Flag	Coverage Δ
cas	`81.85% <ø> (-0.25%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
client/clip_client/client.py	`86.38% <0.00%> (-1.95%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ca03dca...ee7c62e. Read the comment docs.

numb3r3

EMPTY?

ZiniuYu · 2022-07-18T14:17:05Z

docs failing. possible reason: ryanfox/sphinx-markdown-tables#36

docs/user-guides/finetuner.md

bwanglzu

Thanks! Left some general comments.

numb3r3 · 2022-07-19T01:35:24Z

docs failing. possible reason: ryanfox/sphinx-markdown-tables#36

I will fix this issue in main since it breaks our doc building, please rebase your PR then.

docs/user-guides/finetuner.md

docs/requirements.txt

bwanglzu · 2022-07-19T09:35:48Z

i guess we need to tweak the words a bit, I'll invite @violenil to take a look

violenil · 2022-07-19T09:40:33Z

docs/user-guides/finetuner.md

@@ -0,0 +1,186 @@
+# Fine-tune Models
+
+Although CLIP-as-service has provided you a list of pre-trained models, you can also train your models. 


you can also fine-tune your models.

numb3r3 · 2022-07-19T09:30:38Z

docs/user-guides/finetuner.md

+    epochs=5,
+    learning_rate=1e-5,
+    loss='CLIPLoss',
+    cpu=False,


@bwanglzu Just want to confirm will cpu=False work defaultly?

by default, finetuner use cpu, if you want to use gpu, you have to set cpu=False. @numb3r3

Let me clarify my question. What I really want to ask is "Does finetuner currently allows training a model on GPU for free"?

numb3r3 · 2022-07-19T09:40:08Z

docs/user-guides/finetuner.md

+```
+
+The full dataset has been converted to `clip-fashion-train-data` and `clip-fashion-eval-data` and pushed to the cloud.
+You can directly use them in Finetuner.


You can directly use them in Finetuner.

You can directly use them by providing the dataset name in

numb3r3 · 2022-07-19T09:40:27Z

docs/user-guides/finetuner.md

+train_da.push('fashion-sample')
+```
+
+The full dataset has been converted to `clip-fashion-train-data` and `clip-fashion-eval-data` and pushed to the cloud.


which can ...

numb3r3 · 2022-07-19T09:42:38Z

docs/user-guides/finetuner.md

+run = finetuner.fit(
+    model='openai/clip-vit-base-patch32',
+    run_name='clip-fashion',
+    train_data='clip-fashion-train-data',


@bwanglzu will finetuner accept docarray instance (rather than a name in hubble) as the dataset input?

yes, both are supported! it can be a string of docarray name, or a document array python instance @numb3r3

prettry cool

docs/user-guides/finetuner.md

violenil · 2022-07-19T09:44:26Z

docs/user-guides/finetuner.md

+You can push the resulting [`DocumentArray`](https://docarray.jina.ai/fundamentals/documentarray/) to the cloud using the [`.push`](https://docarray.jina.ai/api/docarray.array.document/?highlight=push#docarray.array.document.DocumentArray.push) method.
+
+We use [fashion captioning dataset](https://github.com/xuewyang/Fashion_Captioning) as a sample dataset in this tutorial.
+You can get the description and image url from the dataset: 


Suggested change

You can get the description and image url from the dataset:

The following are examples of descriptions and image urls from the dataset:

Maybe you can show the user how they can view these descriptions + urls themselves?

Yes I can, but i think this will distract users and it is not the main topic of this tutorial

violenil · 2022-07-19T09:49:07Z

docs/user-guides/finetuner.md

+
+Since the tuned model generated from Finetuner contains richer information such as metadata and config, we now transform it to simpler structure used by CLIP-as-service.
+
+* First create a new folder named `clip-fashion-cas` or anything you like. This will be the storage of the models to use in CLIP-as-service.


Suggested change

* First create a new folder named `clip-fashion-cas` or anything you like. This will be the storage of the models to use in CLIP-as-service.

* Firstly, create a new folder named `clip-fashion-cas` or name of your choice. This will be the storage of the models to use in CLIP-as-service.

docs/user-guides/finetuner.md

violenil · 2022-07-19T09:52:19Z

docs/user-guides/finetuner.md

+
+* Second copy and move `clip-fashion/models/clip-text/model.onnx` to `clip-fashion-cas` and rename it to `textual.onnx`.
+
+* Similarly, copy and move `clip-fashion/models/clip-vision/model.onnx` to `clip-fashion-cas` and rename it to `visual.onnx`.


Suggested change

* Similarly, copy and move `clip-fashion/models/clip-vision/model.onnx` to `clip-fashion-cas` and rename it to `visual.onnx`.

* Similarly, copy the vision model `clip-fashion/models/clip-vision/model.onnx` into `clip-fashion-cas` and rename the model to `visual.onnx`.

docs/user-guides/finetuner.md

Co-authored-by: Isabelle Mohr <[email protected]>

docs/user-guides/finetuner.md

bwanglzu

LGTM!

numb3r3 · 2022-07-20T03:12:57Z

docs/conf.py

@@ -80,6 +80,8 @@
 html_show_sourcelink = False
 html_favicon = '_static/favicon.png'

+intersphinx_mapping = {'docarray': ('https://docarray.jina.ai/', None), 'finetuner': ('https://finetuner.jina.ai/', None)}


perfect, good catch

numb3r3 · 2022-07-20T03:13:38Z

docs/user-guides/finetuner.md

+
+| Description                                                                                                                           | Image URL                                                                                                                                                           |
+|---------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| subtly futuristic and edgy this liquid metal cuff bracelet is shaped from sculptural rectangular link                                 | [https://n.nordstrommedia.com/id/sr3/<br/>58d1a13f-b6b6-4e68-b2ff-3a3af47c422e.jpeg](https://n.nordstrommedia.com/id/sr3/58d1a13f-b6b6-4e68-b2ff-3a3af47c422e.jpeg) |


could we display the image, rather than simply print URL?

I also kept the original image urls since it is from the dataset

numb3r3 · 2022-07-20T03:14:00Z

docs/user-guides/finetuner.md

+run = finetuner.fit(
+    model='openai/clip-vit-base-patch32',
+    run_name='clip-fashion',
+    train_data='clip-fashion-train-data',


prettry cool

numb3r3 · 2022-07-20T03:15:42Z

docs/user-guides/finetuner.md

+    epochs=5,
+    learning_rate=1e-5,
+    loss='CLIPLoss',
+    cpu=False,


Let me clarify my question. What I really want to ask is "Does finetuner currently allows training a model on GPU for free"?

docs/user-guides/server.md

github-actions · 2022-07-20T07:28:39Z

📝 Docs are deployed on https://ft-finetuner-docs--jina-docs.netlify.app 🎉

github-actions bot added size/xs area/docs labels Jul 18, 2022

numb3r3 requested changes Jul 18, 2022

View reviewed changes

github-actions bot added size/m area/cicd area/housekeeping and removed size/xs area/housekeeping area/cicd labels Jul 18, 2022

bwanglzu reviewed Jul 18, 2022

View reviewed changes

docs/user-guides/finetuner.md Outdated Show resolved Hide resolved

bwanglzu reviewed Jul 18, 2022

View reviewed changes

docs/user-guides/finetuner.md Outdated Show resolved Hide resolved

bwanglzu reviewed Jul 18, 2022

View reviewed changes

docs/user-guides/finetuner.md Outdated Show resolved Hide resolved

bwanglzu reviewed Jul 18, 2022

View reviewed changes

docs/user-guides/finetuner.md Outdated Show resolved Hide resolved

bwanglzu reviewed Jul 18, 2022

View reviewed changes

numb3r3 requested changes Jul 19, 2022

View reviewed changes

docs/user-guides/finetuner.md Outdated Show resolved Hide resolved

numb3r3 requested changes Jul 19, 2022

View reviewed changes

docs/requirements.txt Outdated Show resolved Hide resolved

ZiniuYu added 7 commits July 19, 2022 10:23

docs: add finetuner docs

8a1221d

docs: add finetuner instruction

8e5bdda

docs: add finetuner instruction

5b0eac6

docs: add finetuner instruction

56f7305

docs: add finetuner instruction

00d8e97

docs: add finetuner instruction

f55c6ab

docs: add finetuner instruction

24e0737

ZiniuYu force-pushed the finetuner-docs branch from d5ba6e8 to 24e0737 Compare July 19, 2022 02:24

ZiniuYu added 3 commits July 19, 2022 10:46

docs: improve narratives

ed99910

docs: minor revision

c1f71b5

docs: minor revision

3079dbc

docs: fix table

1847eb3

violenil reviewed Jul 19, 2022

View reviewed changes

numb3r3 requested changes Jul 19, 2022

View reviewed changes

violenil reviewed Jul 19, 2022

View reviewed changes

docs/user-guides/finetuner.md Outdated Show resolved Hide resolved

violenil reviewed Jul 19, 2022

View reviewed changes

docs/user-guides/finetuner.md Outdated Show resolved Hide resolved

violenil reviewed Jul 19, 2022

View reviewed changes

docs/user-guides/finetuner.md Outdated Show resolved Hide resolved

ZiniuYu and others added 4 commits July 19, 2022 17:55

docs: minor revision

2c5c9b2

Co-authored-by: Isabelle Mohr <[email protected]>

docs: minor revision

4129ca2

Co-authored-by: Isabelle Mohr <[email protected]>

docs: minor revision

2848d75

Co-authored-by: Isabelle Mohr <[email protected]>

docs: address comment

ac26b4f

violenil reviewed Jul 19, 2022

View reviewed changes

docs/user-guides/finetuner.md Outdated Show resolved Hide resolved

ZiniuYu added 3 commits July 19, 2022 18:41

docs: restructure

7a8b46b

docs: intersphinx

85adb22

docs: typo

1d9f967

ZiniuYu marked this pull request as ready for review July 19, 2022 11:18

bwanglzu approved these changes Jul 19, 2022

View reviewed changes

numb3r3 requested changes Jul 20, 2022

View reviewed changes

ZiniuYu added 6 commits July 20, 2022 12:05

docs: add image preview

8127dcd

docs: fix image link

0ef6e41

docs: fix typo

05df0fe

docs: add model path example

83c0866

docs: add model path example

5f33b2b

docs: improve narratives

ee7c62e

numb3r3 merged commit bc6b72e into main Jul 20, 2022

numb3r3 deleted the finetuner-docs branch July 20, 2022 09:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: add finetuner docs #771

docs: add finetuner docs #771

ZiniuYu commented Jul 18, 2022

codecov bot commented Jul 18, 2022 •

edited

Loading

numb3r3 left a comment

ZiniuYu commented Jul 18, 2022

bwanglzu left a comment

numb3r3 commented Jul 19, 2022

bwanglzu commented Jul 19, 2022

violenil Jul 19, 2022

numb3r3 Jul 19, 2022 •

edited

Loading

bwanglzu Jul 19, 2022 •

edited

Loading

numb3r3 Jul 20, 2022

numb3r3 Jul 19, 2022

numb3r3 Jul 19, 2022

numb3r3 Jul 19, 2022

bwanglzu Jul 19, 2022 •

edited

Loading

numb3r3 Jul 20, 2022

violenil Jul 19, 2022

violenil Jul 19, 2022

ZiniuYu Jul 19, 2022

violenil Jul 19, 2022

violenil Jul 19, 2022

bwanglzu left a comment

numb3r3 Jul 20, 2022

numb3r3 Jul 20, 2022

ZiniuYu Jul 20, 2022

ZiniuYu Jul 20, 2022

numb3r3 Jul 20, 2022

numb3r3 Jul 20, 2022

github-actions bot commented Jul 20, 2022

		@@ -0,0 +1,186 @@
		# Fine-tune Models

		Although CLIP-as-service has provided you a list of pre-trained models, you can also train your models.

	You can get the description and image url from the dataset:
	The following are examples of descriptions and image urls from the dataset:


		Since the tuned model generated from Finetuner contains richer information such as metadata and config, we now transform it to simpler structure used by CLIP-as-service.

		* First create a new folder named `clip-fashion-cas` or anything you like. This will be the storage of the models to use in CLIP-as-service.

	* First create a new folder named `clip-fashion-cas` or anything you like. This will be the storage of the models to use in CLIP-as-service.
	* Firstly, create a new folder named `clip-fashion-cas` or name of your choice. This will be the storage of the models to use in CLIP-as-service.


		* Second copy and move `clip-fashion/models/clip-text/model.onnx` to `clip-fashion-cas` and rename it to `textual.onnx`.

		* Similarly, copy and move `clip-fashion/models/clip-vision/model.onnx` to `clip-fashion-cas` and rename it to `visual.onnx`.

	* Similarly, copy and move `clip-fashion/models/clip-vision/model.onnx` to `clip-fashion-cas` and rename it to `visual.onnx`.
	* Similarly, copy the vision model `clip-fashion/models/clip-vision/model.onnx` into `clip-fashion-cas` and rename the model to `visual.onnx`.

docs: add finetuner docs #771

docs: add finetuner docs #771

Conversation

ZiniuYu commented Jul 18, 2022

codecov bot commented Jul 18, 2022 • edited Loading

Codecov Report

numb3r3 left a comment

Choose a reason for hiding this comment

ZiniuYu commented Jul 18, 2022

bwanglzu left a comment

Choose a reason for hiding this comment

numb3r3 commented Jul 19, 2022

bwanglzu commented Jul 19, 2022

Choose a reason for hiding this comment

numb3r3 Jul 19, 2022 • edited Loading

Choose a reason for hiding this comment

bwanglzu Jul 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwanglzu Jul 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwanglzu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jul 20, 2022

codecov bot commented Jul 18, 2022 •

edited

Loading

numb3r3 Jul 19, 2022 •

edited

Loading

bwanglzu Jul 19, 2022 •

edited

Loading

bwanglzu Jul 19, 2022 •

edited

Loading