Add new apple's model #28

tucan9389 · 2019-07-18T00:08:21Z

Models

Vision

Model Name	Problem	Demo Project	Performance Test	Demo Gif	Added in here
MobileNetV2	Image Classification	Done	Done	Done	`TODO`
Resnet50	Image Classification	Done	Done	`TODO`	`TODO`
SqueezeNet	Image Classification	Done	Done	`TODO`	`TODO`
MNIST	Drawing Classification	`TODO`	`TODO`	`TODO`	`TODO`
YOLOv3-Tiny	Object Detection	Done	Done	`TODO`	`TODO`
YOLOv3	Object Detection)	Done	Done	`TODO`	`TODO`
DeeplabV3	Image Segmentation	Done	Done	Done	`TODO`
FCRN-DepthPrediction	Depth Estimation	Done	Done	Done	`TODO`

Text

Model Name	Problem	Demo Project	Performance Test	Demo Gif	Added in here
BERT-SQuAD	Question Answering	`TODO`	`TODO`	`TODO`	`TODO`

FCRN-DepthPrediction Demo

https://github.com/tucan9389/DepthPrediction-CoreML

tucan9389 · 2019-07-27T04:07:26Z

DeeplabV3 Demo

https://github.com/tucan9389/ObjectSegmentation-CoreML

tucan9389 · 2019-07-27T04:07:57Z

MobileNetV2, Resnet50, SqueezeNet Demo

https://github.com/tucan9389/ImageClassification-CoreML/

tucan9389 · 2019-07-27T04:08:19Z

YOLOv3, YOLOv3-Tiny Demo

https://github.com/tucan9389/ObjectDetection-CoreML

tucan9389 · 2019-07-27T04:55:56Z

Image Classification

Model Size (MB), Minimum iOS Version

Model	Size	Minimum iOS Version
MobileNet	17.1	iOS11
MobileNetV2	24.7	iOS11
MobileNetV2FP16	12.4	iOS11.2
MobileNetV2Int8LUT	6.3	iOS12
Resnet50	102.6	iOS11
Resnet50FP16	51.3	iOS11.2
Resnet50Int8LUT	25.7	iOS12
Resnet50Headless	94.4	iOS11
SqueezeNet	5	iOS11
SqueezeNetFP16	2.5	iOS11.2
SqueezeNetInt8LUT	1.3	iOS12

Infernece Time (ms)

Model vs. Device	XS	XS Max	XR	X	7+	7
MobileNet	16	18	19	33	43	35
MobileNetV2	21	18	21	46	64	53
MobileNetV2FP16	20	19	20	48	65	57
MobileNetV2Int8LUT	21	21	20	53	64	53
Resnet50	27	25	26	61	78	63
Resnet50FP16	26	26	27	64	75	74
Resnet50Int8LUT	27	25	26	60	77	75
Resnet50Headless	18	13	18	36	54	53
SqueezeNet	17	17	18	24	35	29
SqueezeNetFP16	17	17	18	24	36	29
SqueezeNetInt8LUT	18	19	18	27	34	30

Total Time (ms)

Model vs. Device	XS	XS Max	XR	X	7+	7
MobileNet	18	20	21	35	46	37
MobileNetV2	23	21	23	48	67	55
MobileNetV2FP16	24	21	23	50	69	60
MobileNetV2Int8LUT	23	23	22	55	67	56
Resnet50	30	28	29	64	82	66
Resnet50FP16	28	28	30	66	78	76
Resnet50Int8LUT	29	28	28	63	80	78
Resnet50Headless	19	13	18	36	54	54
SqueezeNet	18	18	20	25	37	31
SqueezeNetFP16	18	18	19	26	38	31
SqueezeNetInt8LUT	20	20	19	29	37	32

FPS

Model vs. Device	XS	XS Max	XR	X	7+	7
MobileNet	23	23	23	23	20	23
MobileNetV2	23	23	23	20	13	17
MobileNetV2FP16	23	23	23	18	13	15
MobileNetV2Int8LUT	23	23	23	16	13	16
Resnet50	23	23	23	14	11	14
Resnet50FP16	23	23	23	14	11	12
Resnet50Int8LUT	23	23	23	15	11	12
Resnet50Headless	23	23	23	23	16	17
SqueezeNet	23	23	23	23	23	23
SqueezeNetFP16	23	23	23	23	22	23
SqueezeNetInt8LUT	23	23	23	23	23	23

tucan9389 · 2019-07-27T05:57:09Z

Object Detection

Model Size (MB), Minimum iOS Version

Model	Size	Minimum iOS Version
YOLOv3	248.4	iOS12
YOLOv3FP16	124.2	iOS12
YOLOv3Int8LUT	62.2	iOS12
YOLOv3Tiny	35.5	iOS12
YOLOv3TinyFP16	17.8	iOS12
YOLOv3TinyInt8LUT	8.9	iOS12
MobileNetV2_SSDLite	9.3	iOS12
ObjectDetector	63.7	iOS12

Infernece Time (ms)

Model vs. Device	XS	XS Max	XR	X	7+	7
YOLOv3	108	93	100	356	569	561
YOLOv3FP16	104	89	101	348	572	565
YOLOv3Int8LUT	101	92	100	337	575	572
YOLOv3Tiny	46	41	47	106	165	168
YOLOv3TinyFP16	51	41	44	103	165	167
YOLOv3TinyInt8LUT	45	39	39	106	160	161
MobileNetV2_SSDLite	31	31	31	109	141	134
ObjectDetector	24	26	23	63	86	84

Total Time (ms)

Model vs. Device	XS	XS Max	XR	X	7+	7
YOLOv3	108	93	100	357	569	561
YOLOv3FP16	104	89	101	348	572	565
YOLOv3Int8LUT	102	92	102	338	576	573
YOLOv3Tiny	46	42	48	106	166	169
YOLOv3TinyFP16	51	41	44	104	165	167
YOLOv3TinyInt8LUT	45	39	40	107	160	161
MobileNetV2_SSDLite	32	31	32	109	142	134
ObjectDetector	25	26	23	64	87	85

FPS

Model vs. Device	XS	XS Max	XR	X	7+	7
YOLOv3	8	10	9	2	1	1
YOLOv3FP16	9	10	8	2	1	1
YOLOv3Int8LUT	9	10	9	2	1	1
YOLOv3Tiny	21	22	20	8	5	5
YOLOv3TinyFP16	19	23	21	9	5	5
YOLOv3TinyInt8LUT	21	24	23	8	5	5
MobileNetV2_SSDLite	23	23	23	8	6	6
ObjectDetector	23	23	24	14	10	11

tucan9389 · 2019-07-27T06:03:52Z

Table Convert Script

Convert text to markdown table format

dict = {}
device_name = None
device_names = []

f = open("performance-classification.txt", 'r')
lines = f.readlines()
for line in lines:
    componants = line.split()
    # if len(componants) == 0:

    if len(componants) == 0:
        continue

    if len(componants) != 5:
        device_name = " ".join(componants)
        device_names.append(device_name)
        continue
    else:
        model_name = componants[0]
        inference_time = componants[2]
        total_time = componants[3]
        fps = componants[4]

        if model_name not in dict:
            dict[model_name] = {}

        dict[model_name][device_name] = {
            "inference_time": inference_time,
            "total_time": total_time,
            "fps": fps
        }
f.close()




def get_markdown_table(dict, colunm_name = "inference_time"):
    result = ""

    row_text = "| Model vs. Device | "
    row_text = row_text + " | ".join(device_names)
    row_text = row_text + " |"
    result += row_text + "\n"
    # print(row_text)

    row_text = "| ----: | "
    for i in range(len(device_names)):
        row_text += ":----: | "
    result += row_text + "\n"
    # print(row_text)

    for model_name in dict.keys():
        row_text = "| " + model_name + " | "
        for device_name in device_names:
            row_text += dict[model_name][device_name][colunm_name] + " | "
        result += row_text + "\n"

    return result

# inference_time
# total_time
# fps

print()
print()
print()

print("## Object Detection")
print()

print("### Infernece Time (ms)")
print()
print(get_markdown_table(dict=dict, colunm_name="inference_time"))

print("### Total Time (ms)")
print()
print(get_markdown_table(dict=dict, colunm_name="total_time"))

print("### FPS")
print()
print(get_markdown_table(dict=dict, colunm_name="fps"))

Input text file

XS Max
MobileNet           : 18 20 23
MobileNetV2         : 18 21 23
MobileNetV2FP16     : 19 21 23
MobileNetV2Int8LUT  : 21 23 23
Resnet50            : 25 28 23
Resnet50FP16        : 26 28 23
Resnet50Int8LUT     : 25 28 23
Resnet50Headless    : 13 13 23
SqueezeNet          : 17 18 23
SqueezeNetFP16      : 17 18 23
SqueezeNetInt8LUT   : 19 20 23

XR
MobileNet           : 19 21 23
MobileNetV2         : 21 23 23
MobileNetV2FP16     : 20 23 23
MobileNetV2Int8LUT  : 20 22 23
Resnet50            : 26 29 23
Resnet50FP16        : 27 30 23
Resnet50Int8LUT     : 26 28 23
Resnet50Headless    : 18 18 23
SqueezeNet          : 18 20 23
SqueezeNetFP16      : 18 19 23
SqueezeNetInt8LUT   : 18 19 23

7+
MobileNet           : 43 46 20
MobileNetV2         : 64 67 13
MobileNetV2FP16     : 65 69 13
MobileNetV2Int8LUT  : 64 67 13
Resnet50            : 78 82 11
Resnet50FP16        : 75 78 11
Resnet50Int8LUT     : 77 80 11
Resnet50Headless    : 54 54 16
SqueezeNet          : 35 37 23
SqueezeNetFP16      : 36 38 22
SqueezeNetInt8LUT   : 34 37 23

7
MobileNet           : 35 37 23
MobileNetV2         : 53 55 17
MobileNetV2FP16     : 57 60 15
MobileNetV2Int8LUT  : 53 56 16
Resnet50            : 63 66 14
Resnet50FP16        : 74 76 12
Resnet50Int8LUT     : 75 78 12
Resnet50Headless    : 53 54 17
SqueezeNet          : 29 31 23
SqueezeNetFP16      : 29 31 23
SqueezeNetInt8LUT   : 30 32 23

Output text




## Object Detection

### Infernece Time (ms)

| Model vs. Device | XS Max | XR | 7+ | 7 |
| ---- | ---- | ---- | ---- | ---- | 
| MobileNet | 18 | 19 | 43 | 35 | 
| MobileNetV2 | 18 | 21 | 64 | 53 | 
| MobileNetV2FP16 | 19 | 20 | 65 | 57 | 
| MobileNetV2Int8LUT | 21 | 20 | 64 | 53 | 
| Resnet50 | 25 | 26 | 78 | 63 | 
| Resnet50FP16 | 26 | 27 | 75 | 74 | 
| Resnet50Int8LUT | 25 | 26 | 77 | 75 | 
| Resnet50Headless | 13 | 18 | 54 | 53 | 
| SqueezeNet | 17 | 18 | 35 | 29 | 
| SqueezeNetFP16 | 17 | 18 | 36 | 29 | 
| SqueezeNetInt8LUT | 19 | 18 | 34 | 30 | 

### Total Time (ms)

| Model vs. Device | XS Max | XR | 7+ | 7 |
| ---- | ---- | ---- | ---- | ---- | 
| MobileNet | 20 | 21 | 46 | 37 | 
| MobileNetV2 | 21 | 23 | 67 | 55 | 
| MobileNetV2FP16 | 21 | 23 | 69 | 60 | 
| MobileNetV2Int8LUT | 23 | 22 | 67 | 56 | 
| Resnet50 | 28 | 29 | 82 | 66 | 
| Resnet50FP16 | 28 | 30 | 78 | 76 | 
| Resnet50Int8LUT | 28 | 28 | 80 | 78 | 
| Resnet50Headless | 13 | 18 | 54 | 54 | 
| SqueezeNet | 18 | 20 | 37 | 31 | 
| SqueezeNetFP16 | 18 | 19 | 38 | 31 | 
| SqueezeNetInt8LUT | 20 | 19 | 37 | 32 | 

### FPS

| Model vs. Device | XS Max | XR | 7+ | 7 |
| ---- | ---- | ---- | ---- | ---- | 
| MobileNet | 23 | 23 | 20 | 23 | 
| MobileNetV2 | 23 | 23 | 13 | 17 | 
| MobileNetV2FP16 | 23 | 23 | 13 | 15 | 
| MobileNetV2Int8LUT | 23 | 23 | 13 | 16 | 
| Resnet50 | 23 | 23 | 11 | 14 | 
| Resnet50FP16 | 23 | 23 | 11 | 12 | 
| Resnet50Int8LUT | 23 | 23 | 11 | 12 | 
| Resnet50Headless | 23 | 23 | 16 | 17 | 
| SqueezeNet | 23 | 23 | 23 | 23 | 
| SqueezeNetFP16 | 23 | 23 | 22 | 23 | 
| SqueezeNetInt8LUT | 23 | 23 | 23 | 23 |

tucan9389 · 2019-07-30T09:40:12Z

The results are weird. Maximum FPS is 23.. (Not 30)
Need to test again. 😓

tucan9389 self-assigned this Jul 18, 2019

tucan9389 changed the title ~~Add new apple's model~~ [TODO] Add new apple's model Jul 18, 2019

tucan9389 added the performance test label Aug 4, 2019

tucan9389 changed the title ~~[TODO] Add new apple's model~~ Add new apple's model Aug 4, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new apple's model #28

Add new apple's model #28

tucan9389 commented Jul 18, 2019 •

edited

Loading

tucan9389 commented Jul 18, 2019

tucan9389 commented Jul 27, 2019

tucan9389 commented Jul 27, 2019

tucan9389 commented Jul 27, 2019

tucan9389 commented Jul 27, 2019

tucan9389 commented Jul 27, 2019 •

edited

Loading

tucan9389 commented Jul 27, 2019 •

edited

Loading

tucan9389 commented Jul 27, 2019 •

edited

Loading

tucan9389 commented Jul 30, 2019

Add new apple's model #28

Add new apple's model #28

Comments

tucan9389 commented Jul 18, 2019 • edited Loading

Models

Vision

Text

Related Links

tucan9389 commented Jul 18, 2019

tucan9389 commented Jul 27, 2019

FCRN-DepthPrediction Demo

tucan9389 commented Jul 27, 2019

DeeplabV3 Demo

tucan9389 commented Jul 27, 2019

MobileNetV2, Resnet50, SqueezeNet Demo

tucan9389 commented Jul 27, 2019

YOLOv3, YOLOv3-Tiny Demo

tucan9389 commented Jul 27, 2019 • edited Loading

Image Classification

Model Size (MB), Minimum iOS Version

Infernece Time (ms)

Total Time (ms)

FPS

tucan9389 commented Jul 27, 2019 • edited Loading

Object Detection

Model Size (MB), Minimum iOS Version

Infernece Time (ms)

Total Time (ms)

FPS

tucan9389 commented Jul 27, 2019 • edited Loading

Table Convert Script

Input text file

Output text

tucan9389 commented Jul 30, 2019

tucan9389 commented Jul 18, 2019 •

edited

Loading

tucan9389 commented Jul 27, 2019 •

edited

Loading

tucan9389 commented Jul 27, 2019 •

edited

Loading

tucan9389 commented Jul 27, 2019 •

edited

Loading