Update dependency ai.djl.huggingface:tokenizers to v0.31.0 #135

renovate · 2023-07-12T17:50:00Z

This PR contains the following updates:

Package	Change	Age	Adoption	Passing	Confidence
ai.djl.huggingface:tokenizers (source)	`0.22.1` -> `0.31.0`

Release Notes

deepjavalibrary/djl (ai.djl.huggingface:tokenizers)

Key Changes

Engine Updates:
- OnnxRuntime 1.19.0 https://github.com/deepjavalibrary/djl/pull/3446
- Huggingface Tokenizers 0.20.0 https://github.com/deepjavalibrary/djl/pull/3452
Added mask generation task for SAM2 model https://github.com/deepjavalibrary/djl/pull/3450
Text Embedding Inference:
- Added Mistral, Qwen2, GTE, Camembert embedding model support
- Added reranker model support

Enhancement

[api] Avoid non-ascii characters by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3395
[djl-converter] Exit with error if convert model failed by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3399
[api] Support TEI input format to reranking model by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3400
[rust] Adds sigmoid and softmax operator for Rust engine by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3407
[test] Detect GPUs with specified engine by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3409
[api] Adds Criteria.isDownload() api by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3403
[rust] Build .so file for each cuda arch by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3410
[rust] Add mistral embedding model by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3412
[tokenizers] Add supported arch in djl-convert by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3416
[tokenizers] Replace pt file names to safetensors by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3417
[rust] Load model on given device by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3419
[rust] Add qwen2 model by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3420
[rust] Support pre-downloaded rust shared library by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3421
[pytorch] Adds pad operator by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3423
[rust] Provides better error message for unsupported ops by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3424
[api] Adds center fit image operation for Yolo by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3425
[rust] Add GTE and Gemma2 model by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3422
[djl-convert] Sets default max model size limit for importing by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3428
[djl-import] Includes requires version when importing model by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3431
[android] Upgrade DJL version to 0.30.0 by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3432
[rust] Make cublaslt wrapper non static by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3434
[djl-convert] Exclude models in includeTokenTypes by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3435
[rust] Make tensor contiguous in rotary embedding by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3436
[rust] Allows -1 dim for normalize() by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3442
Refactored Identifiers by @congyuluo in https://github.com/deepjavalibrary/djl/pull/3381
[rust] Adds text classification models to Rust model zoo by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3444
[examples] Adds segment anything 2 example by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3449
[api] Refactor ImageFeatureExtractor by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3455
[api] Adds base64 image support for ImageTranslator by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3456
[djl-import] Improve model import speed by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3457
[api] Updates dependencies version to latest by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3454
[api] Optimized text embedding post processing performance by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3459
add drawMarks to android BitMapImageFactory by @sindhuvahinis in https://github.com/deepjavalibrary/djl/pull/3460
[ci] moving to temporary iam credentials for publishing steps by @siddvenk in https://github.com/deepjavalibrary/djl/pull/3462
[OnnxRuntime] Update debug log message by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3463
Increase DJL version to 0.30.0 by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3465
[examples] Adds gradle tasks for each example by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3466
Upgrade dependency versions by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3467
[tokenizers] Converting encoding to int32 NDList by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3468

Bug Fixes

[api] Fixes logging calling convention by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3394
[djl-converter] Fixes import text embedding model from local folder by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3388
[djl-converter] Fixes djl-convert command line return code by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3406
[rust] Fix camembert and distilbert model loading by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3415
[rust] Fix camembert model loading by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3418
[rust] Fixes memory leak by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3433
[djl-convert] Fixes huggingface model converter by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3440
[rust] Fix bert model classifier loading by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3441
[xgb] Fixes alternative NDArray conversion issue by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3453
[djl-import] Fixes missing arguments for onnx import by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3458
[ci][fix] use v2 for aws credentials due to glib issues with node 20 by @siddvenk in https://github.com/deepjavalibrary/djl/pull/3464

Documentation

[examples] Moves nlp examples into nlp folder by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3393
[docs] Build versions.json before mike deploy by @Varun-Dutta in https://github.com/deepjavalibrary/djl/pull/3392
[example] Enable PyTorch for some training example by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3398
[docs] Updates docs website url by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3404
[docs] Fixes broken links in markdown files. by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3408
[djl-import] Fixes missing trust-remote-code arg for import model zoo by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3427
[docs] Updates trace whisper model document by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3426
[tensorflow] Updates tensorflow document by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3430
[docs] Adds segment anything document by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3451

CI/CD

[ci] Fixes serving publish for awscurl release version by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3411
[ci] Remove no_response workflow by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3429

Full Changelog: deepjavalibrary/djl@v0.29.0...v0.30.0

`v0.29.0`: DJL v0.29.0 Release

Compare Source

Key Changes

Upgrades for engines
- Upgrades PyTorch engine to 2.3.1
- Upgrades TensorFlow engine to 2.16.1
- Introduces Rust engine CUDA support
- Upgrades OnnxRuntime version to 1.18.0 and added CUDA 12.4 support
- Upgrades javacpp version to 1.5.10
- Upgrades HuggingFace tokenizer to 0.19.1
- Fixes several issues for LightGBM engine
- Deprecated llamacpp engine
Enhancements for engines and API
- Adds Yolov8 segmentation and pose detection support
- Adds metric type to Metic class
- Improves drawJoints and drawMask behavior for CV model
- Improves HuggingFace model importing and conversion tool
- Improves HuggingFace NLP model batch inference performance
- Adds built-in ONNX extension support
- Adds several NDArray operators in PyTorch engine
- Adds fp16 and bf16 support for OnnxRuntime engine
- Adds CrossEncoder support for NLP models

Enhancements

Adds metric type to Metic class by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3244
Improves drawJoints behavior by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3305
[api] Allows to control json pretty print with env var by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3288
[api] Avoid null dimensions for Metric by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3246
[api] Improve NDArray.toDebugString() output by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3290
[api] Loads native engine in deterministic order by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3300
[api] Refactor drawMask() for instance segmentation by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3304
[api] Refactor nms for yolo translator by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3297
add close method to all nd manager by @lanking520 in https://github.com/deepjavalibrary/djl/pull/3225
ported tools/stats.gradle by @elect86 in https://github.com/deepjavalibrary/djl/pull/3219
use standard GSON output by @lanking520 in https://github.com/deepjavalibrary/djl/pull/3284
[enhancement] Optimize memory copy overhead to enhance performance. by @ewan0x79 in https://github.com/deepjavalibrary/djl/pull/3289
Gradle Kotlin script plus other stuff by @elect86 in https://github.com/deepjavalibrary/djl/pull/3167
Improved incremental build by @benjie332 in https://github.com/deepjavalibrary/djl/pull/3231
Refactored Identifiers by @congyuluo in https://github.com/deepjavalibrary/djl/pull/3276
Refactored Identifiers by @congyuluo in https://github.com/deepjavalibrary/djl/pull/3282
[gradle] Remove unused gradle files by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3280
[jacoco] exclude spark extension since it doesnot contain test by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3230
[Lgbm] support multi classification by @ewan0x79 in https://github.com/deepjavalibrary/djl/pull/3234
[Lgbm] support multi type prediction by @ewan0x79 in https://github.com/deepjavalibrary/djl/pull/3237
[llamacpp] Removing llamacpp support in DJL by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3312
[mxnet-model-zoo] Adds missing translatorFactory in metadata by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3279
[onnx] Adds fp16 and bfp16 support for OnnxRuntime by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3281
[onnxruntime] Add debug message for OnnxRuntime by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3217
[onnxruntime] Adds yolov8n pose model for OnnxRuntime by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3309
[onnxruntime] Adds yolov8n-seg model to onnxruntime model zoo by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3310
[onnxruntime] Load onnx extenstion if available by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3333
[pytorch] Adds Yolov8n-seg model to model zoo by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3308
[pytorch] Adds back PyTorch 2.1.2 support by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3285
[pytorch] Adds yolov8n pose estimation model by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3298
[pytorch] Implements gammaln operator for PyTorch by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3262
[pytorch] Split maven publish into two parts by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3273
[rust] Add tokenizer cuda build workflow by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3322
[rust] Allows -2 as dims for sum() by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3221
[rust] Change loging level to debug by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3336
[rust] Download cu124 jni library for cuda by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3327
[rust] Remove 0-dimension tensor compare in NDArrayTests by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3320
[rust] Update gpu build pipeline to cu122 by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3334
[rust] Upgrade candle version by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3248
[rust] Use fused layer by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3260
[spark] Do not support model_url by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3224
[spark] Update dependency versions by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3241
[spark] Updates spark version to 3.5.1 by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3240
[spark] Use batch predict API by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3242
[text-embedding] Remove CrossEncoderTranslatorFactory in favor of TextEmbeddingTranslatorFactory by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3239
[tokenizer] Adds maxos-13 support back by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3328
[tokenizer] Ensure GPU is used in TextEmbeddingTranslator by @david-sitsky in https://github.com/deepjavalibrary/djl/pull/3212
[tokenizer] Process text embedding input and output in stacked NDArray by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3213
[tokenizer] Recover accidentally deleted file by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3311
[tokenizer] Supports cross encoder for text classification model by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3338
[tokenizers] Download jni lib files for cuda by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3326

Bug Fixes

[api] Fix unitest in GPU docker running on CPU case by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3228
[api] Fixes IdEmbedding memory leak by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3257
[api] Fixes nightly tests on GPU machine by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3302
[api] Fixes unitest by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3210
[fix] fix lgbm bytebuffer native order by @ewan0x79 in https://github.com/deepjavalibrary/djl/pull/3258
Fix Application.of missing some applications by @tadayosi in https://github.com/deepjavalibrary/djl/pull/3277
[mxnet] Fixes GloveWordEmbeddingTranslator bug by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3287
[pytorch-model-zoo]: fix PtSsdTranslator.Builder.self() by @eversnarf in https://github.com/deepjavalibrary/djl/pull/3204
[pytorch] Fixes PyTorch 2.3.1 windows dependencies by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3269
[pytorch] Fixes PyTorch 2.3.1 windows dependencies by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3270
[pytorch] Fixes uploadS3 gradle task by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3263
[rust] Fix NDArrayTests failure on cuda by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3319
[rust] Fix deleteModel error by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3229
[rust] Fix output tensor dtype by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3249
[rust] Fix tokenizer cuda pipeline name by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3325
[rust] Fixes test failure on GPU by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3301
[timeseries] Fixes contentLength issue for inference by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3306
[timeseries] Fixes duration format issue by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3307
[tensorrt] Fixes gradle biuld script by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3253
[tokenizer] Fixes detect include token type logic by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3318
[tokenizer] Fixes tokenizer build workflow by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3323
[tokenizers] Fixes huggingface build for Windows by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3330
[tokenizers] Fixes memory leak when there is overflowing tokens by @baldersheim in https://github.com/deepjavalibrary/djl/pull/3317
[xgb] Fixes gradle build script by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3254

Documentation

[doc] add output formatter schema to LMI docs.djl.ai by @sindhuvahinis in https://github.com/deepjavalibrary/djl/pull/3268
[doc] add release notes to docs.djl.ai by @sindhuvahinis in https://github.com/deepjavalibrary/djl/pull/3266
[docs] Bump up DJL version to 0.28.0 by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3247
[docs] Update example reference by @emmanuel-ferdman in https://github.com/deepjavalibrary/djl/pull/3275
[docs] add dark theme and fixed broken link by @Varun-Dutta in https://github.com/deepjavalibrary/djl/pull/3295
[example] Adds PyTorch action recognition model to model zoo by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3292
[examples] Enabled training unit tests on macOS M1 by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3256
[examples] Fixes ObjectDetection example for macOS m1 by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3206
[examples] Fixes nightly build failure on Windows by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3267
[examples] Remove symbolic training for MXNet by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3299
Update README.md by @bradh in https://github.com/deepjavalibrary/djl/pull/3200

CI/CD

[android] Updates android with PyTorch 2.2.2 by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3236
[api] Updates slf4j version to 2.0.13 by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3329
[bom] Uses release version for tensorflow by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3313
[ci] Disable github actions runner for non-djl repo by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3220
[ci] Fixes nightly publish for nodejs20 issue by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3314
[ci] Fixes publish maven native package by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3264
[ci] Fixes pytorch JNI build by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3339
[ci] Fixes rust jni build for nodejs20 issue by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3315
[ci] Fixes serving nightly publish by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3245
[ci] Fixes windows pytoch jni build by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3209
[ci] Minor github action workflow changes by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3331
[ci] Remove fastertransformer build workflow by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3291
[ci] Update to amazon-ecr-login@v2 by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3250
[ci] Updates OnnxRuntime to 1.18.0 by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3235
[ci] Updates dependency versions to latest by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3332
[ci] Updates spotbugs to 6.0.15 by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3233
[djl] update djl version and readmes by @tosterberg in https://github.com/deepjavalibrary/djl/pull/3202
[MCM 0.29.0] Remove -SNAPSHOT for release v0.29.0 by @ydm-amazon in https://github.com/deepjavalibrary/djl/pull/3345
Add more test logging by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3321
[pytorch] Updates PyTorch to 2.3.1 by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3265
[release] Bump up versions to 0.29.0 in documents to point to new url by @ydm-amazon in https://github.com/deepjavalibrary/djl/pull/3344
[tensorflow] Updates tensorflow to 2.16.1 by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3283

New Contributors

@bradh made their first contribution in https://github.com/deepjavalibrary/djl/pull/3200
@eversnarf made their first contribution in https://github.com/deepjavalibrary/djl/pull/3204
@benjie332 made their first contribution in https://github.com/deepjavalibrary/djl/pull/3231
@emmanuel-ferdman made their first contribution in https://github.com/deepjavalibrary/djl/pull/3275
@tadayosi made their first contribution in https://github.com/deepjavalibrary/djl/pull/3277
@congyuluo made their first contribution in https://github.com/deepjavalibrary/djl/pull/3276
@Varun-Dutta made their first contribution in https://github.com/deepjavalibrary/djl/pull/3295
@baldersheim made their first contribution in https://github.com/deepjavalibrary/djl/pull/3317

Full Changelog: deepjavalibrary/djl@v0.28.0...v0.29.0

`v0.28.0`: DJL v0.28.0 Release

Compare Source

Key Changes

Upgrades for engines
- PyTorch 2.2.2 https://github.com/deepjavalibrary/djl/pull/3155
- Sentencepiece 0.2.0 https://github.com/deepjavalibrary/djl/pull/3163
Enhancements for engines and API
- Adds experimental Rust engine https://github.com/deepjavalibrary/djl/pull/3078

Enhancement

[api] Automatically detect translatorFactory based on task by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3136
[api] Adds OnesBlockFactory to make it easy for testing by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3140
Ensure the alternative ND manager can use GPUs by @david-sitsky in https://github.com/deepjavalibrary/djl/pull/3138
[api] Tries to use the same device for alternative NDManager by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3146
[api] Supports serialize NaN in json by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3156
[rust] Add rust engine implemenation by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3078
[rust] Adds Rust model zoo by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3132
[rust] Support load DJL model for RsModel by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3147
[rust] RsModel delete model in close by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3170
[tokenizers] Updates tokenizer to 0.19.1 by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3143
[tokenizer] Allows use HF_TOKEN to access gated model by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3150
[tokenizers] Create djl_converter package by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3172
[tokenizer] Refactor djl_convert python code by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3179
Updates on djl_converter by @xyang16 in https://github.com/deepjavalibrary/djl/pull/3187
[pytorch] Updates PyTorch to 2.2.2 by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3155
[pytorch] Update PyTorch engine README for version 2.2.2 by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3165
[pytorch] optimize memory copy cost for pytorch NDArray by @ewan0x79 in https://github.com/deepjavalibrary/djl/pull/3137
[pytorch] Updates PyTorch to 2.3.0 by @frankfliu in https://github.com/deepjavalibrary/djl/pull/3192
[sentencepiece] Updates sentencepiece to 0.2.0 by @frankfliu in [https://github

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

If you want to rebase/retry this PR, check this box

This PR was generated by Mend Renovate. View the repository job log.

renovate bot force-pushed the renovate/ai.djl.huggingface-tokenizers-0.x branch from 5836f76 to 431d198 Compare October 4, 2023 05:38

renovate bot changed the title ~~Update dependency ai.djl.huggingface:tokenizers to v0.23.0~~ Update dependency ai.djl.huggingface:tokenizers to v0.24.0 Oct 4, 2023

renovate bot force-pushed the renovate/ai.djl.huggingface-tokenizers-0.x branch from 431d198 to 1684e15 Compare November 6, 2023 05:50

renovate bot changed the title ~~Update dependency ai.djl.huggingface:tokenizers to v0.24.0~~ Update dependency ai.djl.huggingface:tokenizers to v0.25.0 Nov 6, 2023

renovate bot force-pushed the renovate/ai.djl.huggingface-tokenizers-0.x branch from 1684e15 to d2d1562 Compare January 12, 2024 02:50

renovate bot changed the title ~~Update dependency ai.djl.huggingface:tokenizers to v0.25.0~~ Update dependency ai.djl.huggingface:tokenizers to v0.26.0 Jan 12, 2024

renovate bot force-pushed the renovate/ai.djl.huggingface-tokenizers-0.x branch from d2d1562 to a33fd18 Compare March 29, 2024 20:51

renovate bot changed the title ~~Update dependency ai.djl.huggingface:tokenizers to v0.26.0~~ Update dependency ai.djl.huggingface:tokenizers to v0.27.0 Mar 29, 2024

renovate bot force-pushed the renovate/ai.djl.huggingface-tokenizers-0.x branch from a33fd18 to d3d7d04 Compare May 23, 2024 21:00

renovate bot changed the title ~~Update dependency ai.djl.huggingface:tokenizers to v0.27.0~~ Update dependency ai.djl.huggingface:tokenizers to v0.28.0 May 23, 2024

renovate bot force-pushed the renovate/ai.djl.huggingface-tokenizers-0.x branch from d3d7d04 to ab14fcc Compare July 26, 2024 17:58

renovate bot changed the title ~~Update dependency ai.djl.huggingface:tokenizers to v0.28.0~~ Update dependency ai.djl.huggingface:tokenizers to v0.29.0 Jul 26, 2024

renovate bot force-pushed the renovate/ai.djl.huggingface-tokenizers-0.x branch from ab14fcc to 90343ea Compare September 15, 2024 11:48

renovate bot changed the title ~~Update dependency ai.djl.huggingface:tokenizers to v0.29.0~~ Update dependency ai.djl.huggingface:tokenizers to v0.30.0 Sep 15, 2024

Update dependency ai.djl.huggingface:tokenizers to v0.31.0

af29336

renovate bot force-pushed the renovate/ai.djl.huggingface-tokenizers-0.x branch from 90343ea to af29336 Compare November 24, 2024 12:15

renovate bot changed the title ~~Update dependency ai.djl.huggingface:tokenizers to v0.30.0~~ Update dependency ai.djl.huggingface:tokenizers to v0.31.0 Nov 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update dependency ai.djl.huggingface:tokenizers to v0.31.0 #135

Update dependency ai.djl.huggingface:tokenizers to v0.31.0 #135

renovate bot commented Jul 12, 2023 •

edited

Loading

Update dependency ai.djl.huggingface:tokenizers to v0.31.0 #135

Are you sure you want to change the base?

Update dependency ai.djl.huggingface:tokenizers to v0.31.0 #135

Conversation

renovate bot commented Jul 12, 2023 • edited Loading

Release Notes

v0.31.0: DJL v0.31.0 Release

Key Changes

Enhancements

Bug Fixes

Documentation

CI/CD

New Contributors

v0.30.0: DJL v0.30.0 Release

Key Changes

Enhancement

Bug Fixes

Documentation

CI/CD

v0.29.0: DJL v0.29.0 Release

Key Changes

Enhancements

Bug Fixes

Documentation

CI/CD

New Contributors

v0.28.0: DJL v0.28.0 Release

Key Changes

Enhancement

Configuration

renovate bot commented Jul 12, 2023 •

edited

Loading

`v0.31.0`: DJL v0.31.0 Release

`v0.30.0`: DJL v0.30.0 Release

`v0.29.0`: DJL v0.29.0 Release

`v0.28.0`: DJL v0.28.0 Release