Skip to content

Releases: intel/auto-round

v0.4

22 Nov 13:32
@wenhuach21 wenhuach21
v0.4
f7913f9
Compare
Choose a tag to compare
v0.4 Latest
Latest

Highlights

[Experimental Feature] We provide API support for VLM models
[Kernel] We add ipex support for intel cpu
[Bug fix] We fix tuning bug for glm4 model
[Enhancement] better align gradient_accumulate_steps behavior for varied length input

What's Changed

Full Changelog: v0.3.1...v0.4

Contributors

yiliu30, wenhuach21, and 3 other contributors
Assets 2
Loading

Intel® auto-round v0.3.1 Release

21 Oct 04:12
@wenhuach21 wenhuach21
v0.3.1
0c4319c
Compare
Choose a tag to compare
Intel® auto-round v0.3.1 Release

Release Highlights:

New Features:

Full-Range Symmetric Quantization: We’ve introduced full-range symmetric quantization, which often matches or even exceeds the performance of asymmetric quantization, especially at lower bit widths, such as 2.

Command-Line Support: You can now quantize models using the command auto-round --model xxx --format xxx

Default Exporting Format Change: The default format has been updated to auto_round instead of auto_gptq.

Muiti-thread packing: up to 2X speed up on packing phase

Bug Fixes:

Resolved Missing Cached Position Embeddings: Fixed an issue with missing cached position embeddings in Transformer version 4.45.2.

Mutable Default Values Issue: Addressed problems related to mutable default values.

3 bit packing bug for AutoGPTQ format

What's Changed

New Contributors

Full Changelog: v0.3...v0.3.1

Contributors

dependabot, wenhuach21, and 4 other contributors
Assets 2
Loading

Intel® auto-round v0.3 Release

14 Aug 11:33
@wenhuach21 wenhuach21
v0.3
4ac1104
Compare
Choose a tag to compare
Intel® auto-round v0.3 Release

  • Highlights:

    • Broader Device Support:
      • Expanded support for CPU, HPU, and CUDA inference in the AutoRound format, resolving the 2-bit accuracy issue.
    • New Recipes and Model Releases:
    • Experimental Features:
      • Introduced several experimental features, including activation quantization and mx_fp, with promising outcomes with AutoRound.
    • Multimodal Model Support:
      • Extended capabilities for tuning and inference across several multimodal models.

    Lowlights:

    • Implemented support for low_cpu_mem_usage, auto_awq format, calibration dataset concatenation, and calibration datasets with chat templates.
Assets 2
Loading

Intel® auto-round v0.2 Release

30 May 02:13
@wenhuach21 wenhuach21
v0.2
aafb82e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
Intel® auto-round v0.2 Release

Overview

We supported the Intel XPU format and implemented lm-head quantization and inference, reducing the model size from 5.4GB to 4.7GB for LLAMA3 at W4G128. Additionally, we supported both local and mixed online datasets for calibration. By optimizing memory usage and tuning costs, the calibration process now takes approximately 20 minutes for 7B models and 2.5 hours for 70B models with 512 samples by setting disable_low_gpu_mem_usage.

Others:

More accuracy data as presented in [paper](https://arxiv.org/pdf/2309.05516) and [low_bit_open_llm_leaderboard](https://huggingface.co/spaces/Intel/low_bit_open_llm_leaderboard)

More technical details as presented in [paper](https://arxiv.org/pdf/2309.05516)

Known issues:

Large discrepancy between gptq model and qdq model for asymmetric quantization in some scenarios. We are working on it.

Assets 2
Loading

Intel® auto-round v0.1 Release

08 Mar 08:11
@wenhuach21 wenhuach21
v0.1
514aa49
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
Intel® auto-round v0.1 Release

Overview

AutoRound introduces an innovative weight-only quantization algorithm designed specifically for low-bit LLM inference, approaching near-lossless compression for a range of popular models including gemma-7B, Mistral-7b, Mixtral-8x7B-v0.1, Mixtral-8x7B-Instruct-v0.1, Phi2, LLAMA2 and more at W4G128. AutoRound consistently outperforms established methods across the majority of scenarios at W4G128, W4G-1, W3G128, and W2G128 .

Key Features

  • Wide Model Support: AutoRound caters to a diverse range of model families. About 20 model families have been verified.
  • Export Flexibility: Effortlessly export quantized models to ITREX[1] and AutoGPTQ[2] formats for seamless deployment on Intel CPU and Nvidia GPU platforms respectively.
  • Device Compatibility: Compatible with tuning devices including Intel CPUs, Intel Guadi2, and Nvidia GPUs.
  • Dataset Flexibility: AutoRound supports calibration with Pile10k and MBPP datasets, with easy extensibility to incorporate additional datasets.

Examples

  • Explore language modeling and code generation examples to unlock the full potential of AutoRound.

Additional Benefits

  • PreQuantized Models: Access a variety of pre-quantized models on Hugging Face for immediate integration into your projects, with more models under review and coming soon.
  • Comprehensive Accuracy Data: Simplified user deployment with extensive accuracy data provided.

Known issues:

  • baichuan-inc/Baichuan2-13B-Chat has some issues, we will support it soon

Reference:

[1] https://github.com/intel/intel-extension-for-transformers

[2] https://github.com/AutoGPTQ/AutoGPTQ

Assets 2
Loading