| --- |
| license: mit |
| language: |
| - en |
| tags: |
| - Transformer |
| - Tracking |
| - ONNX |
| --- |
| |
| # MixFormerV2 |
|
|
| This version of MixFormerV2 has been converted to run on the Axera NPU using **w8a16** quantization. |
|
|
| This model has been optimized with the following LoRA: |
|
|
| Compatible with Pulsar2 version: 3.4 |
|
|
| ## Convert tools links: |
|
|
| For those who are interested in model conversion, you can try to export axmodel through |
|
|
| - [The repo of original](https://github.com/MCG-NJU/MixFormerV2) |
|
|
| - [The repo of AXera Platform](https://github.com/Jordan-5i/ax650_mixformer2_demo), which you can get the detial of guide |
|
|
| - [Pulsar2 Link, How to Convert ONNX to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/pulsar2/introduction.html) |
|
|
|
|
| ## Support Platform |
|
|
| - AX650 |
| - [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html) |
| - [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html) |
| - AX630C |
| - [爱芯派2](https://axera-pi-2-docs-cn.readthedocs.io/zh-cn/latest/index.html) |
| - [Module-LLM](https://docs.m5stack.com/zh_CN/module/Module-LLM) |
| - [LLM630 Compute Kit](https://docs.m5stack.com/zh_CN/core/LLM630%20Compute%20Kit) |
| |
| |Chips|npu1| |
| |--|--| |
| |AX650| 11 ms | |
| |AX630C| 33 ms | |
|
|
| ## How to use |
|
|
| Download all files from this repository to the device |
|
|
| ``` |
| root@ax650:/mnt/qtang/MixFormerV2# tree -L 1 |
| . |
| ├── ax650 |
| ├── car.avi |
| ├── config.json |
| ├── onnx |
| ├── README.md |
| ├── run_mixformer2_axmodel.py |
| └── run_mixformer2_onnx.py |
| ``` |
|
|
| ### python env requirement |
|
|
| #### pyaxengine |
|
|
| https://github.com/AXERA-TECH/pyaxengine |
|
|
| ``` |
| wget https://github.com/AXERA-TECH/pyaxengine/releases/download/0.1.1rc0/axengine-0.1.1-py3-none-any.whl |
| pip install axengine-0.1.1-py3-none-any.whl |
| ``` |
|
|
| #### others |
|
|
| ``` |
| pip install argparse numpy opencv-python glob2 |
| ``` |
|
|
| #### Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) |
|
|
| ``` |
| root@ax650:/mnt/qtang/ax650_mixformer2_demo# python3 run_mixformer2_axmodel.py --model-path ax650/mixformer_v2.axmodel --frame-path car.avi -r 10 |
| [INFO] Available providers: ['AxEngineExecutionProvider'] |
| [INFO] Using provider: AxEngineExecutionProvider |
| [INFO] Chip type: ChipType.MC50 |
| [INFO] VNPU type: VNPUType.DISABLED |
| [INFO] Engine version: 2.7.2a |
| [INFO] Model type: 0 (single core) |
| [INFO] Compiler version: 3.4-dirty 4ff37520-dirty |
| ====================type================= [1079, 482] <class 'list'> <class 'list'> |
| 第一帧初始化完毕！ |
| Video: tracking 246.0fps |
| Video: tracking 4.0fps |
| Video: tracking 4.0fps |
| Video: tracking 4.0fps |
| Video: tracking 4.0fps |
| Video: tracking 4.0fps |
| Video: tracking 4.0fps |
| Video: tracking 4.0fps |
| Video: tracking 4.0fps |
| Video: tracking 4.0fps |
| Video: tracking 4.0fps |
| Reached the maximum number of frames (10). Exiting loop. |
| video: average finale average tracking fps 31.8 fps |
| root@ax650:/mnt/qtang/ax650_mixformer2_demo# |
| ``` |
|
|
| #### Inference with M.2 Accelerator card |
|
|
| [What is M.2 Accelerator card?](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html), Show this DEMO based on Raspberry PI 5. |
|
|
| ``` |
| (axcl) axera@raspberrypi:~/samples/MixFormerV2 $ python3 run_mixformer2_axmodel.py --model-path ax650/mixformer_v2.axmodel --frame-path car.avi -r 10 |
| [INFO] Available providers: ['AXCLRTExecutionProvider'] |
| [INFO] Using provider: AXCLRTExecutionProvider |
| [INFO] SOC Name: AX650N |
| [INFO] VNPU type: VNPUType.DISABLED |
| [INFO] Compiler version: 3.4-dirty 4ff37520-dirty |
| ====================type================= [1079, 482] <class 'list'> <class 'list'> |
| 第一帧初始化完毕！ |
| Video: tracking 925.0fps |
| Video: tracking 12.0fps |
| Video: tracking 12.0fps |
| Video: tracking 11.0fps |
| Video: tracking 11.0fps |
| Video: tracking 11.0fps |
| Video: tracking 11.0fps |
| Video: tracking 11.0fps |
| Video: tracking 10.0fps |
| Video: tracking 10.0fps |
| Video: tracking 10.0fps |
| Reached the maximum number of frames (10). Exiting loop. |
| video: average finale average tracking fps 114.9 fps |
| (axcl) axera@raspberrypi:~/samples/MixFormerV2 $ |
| ``` |
