New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bf16 problem #7365

Open

Zibri opened this issue May 18, 2024 · 1 comment

Labels

question

Zibri commented May 18, 2024

I have a model that has been converted from the original to bf16...
now I want to make some quantization testing with that but quantize says:

cannot dequantize/convert tensor type bf16

I don't understand why, since bf16 and f16 are not that different...

Contributor

jukofyork commented May 20, 2024 •

edited

If you first up cast to fp32 it will probably work as bf16 if just fp32 with 16 of the precision bits truncated:

https://en.m.wikipedia.org/wiki/Bfloat16_floating-point_format

It will just set these extra bits to zero when you up cast and then quanize should work on the fp32 model (I think).

mofosyne added the question label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment