-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Handling materialization of lazy arrays #748
Comments
I think this topic will have to be addressed in v2024, as it's too big to be squeezed in v2023 which we're trying very hard to wrap up 😅 |
A few quick comments:
|
No pressure. 😉
Thanks Ralf -- That'd be a big help indeed. Materializing an entire array as opposed to one element is something that should be a common API across libraries, IMHO, I changed the title to reflect that. |
Cross linking #728 as it may be relevant to this discussion. |
Background
Some colleagues and me were doing some work on
sparse
when we stumbled onto a limitation of the current Array API Standard, and @kgryte was kind enough to point out that it might have some wider implications than justsparse
, so it would be prudent to discuss it with other relevant parties within the community before settling on an API design to avoid fragmentation.Problem Statement
There are two notable things missing from the Array API standard today, which
sparse
, and potentially Dask, JAX and other relevant libraries might also need.sparse
, this would be the format of the sparse array (CRS
,CCS
,COO
, ...).sparse
/JAX might use this to build up kernels before running a computationPotential solutions
Overload the
Array.device
attribute and theArray.to_device
method.One option is to overload the objects returned/accepted by these to contain a device + storage object. Something like the following:
To materialize an array, one could use
to_device(default_device())
(possible after #689 is merged).Advantages
As far as I can see, it's compatible with how the Array API standard works today.
Disadvantages
We're mixing the concepts of an execution context and storage format, and in particular overloading operators in a rather weird way.
Introduce an
Array.format
attribute andArray.to_format
method.Advantages
We can get the API right, maybe even introduce
xp.can_mix_formats(...)
.Disadvantages
Would need to wait till the 2024 revision of the standard at least.
Tagging potentially interested parties:
The text was updated successfully, but these errors were encountered: