feature: zero-copy for accounts with resizable lists

[ckamm]

This proposal is about adding support for resizable lists in zero-copy accounts. It comes from existing unusual anchor use in the Mango program and the needs of OpenBook v2.

Resizing accounts and Vec<T> fields in accounts work well with the usual borsh serialization but not with zero-copy. Often zero-copy is essential to reduce compute costs in programs.

Wants

• accounts can store multiple lists of zero-copy values
• access to (potentially mut) slices of those values
• functions to change the size of these lists and deal with account resizing
• retain zero-copy access to fixed-offset fields in an account
• account layout stays compatible with current idl and borsh serialization

Why

• zero copy access needed for low compute use
• benefit to standardizing how dynamic parts of those accounts work
• MangoAccounts want to store a variable number of token positions, openbook open orders, perp market positions, trigger orders etc. Users want to be able to grow their accounts when necessary.
• OpenBook orderbooks and Mango perp orderbooks want to store a variable number of order slots and the number of these slots can be large. Market creators want to start with a low number to save on rent and extend if the market gains traction.
• compatibility with existing IDL and borsh allows for easy integration with libraries that support anchor and the solana explorer

Example

This doesn't intend to define a concrete API, just wants to show what API could look like.

#[zero_copy]
struct Foo {
  data: u64,
}

#[zero_copy]
struct MyAccountFixedOffsetData {
  field: u64,
}

#[zero_copy_dynamic]
struct MyAccountDynamic {
  // possibly this will have a different type or some annotation
  foo_list: Vec<Foo>,
  
  // note how the offset of the length of the second list depends on the size
  // of the first list
  int_list: Vec<u64>,
  
  // probably static fields are forbidden: no static offsets
}

#[account(zero_copy_extendable)]
struct MyAccount {
  fixed: MyAccountFixedOffsetData,
  dynamic: MyAccountDynamic,
}

// MyAccount's IDL and binary layout should be identical to
#[account]
struct MyAccountBorsh {
  field: u64,
  foo_list: Vec<Foo>,
  int_list: Vec<u64>,
}

fn usage() {
  // ctx.accounts.myacc has type AccountLoader<MyAccount>
  let acc: &MyAccount = ctx.accounts.myacc.load()?;
  
  // direct access as before
  let a = acc.fixed.field;
  
  // can read slices
  let l1: &[Foo] = acc.dynamic.foo_list_slice();
  let l2: &[u64] = acc.dynamic.int_list_slice();

  let acc = ctx.accounts.myacc.load_mut()?;
  
  // resize the list, moving later data up or down as needed
  // this might need to realloc the account to create more space; possibly this is a separate step before
  acc.dynamic.resize_foo_list(15); 
}

[acheroncrypto]

This would certainly be a great improvement over the current sized one.

Regarding the example API you shared, why is it necessary for the user to differentiate between zero_copy and zero_copy_dynamic?

[nabeel99]

@acheroncrypto just curious here on the impl of zero-copy for heap allocated types and how is it even possible, isnt anchor current approach to zero-copy is via bytemuck and its pod traits which strictly is not impl for type such as a Vec since it is not a PODc, if so how would zero-copy on growable lists work then ?

[nabeel99]

@ckamm I remember solana having something similar implement, with the account resizing feature isnt the resulting solution for the above feature going to be what this is : https://github.com/solana-labs/solana-program-library/blob/master/stake-pool/program/src/big_vec.rs

[ckamm]

@acheroncrypto In the example, MyAccountDynamic is just describing data, it's not literally a zero-copy view of account bytes. That's because the Vec's heap memory and also I think borsh serializes the vec size as 4 bytes, not 8.

One path is to add more magic and to somehow allow Vec<> fields in zero_copy annotated structs, but I'd say it's saner to keep that for the case where the struct does indeed describe the account bytes exactly.

@nabeel99 That big_vec has some similarities, but is not sufficient.

• while their .retain() has the memmoves to shuffle the entry bytes around, that's not sufficient when you have two of these dynamically-sized lists in an account: imagine you shrink the first list, then all bytes from the second list need to move even though the second list wasn't touched.
• it seems to deal with a heterogenous list of borsh-serializable types - we're interested in a homogenous list and want things like indexing to get a &T
• I haven't read their code in detail, but it's important to not use borsh (de-)serialization at all usually