Dynamic storage

[xgreenx]

Right now, storage is done in 32 byte slots. While the store / read API is simple, for basic objects like a struct, it creates many indexes which have to each be merkalized. This becomes a very costly endeavor.
It would be far simpler to have a storage API that is dynamic, example OPCodes below:

sstore(REG_KEY_IN_MEM[0, 32], REG_READ_MEM_START, --, IMM_VALUE_LENGTH)
sget(REG_KEY_IN_MEM[0, 32], REG_WRITE_MEM_START)

This way if we wanted to store a struct for example, we could just use one STORE operation to do so VS many.
One issue with this is knowing how much memory you will need to allocate for getting values from storage, but we can have something like:

• S_STORE key_addr data_ptr data_len stores MEM[data_ptr, data_len] bytes to storage slot MEM[key_addr, 32]
• S_SIZE dst_reg key_addr loads size of storage array to dst_reg
• S_LOAD key_addr dst_addr offset len copies STORAGE[MEM[key_addr, 32]][offset, len] to MEM[dst_addr, len] S_LOAD_STACK key_addr offset len allocates len bytes of stack space and copies STORAGE[MEM[key_addr, 32]][offset, len] to it
• S_LOAD_HEAP key_addr offset len allocates len bytes of heap space and copies STORAGE[MEM[key_addr, 32]][offset, len] to it
• (the last two can of course be done using CFE/ALOC and are just possible optimizations)

Also if we want partial key updates, we could have some of the following operations

• S_PARTIAL_WRITE key_addr ptr offset len copies MEM[ptr, len] bytes to STORAGE[MEM[key_addr, 32]][offset, len] updating it partially. We could also allow exceeding the length of the original value, allowing this to extend a key
• S_TRUNCATE key_addr len Truncate storage value to len bytes
• S_DUPLICATE key1_addr key2_addr Duplicate the contents of a storage slot without having to copy in into the vm memory

We could also have variants of the operations with immediate-encoded length, since those are likely common operations.

[Voxelot]

Will S_SIZE incur similar overhead to loading the entire data from rocksdb? If we always have to use S_SIZE in conjunction with S_LOAD, it seems like we may double the overhead.

What about S_LOAD* opcodes that allocate and update the memory dynamically? This way we only need to do a single db read.

I.e.
S_LOAD_STACK key_addr dynamically allocates L = len(STORAGE[MEM[key_addr, 32]]) bytes of stack space and copies MEM[SP, L] = STORAGE[MEM[key_addr, 32]][0, L]

Also, what is the imagined usecase for partial reads and writes? If it's used to manage things like a Vec within a single storage blob, would that incur more overhead than slots?

[MitchTurner]

Is this something we want before mainnet @Voxelot @xgreenx ? If not, we should probably remove this from the Fuel Project Board