[Feature Request] Share Move libraries between Move blockchain projects

[jolestar]

🚀 Feature Request

Motivation

Now we have several Blockchain projects using Move, but it is hard to share Move libraries between different blockchains.

Perhaps the ideal way of Move project dependency like:

                                     ┌─────────────┐
                                     │Move Std lib │
                                     └─────────────┘


                         ┌─────────────┐          ┌───────────────────┐
                         │Move DeFi lib│          │Move Commons lib   │
                         └─────────────┘          └───────────────────┘



                                                                      ┌───────────────────────────────┐
                                                                      │ Move Chain independent lib    │
                                                                      └───────────────────────────────┘


┌───────────────────┐     ┌───────────────────┐    ┌───────────────────┐   ┌───────────────────┐
│ StarcoinFramework │     │ DiemFramework     │    │ 0lFramework       │   │ PontemFramework   │
└───────────────────┘     └───────────────────┘    └───────────────────┘   └───────────────────┘

1. Move std lib: provide the basic modules for Move language.
2. Move DeFi lib: provide the basic modules for DeFi, such as Token and NFT standards.
3. Move Commons lib: provide some utils module for Move language, like java's apache-commons.
4. Frameworks: the chain-specific modules.

However, this is not yet possible and there is an address conflict issue.

Move stdlib uses the 0x1 address and will conflict with the framework. For example, if StarcoinFramework defines a dependency on Move stdlib in Move.toml, the Move project will produce an error.

One approach to resolve this is to auto-embed Move stdlib and Move DeFi lib to the chain framework, but this also has the problem of naming conflicts.

I think this is very important for building the Move ecosystem, and this is an advantage of Move over Solidity ecosystem.

Is your feature request related to a problem? Please describe.

Pitch

Describe the solution you'd like

Describe alternatives you've considered

Are you willing to open a pull request? (See CONTRIBUTING)

Additional context

[sblackshear]

Thanks for starting this discussion! This is a tough and not-yet-resolved question. I want to actually broaden the topic a bit to discuss Move compatibility in general. There were already some compatibility challenges due to differences in frameworks (e.g., libraries that depend on DiemAccount aren't compatible with ones that depend on StarcoinAccount), and Mysten Move's data model (details to be shared soon) that differs from core Move's global storage will throw another wrench in the works. Some incompatibility is to be expected given the nature of Move (a platform-independent language that gives platform designers a lot of flexibility to customize both the Move runtime and libraries for their platform). But of course, we should still strive for as much compatibility as possible and suggest best practices that will help with this.

Some early thoughts on this:

• Compatibility should be a source/package level concern. Striving for compatibility at the bytecode level across different networks is a much harder task, and it's not clear that it's needed (and thus, I think is a task that we should not attempt). There are packages that can be 100% compatible at the source level, but incompatible at the bytecode level due to hardcoded addresses, address length, and in the future, possibly other factors like varying the serialization format ([Feature Request] Make VM parametric in serialization format #28) However, none of these are blockers for source compatibility, at least not in principle.
• An obvious best practice is to suggest not writing "non-portable" code that depends on address length or serialization format. This is very similar to (e.g.) not writing C code that depends on the pointer size.
• We can suggest "tiering" modules based on how likely they are to be reusable, and encourage packages to only group modules in the same tier. Otherwise, someone might want to depend on some reusable module in a package, but be blocked by some incompatible (but unused) module in the same package. Some natural tiers that come to mind are:

1. Modules that only define utility functions and do not define any types (e.g., a Math.move module). They must (transitively) only depend on other tier 1 packages. These are fully generic and can be used by any platform.
2. Modules that define utility functions and expose utility types (e.g., ASCII::String in the stdlib). They must (transitively) only depend on other tier 1-2 packages. These can be used by any platform in principle, but there's nothing that stops two platforms from (e.g.) defining and reusing type-incompatible versions of UTF8::String. We should try to avoid this by working as a Move community to build canonical libraries of these types.
3. Same as (2), but also has some native functions. They must (transitively) only depend on other tier 1-3 packages. These can be used by any platform in principle, but some platforms might not want tto use the native functions defined by some other platform because native functions can violate the rules of core Move (e.g., by creating addresses from scratch).
4. Same as (3), but also has entrypoints. Different platforms can (and will) choose different rules for entrypoints that may not be compatible (see [Feature Request] Move signature enforcement for script functions from VM -> adapter(s) #76). These must (transitively) only depend on other tier 1-4 packages. These are only reusable across platforms with the same entrypoint structure.
5. Same as (4), but also uses global storage (e.g., borrow_global, tables once we have them, or ). These must (transitively) only depend on other tier 1-5 packages. These are only reusable across platforms with the same entrypoint structure and global storage.
6. Same as (5), but also uses platform-specific packages/types (e.g. accounts, transactions, time, validator set management, ...). These are not likely to be compatible with other packages. Perhaps over time, we can define source-level "interfaces" for some of these concepts to allow compatibility for similar platforms, but I wouldn't want to rush into this. Some level of incompatibility is always going to be expected here.

I'm not sure whether this tiering system should just be an informal guideline, or also something that we try to enforce progammatically in the package system and/or linters. It's worth noting that packages are not organized according to this tiering system today, although some are close (e.g., there's very little global storage used in the Move stdlib). But the overwhelming tendency is toward monolithic tier 6 packages, which I think we should change.

Once we have reorganized packages using these principles, I think there will be a lot more boxes in your diagram above, but the tiering system will make it easier to figure out where we can/should draw dependency arrows and where we shouldn't.