[native_assets] Add support for link hooks

[dcharkes]

This PR adds support invoking link.dart hooks.

Link hooks can add new assets. Link hooks can transform assets sent to link hook from build hooks.

This PR does not yet add support for getting tree-shake information in the link hooks. This is pending on defining the resources.json format (dart-lang/sdk#55494).

Issue:

• hook/link.dart support in flutter_tools #146263

Implementation considerations

The build hooks could be run before Dart compilation and the link hooks after Dart compilation. (This is how it's done in Dart standalone.) However, due to the way the Targets are set up, this would require two targets and serializing and deserializing the BuildResult in between these. This would lead to more code but no benefits. Currently there is nothing that mandates running build hooks before Dart compilation.

Testing

• The unit tests verify that the native_assets_builder link and linkDryRun would be invoked with help of the existing fake.
• The native assets integration test now also invokes an FFI call of a package that adds the asset during the link hook instead of the build hook.
  • In order to keep coverage of the flutter create --template=package_ffi, flutter create is still run and the extra dependency is added and an extra ffi call is added. (Open to alternative suggestions.)

Pre-launch Checklist

• I read the Contributor Guide and followed the process outlined there for submitting PRs.
• I read the Tree Hygiene wiki page, which explains my responsibilities.
• I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
• I signed the CLA.
• I listed at least one issue that this PR fixes in the description above.
• I updated/added relevant documentation (doc comments with ///).
• I added new tests to check the change I am making, or this PR is test-exempt.
• I followed the breaking change policy and added Data Driven Fixes where supported.
• All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

[dcharkes]

@christopherfujino Is there a way to allow arbitrary git dependencies on the CI here?
(If I make a PR upstream to support a PR here, having to merge and publish that upstream dependency before being able to run the CI here means its likely something will be forgotten upstream and we end up having to do more than one PR upstream.)

[christopherfujino]

@christopherfujino Is there a way to allow arbitrary git dependencies on the CI here? (If I make a PR upstream to support a PR here, having to merge and publish that upstream dependency before being able to run the CI here means its likely something will be forgotten upstream and we end up having to do more than one PR upstream.)

You mean, you want to run presubmit tests against the version of a pub package on some random branch on the internet? You should be able to do dependency_overrides with a git dependency in a pubspec.yaml. You may get an analyzer error, but I would expect everything else to Just Work:tm:

[dcharkes]

@christopherfujino Is there a way to allow arbitrary git dependencies on the CI here? (If I make a PR upstream to support a PR here, having to merge and publish that upstream dependency before being able to run the CI here means its likely something will be forgotten upstream and we end up having to do more than one PR upstream.)

You mean, you want to run presubmit tests against the version of a pub package on some random branch on the internet? You should be able to do dependency_overrides with a git dependency in a pubspec.yaml. You may get an analyzer error, but I would expect everything else to Just Work:tm:

👍
I also had to make some hacks in packages/flutter_tools/lib/src/commands/update_packages.dart.
And I had to use the googlesource mirrors instead of github. (We have some allowed hosts list somewhere in the infra but I couldn't find out where.)

Anyway, the CI then looked green enough to hit the publish button on the upstream changes, so the PR now only uses published deps.

[christopherfujino]

@christopherfujino Is there a way to allow arbitrary git dependencies on the CI here? (If I make a PR upstream to support a PR here, having to merge and publish that upstream dependency before being able to run the CI here means its likely something will be forgotten upstream and we end up having to do more than one PR upstream.)

You mean, you want to run presubmit tests against the version of a pub package on some random branch on the internet? You should be able to do dependency_overrides with a git dependency in a pubspec.yaml. You may get an analyzer error, but I would expect everything else to Just Work:tm:

👍 I also had to make some hacks in packages/flutter_tools/lib/src/commands/update_packages.dart. And I had to use the googlesource mirrors instead of github. (We have some allowed hosts list somewhere in the infra but I couldn't find out where.)

Anyway, the CI then looked green enough to hit the publish button on the upstream changes, so the PR now only uses published deps.

🤌

[dcharkes]

All green, ready for review @christopherfujino.
lmk if you want to chat about it.

[christopherfujino]

cc @andrewkolos

[christopherfujino]

Sorry for the delay @dcharkes, I forgot about this. I will take a look today, and if I have outstanding questions I will schedule a 1:1.

[christopherfujino]

why do we need to dry run the link phase?

[dcharkes]

#148474 (comment)

[christopherfujino]

Overall LGTM. I would appreciate a refresher on why the dry runs are needed in general, and specifically for the link phase.

[dcharkes]

Overall LGTM. I would appreciate a refresher on why the dry runs are needed in general, and specifically for the link phase.

Why dry run in flutter run?

• flutter run compiles native assets in dry run has the kernel file with a native assets mapping sitting around unused until the first hot restart. (The native assets mapping is coming from a dry run, it doesn't build the actual assets.)
  which invokes
  • xcodebuild which invokes
    • flutter assemble which compiles Dart to kernel with a native assets mapping and pushes it to the device. This kernel file with native assets mapping is used for hot reloads until the first hot restart. (This native assets mapping is coming from a normal run and will build the native assets.)

Because we want to support building a flutter app from the xcode project, the native assets mapping is needed in flutter assemble.
Because we want to support hot restart, the native assets mapping is also needed in flutter run.

The reason that flutter run does not simply do a native assets build for all target architectures of the current OS is that we would have to block until those builds are done before invoking xcodebuild. Since build hooks and link hooks cannot be invoked for the same configuration concurrently. Dry run is a different configuration then wet run, so in the current set up the dry-runs in flutter run are fired off concurrently to the xcodebuild process call. The fact that we do a kernel compilation in flutter run eagerly instead of on the first hot restart or hot reload request is to make the first hot restart or hot reload fast.

Why run link hooks in dry run?

The build hooks can produce temporary assets (for example a static library) that are linked in a link hook (for example a dynamic library). So in order to embed the correct native assets mapping, we need the output from both hooks. (In dry run both hooks are not expected to actually produce any files, just the information about which files will be produced in a non-dry-run.)

[dcharkes]

Thanks @christopherfujino!