Consolidate and Automate tutorial testing

[critesjosh]

The following is from Rahul:

There are several issues with our tutorials today:

• Brittle: it is easy for them to break and we wouldn't even realize
• Hard to test on CI
• A dev following tutorial has to keep copy pasting so much code - breaks their flow. I have heard from several high quality devs that they dislike this and instead just switch to the monorepo.
• So much work for us to build and maintain them

Potential solution:
Document our contracts and relevant e2e tests very very well.
Tutorials will now

• Folder Setup: point to the codespace for setup (today we talk through the aztec-nargo and jest setup. Codespaces are neater and quicker - Less copy paste)
• link to contract code in the monorepo.
• typescipt glue cpde -> the only place e2e code differs from sandbox is the setup (before() section). We can add multi line comments saying "UNCOMMENT IF USING SANDBOX" and add relevant setup their.

This does a few things:

• Following tutorials is cleaner and quicker
• Tutorials are constantly tested (except the small typescript setup code which itself is tested in other places)
• We streamline our focus on high quality devs (at this stage, we don't want many devs. We want many high quality devs)

there is an additional problem I am trying to solve for -> minimising the number of copy-pastes and instead having 1-2 files of really well documented code.

• draft a doc that explains the requirements, research and rationale for the solutions chosen to make it easier for everyone to get up to speed on how to maintain tutorials

[critesjosh]

Another possible solution: https://schwartz.hashnode.dev/end-to-end-testing-for-developer-tutorials

see discussion below

[rahul-kothari]

I am worried that this issue will be understood as just test tutorials on ci

When in reality, I want to solve the problem of constant copy-pasting and tutorial updates. If all tutorial is just one well commented contract code, then you don't need constant updates or maintenance

E.g. recently Jan replaced address in a note with nullifier_key_hash. Now devrel has to sit through the docs and update tutorial explanations. If the tutorial was just the contract code, there was nothing devrel has to do.

Think a spike on this is necessary - maybe create a different issue? wdyt @critesjosh

[critesjosh]

I like the suggestion of having all of the code / text for a tutorial in 1 file (right now it is spread across the source code and docs files). We could write the tutorial text directly in contract file and create a parser to extract text blocks from the code, similar to how the #include_code macro works.

At the top of each tutorial page, we could have a button to open the tutorial in a Github codespace. The codespace would open with the relevant aztec contract, which would have all of the tutorial text directly embedded in it, so devs wouldnt have to leave the codespace to go through the tutorial. Alternatively, if someone was just using the page as a reference, all of the text and code from the Aztec contract file would be displayed to read on the docs.

I like this idea. Is this what you are thinking Rahul? If yes, I don't think we need another issue, We can just use this one to pursue the path I described and I will updated my comments above.

Taking this route would mean that the link I shared above is not a suitable path. That guide doesn't consider source code injection like we have with #include_code.

[rahul-kothari]

Yes exactly!!

[signorecello]

So talking with @critesjosh we concluded that we want it to be easily maintainable so it makes sense to pull as much as possible from source. But we also want a tutorial to be interactive, not a reading document.

I'll have this in consideration, I'll also consider a third possibility which is to make it test-driven, sort of like a "game". The tutorial being essentially a codespace that has a number of tests already running, and as it proceeds on the tutorial, some tests will start passing. At the end the user has a bunch of green ticks, has learned, and has a boilerplate.

This would make the tutorial both easy to maintain (we're reusing existing tests for these very same contracts), but also interactive.

In any case I'll spike and share in a notion doc as usual hehe

[rahul-kothari]

But we also want a tutorial to be interactive, not a reading document.

I have been receiving consistent feedback on how they don't want it to be interactive

You can do both though - hyperlink the reading doc. And then tell people to read the interactive. Note that if you do interactive you have the exact same problems as today (where explanations are always out of date and engineers need to keep maintaining it)

[critesjosh]

If the source code has comments that get injected into the tutorial, it should be clear when the explanations for the following/preceding function are out of date, since it's right next to the source.

But I agree we could do both. There could be a link at the top of the tutorial to see the complete tutorial code in a codespace, if the reader just wants a reference, or they could proceed through the interactive tutorial to go through the motions and actively learn.

[signorecello]

Finally had some time to think about this.

Short conclusion: I'm dropping the idea of keeping the tutorial text together with the code.

Long conclusion:

They have different goals

This is the main reason.

• The code is meant to work in a complete, contextual, safe, and self-sufficient manner.
• The tutorial is meant to minimize time-to-first-win, while engaging and leading the developer towards autonomy.

In short, a tutorial is meant to be sound, not complete. The copy-pasting is a symptom that we're expecting completeness.

Copy-pasting is the equivalent of making an amazing drawing, giving it to your kid to paint a little corner, and convincing him he drew the whole thing. Heck, I would play a full song in the piano so my students could play the only note they knew how to play and be like "look mom! I'm making music!".

This is not easy to do with grown-up software developers. So we should give up copy-pasting.

My proposal

I made an ordered list of tutorial capital sins. You may disagree, but I had to start somewhere.

1. Not being reproducible. This can have catastrophic motivational consequences. You spend 10min on something that gives you a big fat undebuggable red error ==> we won't even hear from you.
2. Not being interactive enough (i.e. copy-pasting). You're telling me the code, I want you to show me what it does.
3. Abstracting so much complexity from the learner, they will feel misled. This is one of the "learning Rust" sins. You always feel like it can't be that easy. And yep, it isn't.
4. Not teaching enough about the thing we want to teach.

I believe we are focusing too much on the 4th instead of the other ones. We want to teach incredibly complex software as #1 prio and I believe that's a mistake.

So going down the list, we start writing tutorials that are:

1. So simple they are hard to get wrong, yet easy to test in a playwright script.
2. Abstract away the critical details into how-to guides, which are much more forgiving towards knowledge pre-requisites and complexity.
3. Provide a quick win.
4. Provide an absurd amount of links for basically anything that is >40 IQ-level of complexity. We let the reading lead their own learning, which should allow us to hit a broader range of devs.

This means I would not have a tutorial for anything currently in our codebase. I would use those in how-to guides, which allow for a boilerplate and extensive pre-requisites. You can tell the reader "we assume you already finished the tutorial" in order to skip the copy-pasting, start from an arbitrary knowledge pre-requisite, and go straight to the point.

I'll try refactoring one of the tutorials, and we can decide if this would be a good way forwards

[critesjosh]

I chatted with @signorecello a bit about this this morning and he said it succinctly. If a tutorial is being used to refer to how to do something, it should be in the Guides section, not a Tutorial. Tutorials should be strictly for learning and should not be used as a reference.

I think the feedback that @rahul-kothari received is an indication that devs couldn't find a section of the docs that showed them how to do the thing that they wanted to do, so they were going through the tutorial because they knew the info was in there. Hopefully the docs structure refactor makes the relevant information easier to find, and if not we should add the relevant how-tos to the Guides section.