DOCS: TOC builder (for functions and providers)

[systemcrash]

first attempt to automate TOC creation.

This is a draft so there's probably a few misses in here to sort through after some test doc generations.

Summary
Automatically generate the TOC in SUMMARY.md and fill it with the hierarchical doc structure found in the documentation folder. Now we don't need to manually update it or its links. Folder structure depth is mirrored in the TOC.

It's possible to supply file/folder name exceptions at various points.

Checks are simple and can possibly be defeated.

Due to the (renames and) changes for this PR, it's not advisable to 'merge master' into this branch.... yet.

[systemcrash]

OMG. So much structure. So many doc. Everything still appears intact on the doc builds. Could do with some polish for filenames.

[systemcrash]

I may invest some time to program a scour mechanism that goes through all markdown and updates all the links if a file has moved location or got renamed. But that can come later.

[systemcrash]

That would work.

[cafferata]

Thanks @systemcrash! Another step in the right direction! I myself have had similar ideas see #1998. I just don't know if I would make the same choice to generate the full SUMMARY.md myself. With this you lose the ability to show better menu names than in the new situation.

Some examples:

-CI/CD example for GitLab
+Ci-Cd-Gitlab

-Why CNAME/MX/NS targets require a "dot"
+Why Targets Require The Final Dot

I've always thought of two (or more) GitBook menu structures:

1. The providers
2. The language reference

I'm curious about your thoughts! @systemcrash @tlimoncelli

[systemcrash]

OK - I'm satisfied with the results now.

Thanks @systemcrash! Another step in the right direction! I myself have had similar ideas see #1998. I just don't know if I would make the same choice to generate the full SUMMARY.md myself. With this you lose the ability to show better menu names than in the new situation.

Some examples:

-CI/CD example for GitLab
+Ci-Cd-Gitlab

-Why CNAME/MX/NS targets require a "dot"
+Why Targets Require The Final Dot

Just a question of renaming the file (as closely) as possible. Or modifying the code to pick out the first # header from each md and updating every md to have one, if perfection is required. But perfection is often the enemy of good.

I've always thought of two (or more) GitBook menu structures:

1. [The providers](https://github.com/StackExchange/dnscontrol/blob/f912b15/documentation/SUMMARY.md?plain=1#L89-L129)

2. [The language reference](https://github.com/StackExchange/dnscontrol/blob/f912b15/documentation/SUMMARY.md?plain=1#L15-L83)

I'm curious about your thoughts! @systemcrash @tlimoncelli

Do you think they're sufficiently mirrored in the current test results? Take a look and let me know @cafferata

[systemcrash]

In tandem with this PR, I recommend that (a few) providers might also be renamed (and their variables too?) to match the docs. Things like Route 53 are findable by search, although this one is actually called "Amazon Route 53".

( PR has overview of which provider md files got renamed )

[cafferata]

@systemcrash a number of feedback points:

1. The absolutes file paths are not supported by GitBook. This causes the URLs to not work at this time.
2. By redesigning folders/summary, this pull request not only has a generation process been built, but also a new look and feel. As far as I'm concerned, we take it step by step so that there is clarity about exactly what has changed and for what reason. Perhaps @tlimoncelli thinks differently about this? As far as I'm concerned, this state of pull request is not ready to merge.
3. See also DOCS: TOC builder (for functions and providers) #2197 (comment)

Before	After

[systemcrash]

Yes - thanks for the feedback. I made a few minor adjustments to retain the original structure as closely as possible. Seems to have worked nicely. The only part that won't align is the provider names - the provider .md files and the providers need renaming to synchronize or, another solution.

When naming freely the provider md files, it looks like you want:
https://docs.dnscontrol.org/~/revisions/284FcG65SMBYAdV2iV3g/providers/providers

If there is some structuring you don't like, one just has to rearrange the files/folders (bearing in mind to adjust any hyperlinks.)

I think this is where we gain by "good code documents itself" - by renaming the providers so everything looks as you want it.

[systemcrash]

Hi @cafferata - are there any other changes you'd like to see for the long term?Or is it now just a question of modifying the processes on the docs server?

[cafferata]

I'm especially curious if @tlimoncelli agrees with the structure change.

1. The new menu structure shows the technical details/directory names.

   -Getting Started
   +01 Getting Started

   -Language Reference
   +02 Language Reference

2. Determining the order of menu items is no longer possible.

   1. Example Getting Started menu items

      -Overview
      -Examples
      -Migrating zones to DNSControl
      -TypeScript autocomplete and type checking
      +Examples
      +Getting Started
      +Migrating Zones To Dnscontrol
      +TypeScript Autocomplete And Type Checking

   2. Example Language Reference

      -Top Level Functions
      -Domain Modifiers
      -Record Modifiers
      +Domain Modifier Functions
      +Record Modifier Functions
      +Top Level Functions

   3. The chapters within the menu structure

      -Service Providers
      -Commands
      -Advanced features
      -Developer info
      -Release
      +Advanced Features
      +Commands
      +Developer Info
      +Providers
      +Release

3. External links
   Randomly adding external links, such as the GitHub releases, is no longer possible.

4. Provider names that change

   -Microsoft DNS Server on Microsoft Windows Server
   +Msdns

   • https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/providers/providers/msdns
   • https://docs.dnscontrol.org/service-providers/providers/msdns

5. Absolutes file paths DOCS: TOC builder (for functions and providers) #2197 (comment)
   This causes the URLs to link to the GitHub markdown view instead of GitBook documentation URLs.
   Question, how are the renames ensured that this goes well? How has this been tested?

   1. Example Introduction to DNSControl > Quick start tutorial

      • https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
      • https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=quick%20start%20tutorial

   2. Example Introduction to DNSControl > migrate

      • https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
      • https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=migrate

   3. Example Introduction to DNSControl > language spec

      • https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
      • https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=language%20spec

   4. Example Providers > Provider table > Domain modifier deeplink

      • https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-28e66bede643a7156cf9c19950731e1a3a9ab66becf1a063f7705076bfc4dbdaR78
      • https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/providers/providers

   5. Example Record Modifier Functions > Service Provider Specific > Amazon Route 53 > R53_ZONE

      • https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-64fa1fd972040542557ac120c90268ff7e8e6fa6407826588f2cf9a82fdbdf7aR15
      • https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/02-language-reference/record-modifier-functions/service-provider-specific/amazon-route-53/r53_zone

[systemcrash]

I'm especially curious if @tlimoncelli agrees with the structure change.

1. The new menu structure shows the technical details/directory names.
   ```diff
   -Getting Started
   +01 Getting Started
   ```

I don't see the problem with technical details. If anything, it makes it easier for new-comers to find and contribute.

I did not find a convenient way of forcing something to appear first, when everything is sorted alphanumerically, except perhaps naming entries G... instead of g... and sorting alphabetically (A-Z first, then a-z). That has its own problems, eg entries that begin DNS.... This whole project is about DNS...

What is the documentation trying to be? Reference? Howto? User guide (examples)? Reference should be alphabetically sorted (providers, functions). Howtos can land in a different folder and get a different TOC...? But we're trying to avoid all of this.

   ```diff
   -Language Reference
   +02 Language Reference
   ```

2. Determining the order of menu items is no longer possible.
   
   1. **Example Getting Started menu items**
      ```diff
      -Overview
      -Examples
      -Migrating zones to DNSControl
      -TypeScript autocomplete and type checking
      +Examples
      +Getting Started
      +Migrating Zones To Dnscontrol
      +TypeScript Autocomplete And Type Checking
      ```
   2. **Example Language Reference**
      ```diff
      -Top Level Functions
      -Domain Modifiers
      -Record Modifiers
      +Domain Modifier Functions
      +Record Modifier Functions
      +Top Level Functions
      ```
   3. **The chapters within the menu structure**
      ```diff
      -Service Providers
      -Commands
      -Advanced features
      -Developer info
      -Release
      +Advanced Features
      +Commands
      +Developer Info
      +Providers
      +Release
      ```

Do we need to? 'Everything else' in the project is sorted alphabetically, which makes it easier to locate entries (reference).

3. External links
   Randomly adding external links, such as the [GitHub releases](https://github.com/StackExchange/dnscontrol/releases/latest), is no longer possible.

Such entries need to land outside of the tags, true. Other 'decorative' static entries can land somewhere in the middle, but that means more tags and more code complexity. I would rather code be easy to understand.

'Randomly' adding content goes into an existing file or makes a new file somewhere, which gets checked in anyway. Don't see the problem.

4. Provider names that change
   ```diff
   -Microsoft DNS Server on Microsoft Windows Server
   +Msdns
   ```

I made an earlier suggestion that we rename provider entries, to align with the real world. Only drawback is that users might have to (oh no) change a text string in their .js files.

e.g. hedns -> hurricane_electric_dns (underscores replaced to spaces). Frankly I find real world naming much easier. Putting some of the short forms into google to find who they were was difficult for some, and uncertain for others.

   * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/providers/providers/msdns
   * https://docs.dnscontrol.org/service-providers/providers/msdns

5. Absolutes file paths [DOCS: TOC builder #2197 (comment)](https://github.com/StackExchange/dnscontrol/pull/2197#issuecomment-1474936257)
   This causes the URLs to link to the GitHub markdown view instead of GitBook documentation URLs.

So... how does one fix this, if gitbook is the solution? Relative but with some path depth?

   **Question**, how are the renames ensured that this goes well? How has this been tested?
   
   1. **Example `Introduction to DNSControl` > `Quick start tutorial`**

The renames were made to follow how they are manually named in the TOC. Just rename the source file. 1:1 naming. Everything else was just link decoration. Now it's automatic. File == page == link.

      * https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
      * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=quick%20start%20tutorial
   2. **Example `Introduction to DNSControl` > `migrate`**
      
      * https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
      * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=migrate
   3. **Example `Introduction to DNSControl` > `language spec`**
      
      * https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
      * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=language%20spec
   4. **Example `Providers` > `Provider table` > Domain modifier deeplink**

This one I'm not sure how to resolve - I don't know what happens server side. ¯_(ツ)_/¯
But I'll stick with something that works.

      * https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-28e66bede643a7156cf9c19950731e1a3a9ab66becf1a063f7705076bfc4dbdaR78
      * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/providers/providers
   5. **Example `Record Modifier Functions` > `Service Provider Specific` > `Amazon Route 53` > `R53_ZONE`**

Same content.

      * https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-64fa1fd972040542557ac120c90268ff7e8e6fa6407826588f2cf9a82fdbdf7aR15
      * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/02-language-reference/record-modifier-functions/service-provider-specific/amazon-route-53/r53_zone

Documentation search precludes any of this from being a problem. Know what you want to find? ⌘K.

[tlimoncelli]

I'm especially curious if @tlimoncelli agrees with the structure change.

1. The new menu structure shows the technical details/directory names.

   -Getting Started
   +01 Getting Started

   -Language Reference
   +02 Language Reference

My concern here is that if we add a section a lot of links will break. I would prefer a system where the name the user sees is not the directory name. This will also permit us to use characters that can't be in a directory name, such as /.

Each file has a preamble (for example documentation/functions/domain/FRAME.md). Could this be part of the solution?

2. Determining the order of menu items is no longer possible.

That's also a problem. It is important that articles are listed in an order that improves usability/readability. For example, the more broad/introductory entries should be at the top of the list.

(Fun fact: When you are at a restaurant for the first time and don't know what to order, the first 1-2 items in any section of the menu are the "safe" items that a new and non-adventurous eater should pick. Likewise, Apple's style-guide says that the items in a drop-down menu should be in the order a typical user will learn them. That's why Open comes before Save, and the last few items in the drop-down menu are the obscure features that most people won't use)

3. External links
   Randomly adding external links, such as the GitHub releases, is no longer possible.

We only have 1 such link. No biggie. We could delete it or have a doc that tells people to go to the link.

4. Provider names that change

-Microsoft DNS Server on Microsoft Windows Server
+Msdns

This is fine. Though we probably want to shorten some of the full descriptions.

[cafferata]

Hi @systemcrash, your response feels personal. @tlimoncelli has asked me to provide feedback/view of this pull request. So I've listed the changes that aren't immediately obvious from the GitHub pull request description (or the big git diff). In addition, I explicitly asked how @tlimoncelli looks at these feedback points. Now it feels like we have to discuss each other's comments and that's not up to me.

Would I personally have done it differently? Yes, I would have chosen to narrow the scope down to what you call 'reference' (language reference and providers).

Am I grateful that you have taken up this work? Absolute!

[systemcrash]

Hi @systemcrash, your response feels personal. @tlimoncelli has asked me to provide feedback/view of this pull request. So I've listed the changes that aren't immediately obvious from the GitHub pull request description (or the big git diff). In addition, I explicitly asked how @tlimoncelli looks at these feedback points. Now it feels like we have to discuss each other's comments and that's not up to me.

Would I personally have done it differently? Yes, I would have chosen to narrow the scope down to what you call 'reference' (language reference and providers).

Am I grateful that you have taken up this work? Absolute!

Better we find a long-term solution than just mashing this in there, since I understand that you put in some work server-side to make this work out. And yes, it's personal: it's my opinion based on this work.

How about naming files with a number prefix for some kind of ordering, but the automation trims the number prefix away?

The only other way I can think of is embedding the title you want in the first line(s) of the .md files (in certain folders) and using those. More complex, but it seems to strike a balance. just have to go through all files and ensure the first line always adheres to this standard. ( Similar to the preamble @tlimoncelli mentions, but not for alphabetically sorted reference stuff )

[tlimoncelli]

@systemcrash I appreciate all the effort you're putting into this. Good, readable, documentation is probably the most important way we bring new people into the project.

I like all the ideas so far. Here are my thoughts about them:

1. Filename prefixes: Use 3-digit numbers with big gaps between them (100_foo, 200_bar, etc) so that new files can fit between them. (Did you program in BASIC on an 8-bit computer?) The downside is that it may be confusing to not see the numbers in the URL.
2. File preamble: Add a preamble to the individual files: Hugo does this. The file has a preamble where you can set variables. For example add weight = 5 to a file's preamble and it will be listed before a file with weight = 6. Files without preambles are listed last.
3. A "toc file". That is... if a directory includes a file called toc.yaml, that file could list the filenames in the order they are to appear (files not listed can go at the end).

Thoughts?

Tom

[systemcrash]

OK, updates:

• The TOC auto-creation is reduced in scope now to only ref material, and providers.

• Each section has its own <!-- --> (replacement function more generic now).

• I added a link rectifier which replaces links to other .md files with a relative link.

  • This operates on files within specified folders - see docuLocalLinkRectifier.go and generate.go

As a result, all links work. TOC is auto-generated. Everybody is happy (I think).

(I may have missed a link or two somewhere).

renames/moves:
providers -> service_providers
functions (etc) -> language_reference

We can increase scope/complexity of auto-generation later with these ideas if we are satisfied with how this currently works

[systemcrash]

Shall we use this automation?