@hoodie/api

[gr2m]

An issue like this should go through an RFC process which we discuss at #85 but did not setup yet, so this issue must do :)

🤔 Some background information for context

Hoodie always had an API in the client, that’s how it all started about 5 years ago: we dreamed up the nices client API for common web applications and made it work backwards. Today’s Hoodie Client API is only slightly different from what we came up back then.

JavaScript APIs on the server are new, we did not plan them, they grew organically. Hoodie’s core modules are Account & Store, both have a client and a server component which work together. The server part is using hapi to define its routes and define hooks around request life cycles. The logic to user account documents, user databases, security settings etc have all been part of @hoodie/account-server and @hoodie/store-server. Once we identified the patterns we moved the logic out into its own repositories: @hoodie/account-server-api and @hoodie/store-server-api. We are also working on the task module which will be split up in a similar way.

So things grew organically, and Hoodie’s architecture (for the server side) looks currently like this:

hoodie
└─┬ @hoodie/server
  ├─┬ @hoodie/account-server
  │ └── @hoodie/account-server-api
  └─┬ @hoodie/store-server
    └── @hoodie/store-server-api

Because of that, accessing the account server api is possible at server.plugins.account.api.

But what would make much more sense is server.app.hoodie.account. The server.app comes from hapi, it’s meant to be used for app-specific state.

In order to make that possible, I suggest a new architecture

hoodie
└─┬ @hoodie/server
  ├── @hoodie/account-routes
  ├── @hoodie/store-routes
  └─┬ @hoodie/api
    ├── @hoodie/account-api
    └── @hoodie/store-api

❓ The Motivation

For one, it makes sense to have an API on the server just like we have it on the client. The difference is that while the Hoodie Client API is scoped to a signed in user, the Hoodie Server API would be from the admin perspective, allowing to manage user accounts, databases, etc.

The benefit for plugin developers is two fold:

1. server.app.hoodie is simpler to explain and understand. We could have a central place where we document Hoodie’s Server API just like we do with the client API
2. Plugins could run in their own processes.

The second point is important. CouchDB’s /_changes feed is a great API that can be used as an event bus for distributed systems. Plugins could run in own processes, even on different machines, and could be scaled up and down independently. When they crash, they won’t take down the entire server, and the other way around. I’ve created such architectures several times in the past, so did many others – and we want to make it a first class citizen at Hoodie.

Last but not least, having the Hoodie Server API which talks directly to the CouchDB (or PouchDB, what ever is configured on the Server) can even be used by itself in things like a Hoodie CLI tool, e.g. think $ hoodie console. I’m sure people will come up with many interesting use cases beyond what we can imagine today.

Because of the prospect of @hoodie/api being used outside of a server, I prefer @hoodie/api over @hoodie/server-api and e.g. @hoodie/account-api over @hoodie/account-server-api. We need to clarify the naming in our documentation to make sure people understand the difference between @hoodie/client and @hoodie/api, but after giving this lots of thoughts, it makes perfectly sense. @hoodie/api is talking to the database directly using PouchDB, data is directly read from / written to the database. All security checks must be done before calling the respective APIs. @hoodie/client is not talking directly to the PouchDB where all of users’s data and all user accounts are store, instead it is talking through @hoodie/server which defines routes. Within its route handlers all security checks like session checks are taking place.

Here’s a diagram

+----------------+                   +----------------+              +---------------+
|                |   http requests   |                |     uses     |               |
| @hoodie/client |                   | @hoodie/server |              |  @hoodie/api  |
|                <------------------->                +-------------->               |
+-^--------------+                   +----------------+              +-^-------------+
  |                                                                    |
  |                                                                    |
  |  reads/writes                                                      | reads/writes
  |                                                                    |
  |                                                                    |
+-v--------------+                                                   +-v-------------+
|                |                                                   |               |
|   client db    |                                                   |   server db   |
|(current user’s |                                                   | (all accounts |
|   data only)   |                                                   |  & user data) |
|                |                                                   |               |
+----------------+                                                   +---------------+

📋 The plan ™️

1. create a repository for @hoodie/api. Add @hoodie/store-server-api and @hoodie/account-server-api as dependencies. This module should work similar to how @hoodie/client works today. It should accept a PouchDB constructor and other options and return an API for .account and .store. It should also implement a .plugin method like @hoodie/client’s hoodie.plugin method.
2. rename hoodie-account-server-api repository to hoodie-account-api
3. rename @hoodie/account-server-api in package.json to @hoodie/account-api
4. manually release @hoodie/account-api to npm with the next breaking version after the current version of @hoodie/account-server-api
5. create a git tag and a release on GitHub as if `semantic-release did it.
6. rename hoodie-store-server-api repository to hoodie-store-api
7. rename @hoodie/store-server-api in package.json to @hoodie/store-api
8. manually release @hoodie/store-api to npm with the next breaking version after the current version of @hoodie/store-server-api
9. create a git tag and a release on GitHub as if `semantic-release did it.
10. rename @hoodie/account-server to @hoodie/account-routes. Remove the dependency on @hoodie/account-server-api. Adapt the implementation to consume the API set on server.app.hoodie.account instead of instantiating its own API required from @hoodie/account-server-api. Then manually release @hoodie/account-routes to npm with next breaking version after the current version of @hoodie/account-server. Create a git tag and a release as if semantic-release did it to keep the changelog of past releases.
11. rename @hoodie/store-server to @hoodie/store-routes. Remove the dependency on @hoodie/store-server-api. Adapt the implementation to consume the API set on server.app.hoodie.store instead of instantiating its own API required from @hoodie/store-server-api. Then manually release @hoodie/store-routes to npm with next breaking version after the current version of @hoodie/store-server. Create a git tag and a release as if semantic-release did it to keep the changelog of past releases.
12. add @hoodie/api as dependency to @hoodie/server and instantiate it with it. All the cross-initialisation between @hoodie/account-api and @hoodie/store-api (e.g. create a user database when an account gets created) should still live in @hoodie/server, not in @hoodie/api. Set server.app.hoodie to the @hoodie/api instance. Remove @hoodie/account-server & @hoodie/store-server from dependencies and add @hoodie/account-routes & @hoodie/store-routes instead. Release a new breaking version
13. update @hoodie/server dependency in hoodie. In the same PR, update documentation in the docs/ folder. The Hoodie Client API should be renamed to Client and The Hoodie Server API should be renamed to API. We should also add Routes but we can leave that out for now or create a placeholder page which has a link to an issue where we track the progress on it. Release a new breaking version.
14. deprecate @hoodie/account-server on npm
15. deprecate @hoodie/account-server-api on npm
16. deprecate @hoodie/account-store on npm
17. deprecate @hoodie/account-store-api on npm

💭🤔💡💬 Discuss!

This was all in my head for way too long. Feels good to get it out there. I know it’s a lot to digest. Let me know what you think :)

🍬🍩 Bonus ✨

As mentioned above, Hoodie’s server API grew organically, it was not thought through and discussed endlessly like we did with Hoodie Client. Now if we make @hoodie/api a first class citizen, we should probably do that now.

For example, instead of having an API like hoodie.account.accounts.find() and hoodie.account.account(id).tokens.add() we probably rather want hoodie.account.find() and hoodie.account(id).tokens.add().

Time for some dreamcoding :)

Last input I’d like to share is that @hoodie/api could have a very similar API as @hoodie/admin-client – ideally it would even be the same! The difference would be that @hoodie/api would talk directly to PouchDB while @hoodie/admin-client would talk to routes defined by @hoodie/admin-routes, so it would need to know the admin credentials for the respective Hoodie app.

[janl]

I like the idea of a server API. We’ve had something like this in the past and punted on it in favour for using raw PouchDB as the Server client API for the current iteration, but I always envisioned that we bring back a higher level API on the server.

We need to clarify the naming in our documentation to make sure people understand the difference between @hoodie/client and @hoodie/api, but after giving this lots of thoughts, it makes perfect sense.

I don’t see at all how this makes perfect sense. I think it is very confusing. In addition, I’m also confused by a lot of the intermediate modules we started having (so you could e.g. use hoodie-account as a standalone thing), so that doesn’t help clear anything up.

For the moment, I think @hoodie/server-api is very clear and not at all confusing, if not pretty, but I’d be okay with that.

[gr2m]

I’m also confused by a lot of the intermediate modules we started having (so you could e.g. use hoodie-account as a standalone thing)

they are not part of hoodie, they have been very helpful while working on the new Hoodie implementation, in order to have a server where the client & server counterparts for the store or the account module could be tested together. Probably a good idea to deprecate the packages on npm and move the repositories out of the hoodiehq org

[gr2m]

@hoodie/server-api

I foresee that the API will be used outside of the server, e.g. in a @hoodie/cli package. But I agree that it makes things clearer. Too many things are called API these days :D

[ransomw]

continuing a discussion from the IRC, a modified diagram might look something like

+---------------+                    +----------------+
| some other    <-------------------->    app plugin  |
| (non-JS?)     |    http requests   +----------------+
|   client      |                            |          \__
+---------------+                         accesses         \
                                             |               indirecctly accesses
                                            \/                            \/
+----------------+                   +----------------+              +---------------+
|                |   http requests   |                |    exposes   |               |
| @hoodie/client |                   | @hoodie/server |              |  @hoodie/api  |
|                <------------------->                +-------------->               |
+-^--------------+                   +----------------+              +-^-------------+
  |                                                                    |
  |                                                                    |
  |  reads/writes                                                      | reads/writes
  |                                                                    |
  |                                                                    |
+-v--------------+                                                   +-v-------------+
|                |                                                   |               |
|   client db    |                                                   |   server db   |
|(current user’s |                                                   | (all accounts |
|   data only)   |                                                   |  & user data) |
|                |                                                   |               |
+----------------+                                                   +---------------+

idea being that application-specific plugins could use this setup to modify the server db.

---

i generally agree that it's beginning to look like a hurd of cats/dogs/chickens/bears/whoknowswhat, but this really is exactly what i was thinking the existing @hoodie/store-server-api was supposed to do without even knowing the issue was open 👾 🎐 ...

perhaps the pouchdb-hoodie-api might be is way to get around the hurding? is it possible to modify that package to take some parameter (e.g. hoodie data path or db url) that'll allow a separate in-memory JS object (PouchDB instance) to access the same sever on-disk db as the in-memory JS object used by the hoodie server? i'm not familiar enough with pouch/couch to know if locks will prevent this. anyone?

+---------------+                    +----------------+
| some other    <-------------------->    app plugin  |
| (non-JS?)     |    http requests   |      cli       |
|   client      |                    |      etc.      |
+---------------+                    +-  -  -  -  -  -+
                                     | hoodie pouchdb |
                                     +-----^----------+
                                           |
                                           | reads/writes
                                           |
                                         +-v-------------+
                                         |               |
                                         |   server db   |
                                         | (all accounts |
                                         |  & user data) |
                                         |               |
                                         +---------------+

---

i dunno, it's really a tough call, kind a "can't please all the people all the time" cohesion/decoupling quandry. keeping that original "hey i made a webapp in an hour!" vibe is probably as good a guiding light as any to follow in these code design considerations.

[gr2m]

idea being that application-specific plugins could use this setup to modify the server db.

I think your use case is already covered by the original idea. You’ll be able to do something like this in plugins

server.app.hoodie.store.open('mydb')

.then((db) => {
  // do something with the database
})

Or am I missing what you need? Maybe describe it on a higher level?

[ransomw]

I think your use case is already covered by the original idea.

yes, it definitely is: the first high-level block diagram in my previous comment is indeed an extension of the the imagined architecture of the original idea.

in the second block diagram, the low-level api details or dreamcode might look something like

const PouchDB = require('pouchdb')
PouchDB.plugin(require('pouchdb-hoodie-api')(
  server.some.path\....
))
const dbApi = (new PouchDB('mydb')).hoodieApi();
// do things (open, create, etc.)

and the thing that having to write a few lines of boilerplate in plugins would buy is the simpler high-level diagram and possible associated code-design benefits:

there can be fairly tight cohesion if the hoodie/hapi server object has just the right config to pass to pouchdb-hoodie-api (and possibly elsewhere, depending on impedance matching with pouchdb api, e.g., to ensure in-memory storage, no/minimal caching, etc.). yet cohesion is not required (e.g. the server's config defaults can be modified), because the plugin (or cli, etc.) is relying on pouchdb-hoodie-api, not server to access the data store. in fact, i've managed to convince myself that it "makes sense":tm: not to require an http server in order to access data storage.

or, if i've grossly misread the purpose of pouchdb-hoodie-api, replace the above with something like

const dbApi = require('@hoodie/api')(
  server.some.path\...
)

and the previous paragraph still applies with a few name changes.

---

the overall message was, "yes, this does appear to do exactly what i was asking about in IRC re. server.plugins['store']['api']" (and that's awesome and such a wild coincidence!) with the alternate design stuff more as a footnote or something to consider as a way to address the comment about "this is confusing" rather than something additional that's necessary for my particular use case.

i dunno if the "inversion of config" (somewhat like IoC) design of the second block diagram and dreamcode snippets above is the best way to address @janl 's comment. it is a semi-definite point of departure for discussion.

---

a final, and perhaps the most important, note that i can bring to the table:

unless the only use-case is to have databases that are entirely plugin, cli, etc. dependent, it might be more instructive to use more definite names than 'somedb' or 'mydb' in examples. if a use case is to access a database created by the hoodie client in the browser via this setup, use the actual database name created by the hoodie client. or if a use case is to access a cli, plugin, etc. specific database from the hoodie client (or client extension), accompany the examples with some snippet showing how to do so.

in short, adding the context of an end-to-end use scenario to some of examples that currently only use 'mydb' could substantially illuminate the documentation.