Introduce profiles attribute for depends_on to define profiles dependency

[Vigilans]

What this PR does / why we need it:

Before compose version 2.18.1, starting a service depending on services in an inactivated profile will not trigger error. This behavior gets fixed in docker/compose#10602.

However, the problem with this feature is that it's implicit, but not that it's of no value. In fact, before and after the fix landed many people report and request about this feature to be available. required keyword was added for it in #382.

This PR takes the solution further, to explicitly support specifying profiles as dependency in depends_on entries.

Behavior of depends_on.profiles:

1. Services in specified profiles are selected for condition checking, and will be started when not running.
2. Stop of compose project will not select these services.

Example config:

services:
  db:
    profiles: [db]
    depends_on:
      - db-init

  db-init:
    profiles: [db]

  web:
    profiles: [web]
    depends_on:
      db:
        condition: service_healthy
        profiles: [db]
      web-redis:

  web-redis:
    profiles: [web]

db is central db for all services, web-redis is redis instance that only serves web.
Here:

• Run docker compose --profile web up when db not created/started:
  db-init and db will be created and started, then web-redis and web is created and started.
• Run docker compose --profile web up --force-recreate when db is healthy:
  Only web-redis and web is selected and created/started, since db is already ready.
• Run docker compose --profile web down when db is healthy:
  db will not be selected, only web-redis and web is stopped and removed.
• Run docker compose --profile web restart when db is healthy:
  db will not be selected, only web-redis and web is restarted.

More formal use cases of this feature will be elaborated on subsequent comments.

Which issue(s) this PR fixes:

docker/compose#10751
docker/compose#10804
#274
docker/compose#10993

[Vigilans]

Proposal: Recommended practice for component switching

This proposal proposes a better way to manage a compose project of multiple components: use include + profiles + depends_on.profiles.

Environment setup

Say we have following directory layout:

homelab/
├── .env
├── compose.yml
├── gitlab/
│   └── compose.yml
├── minio/
│   └── compose.yml
└── sharelatex/
    └── compose.yml

Where root compose.yml is:

networks:
  ...

secrets:
  ...

include:
  - minio/compose.yml
  - gitlab/compose.yml
  - sharelatex/compose.yml

And root .env contains:

COMPOSE_PROFILES=minio,gitlab,sharelatex # or '*' for all profiles

Here:

• We call each of homelab's subfolder a component, and all its services in sub compose.yml will have the same profile, .e.g. sharelatex profile in sharelatex/compose.yml services.
• minio is a component for S3 storage, therefore all other services using S3 as backend should include it as dependency, e.g. gitlab and sharelatex.

Now, we can select one of the components through:

$ docker compose --profile sharelatex config

And manage all services the old-school way by not providing profile:

$ docker compose config

It works because root .env's COMPOSE_PROFILES explicitly set all profiles or *. It is recommended to explicitly list all default components (and exclude some optional components in it).

Command line provided COMPOSE_PROFILES will also override the value in .env file:

$ COMPOSE_PROFILES=sharelatex docker compose config

So .env provided COMPOSE_PROFILES will not interfere existing behaviors of COMPOSE_PROFILES=... docker compose and docker compose --profile ....

Use depends_on.profiles for cross-component dependency

Since sharelatex depends on minio to be healthy to utilize its S3 storage, we declare it as dependency:

services:
  sharelatex:
    depends_on:
      minio:
        profiles: [minio]
        condition: service_healthy

By explicitly specifying minio profile in minio service dependency, starting of sharelatex service will start minio service (and other services minio relies on in minio profile) and wait until it's healthy, but do not select minio services into management.

That is, docker compose --profile sharelatex restart command will 1) only restart sharelatex services, and 2) start minio service if not up, and wait for it to be healthy.

Drawbacks of existing solutions

List of existing solutions are quoted from docker/compose#10751.

1. Multiple compose files

The simplest approach for now is to rely on multiple compose files.

By using multiple compose files, one selects components with:

$ docker compose -f compose.yml -f sharelatex/compose.yml config

But sharelatex depends on minio component. Therefore you have to include one more file in command:

$ docker compose -f compose.yml -f minio/compose.yml -f sharelatex/compose.yml config

This:

1. Breaks the convention that user should not care about what the dependencies are once written in config.
2. Even if user takes the trouble to manage it, how can user know how many dependencies a component has once the dependency chain gets deeper? It also cannot be easily discovered through external script.

It also doesn't have a good way to manage all services without using external script.

2. Profiles + write child component's profile to parent dependency's profiles

About your proposal, same can be achieved already by declaring service a with profiles: [a,b]

By using profiles in this way, one selects components with:

$ docker compose --profile sharelatex config

Since minio component also gets written with sharelatex profile, above command need not change:

minio:
  profiles: [minio, sharelatex]

Seems good, but still with notable drawbacks:

1. Breaks the convention that children dependencies should be transparent to parent.

   Why should minio component's yml should know it has a sharelatex children? When there are more services that relies on S3 storage, they all need be written to minio's compose file.

   What's more, minio is a component with mutiple services. Say we have m minio services with n children components, then the profiles have to be written m*n times to minio's compose file. This is really a burden to maintainer.

2. It is not optimal to select parent services into management.

   When selecting sharelatex profile without minio services, only sharelatex services will be selected for management:

   $ docker compose --profile sharelatex restart
   [+] Restarting 5/5
   ✔ Container sharelatex-mongo    Started
   ✔ Container sharelatex          Started
   ✔ Container sharelatex-texlive  Started
   ✔ Container sharelatex-mount    Started
   ✔ Container sharelatex-redis    Started
   

   However, when specifying minio services with sharelatex profiles, restart of compose will also restart minio services:

   docker compose --profile sharelatex restart                                                                                                                           
   [+] Restarting 7/7
   ✔ Container sharelatex-mongo    Started
   ✔ Container sharelatex-mount    Started
   ✔ Container sharelatex          Started
   ✔ Container sharelatex-texlive  Started
   ✔ Container minio-mount         Started
   ✔ Container minio               Started
   ✔ Container sharelatex-redis    Started
   

   This is not optimal and even unacceptable, since minio is an important base service for storage, how can a restart of sharelatex component also restarts minio component that breaks all other components relying on it?

3. Multiple compose files + include

this is why include has been introduced, so you can rely on third-party components without having to know about implementation details

You can then use:

include:
    - ${BACKEND:-sharelatex}/compose.yml

.. which will load sharelatex implementation by default, but you can select another one by just setting BACKEND variable so this point to another compose file, as long as same service name is declared

By using multiple compose files in this way, one write component config sharelatex/compose.yml with:

include:
  - ../minio/compose.yml

services:
  sharelatex:
    depends_on:
      minio:
        condition: service_healthy

Then selects components with:

$ BACKEND=sharelatex docker compose up

It solves problem 1.1, problem 1.2 and problem 2.1, but still faces the same problem with problem 2.2 that selecting minio services for management may not be optimal, like restarting sharelatex component will also restart minio component that breaks other services (e.g. gitlab) that depends on minio.

---

Therefore, proposed way solves all drawbacks above, and presents a very good solution for component switching because it only uses simple docker compose command as entrypoint, without resorting to external script.

[Vigilans]

Quoted from docker/compose#10993 (comment):

While skimming through this issue and related PR, I came up with these two questions:

• When using up (without profiles), should the dependency check pass in case required dependencies are in profiles, but they're already up?
• When using down (without profiles), should the services without profiles be stopped and removed, regardless of any dependencies in profiles?

I'm trying to use a single instance of database as required dependency of multiple Compose projects. My idea was to merge the database's Compose file (via COMPOSE_FILES), and put the database service in a profile, so I don't accidentally destroy the database when playing with database-dependent Compose projects. Obviously it's not working this way, at least not in Compose 2.21.0.

@GethDeeo Your idea can be solved using depends_on.profiles attribute proposed here.

With following config:

services:
  db:
    profiles: [database]

  web:
    depends_on:
      db:
        condition: service_healthy
        profiles: [database]

• When using up (without profiles), the dependency check will pass because db is already up. If db is not up, it will be created/started.
• When using down (without profiles), the web service (without profiles) will be stopped and removed, regardless of db dependency in profile database that is not activated.