[Bug] 1.23.1 images don't work on linux/arm64 (Azure)

[robcao]

What are you really trying to do?

I'm trying to run the newest version of temporal server, 1.23.1, on an Azure linux/arm64 virtual machine. There are no issues with running temporal server 1.23.0.

There are no issues with running temporal server 1.23.1 on an Azure linux/amd64 virtual machine.

I have confirmed the issue is present on the following images for 1.23.1

• admin-tools
• server
• auto-setup

Describe the bug

Binaries such as temporal-server, temporal-sql-tool, etc in the 1.23.1 images are failing to start on linux/arm64 machines with the following error: /usr/local/bin/temporal-server: cannot execute binary file: Exec format error

azureuser@temporal:/temporal-test$ docker compose up 
WARN[0000] /temporal-test/docker-compose.yaml: `version` is obsolete 
[+] Running 5/3
 ✔ Network temporal-network        Created                                                                            0.0s 
 ✔ Container temporal-postgresql   Created                                                                            0.1s 
 ✔ Container temporal              Created                                                                            0.1s 
 ✔ Container temporal-ui           Created                                                                            0.1s 
 ✔ Container temporal-admin-tools  Created                                                                            0.1s 
Attaching to temporal, temporal-admin-tools, temporal-postgresql, temporal-ui
temporal-postgresql   | The files belonging to this database system will be owned by user "postgres".
temporal-postgresql   | This user must also own the server process.
temporal-postgresql   | 
temporal-postgresql   | The database cluster will be initialized with locale "en_US.utf8".
temporal-postgresql   | The default database encoding has accordingly been set to "UTF8".
temporal-postgresql   | The default text search configuration will be set to "english".
temporal-postgresql   | 
temporal-postgresql   | Data page checksums are disabled.
temporal-postgresql   | 
temporal-postgresql   | fixing permissions on existing directory /var/lib/postgresql/data ... ok
temporal-postgresql   | creating subdirectories ... ok
temporal-postgresql   | selecting dynamic shared memory implementation ... posix
temporal-postgresql   | selecting default max_connections ... 100
temporal-postgresql   | selecting default shared_buffers ... 128MB
temporal-postgresql   | selecting default time zone ... Etc/UTC
temporal-postgresql   | creating configuration files ... ok
temporal-postgresql   | running bootstrap script ... ok
temporal              | TEMPORAL_ADDRESS is not set, setting it to 172.22.0.3:7233
temporal              | Waiting for PostgreSQL to startup.
temporal-postgresql   | performing post-bootstrap initialization ... ok
temporal-ui           | 2024/05/01 04:55:16 Loading config; env=docker,configDir=config
temporal-ui           | 2024/05/01 04:55:16 Loading config files=[config/docker.yaml]
temporal-ui           | 2024/05/01 04:55:16 Loading config; env=docker,configDir=config
temporal-ui           | 2024/05/01 04:55:16 Loading config files=[config/docker.yaml]
temporal-ui           | 
temporal-ui           |    ____    __
temporal-ui           |   / __/___/ /  ___
temporal-ui           |  / _// __/ _ \/ _ \
temporal-ui           | /___/\__/_//_/\___/ v4.9.0
temporal-ui           | High performance, minimalist Go web framework
temporal-ui           | https://echo.labstack.com
temporal-ui           | ____________________________________O/_______
temporal-ui           |                                     O\
temporal-ui           | ⇨ http server started on [::]:8080
temporal-postgresql   | syncing data to disk ... ok
temporal-postgresql   | 
temporal-postgresql   | 
temporal-postgresql   | Success. You can now start the database server using:
temporal-postgresql   | 
temporal-postgresql   |     pg_ctl -D /var/lib/postgresql/data -l logfile start
temporal-postgresql   | 
temporal-postgresql   | initdb: warning: enabling "trust" authentication for local connections
temporal-postgresql   | You can change this by editing pg_hba.conf or using the option -A, or
temporal-postgresql   | --auth-local and --auth-host, the next time you run initdb.
temporal-postgresql   | waiting for server to start....2024-05-01 04:55:16.500 UTC [48] LOG:  starting PostgreSQL 13.14 (Debian 13.14-1.pgdg120+2) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
temporal-postgresql   | 2024-05-01 04:55:16.504 UTC [48] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
temporal-postgresql   | 2024-05-01 04:55:16.517 UTC [49] LOG:  database system was shut down at 2024-05-01 04:55:16 UTC
temporal-postgresql   | 2024-05-01 04:55:16.524 UTC [48] LOG:  database system is ready to accept connections
temporal-postgresql   |  done
temporal-postgresql   | server started
temporal-postgresql   | CREATE DATABASE
temporal-postgresql   | 
temporal-postgresql   | 
temporal-postgresql   | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
temporal-postgresql   | 
temporal-postgresql   | 2024-05-01 04:55:16.788 UTC [48] LOG:  received fast shutdown request
temporal-postgresql   | waiting for server to shut down....2024-05-01 04:55:16.793 UTC [48] LOG:  aborting any active transactions
temporal-postgresql   | 2024-05-01 04:55:16.795 UTC [48] LOG:  background worker "logical replication launcher" (PID 55) exited with exit code 1
temporal-postgresql   | 2024-05-01 04:55:16.795 UTC [50] LOG:  shutting down
temporal              | Waiting for PostgreSQL to startup.
temporal-postgresql   | 2024-05-01 04:55:16.833 UTC [48] LOG:  database system is shut down
temporal-postgresql   |  done
temporal-postgresql   | server stopped
temporal-postgresql   | 
temporal-postgresql   | PostgreSQL init process complete; ready for start up.
temporal-postgresql   | 
temporal-postgresql   | 2024-05-01 04:55:16.925 UTC [1] LOG:  starting PostgreSQL 13.14 (Debian 13.14-1.pgdg120+2) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
temporal-postgresql   | 2024-05-01 04:55:16.925 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
temporal-postgresql   | 2024-05-01 04:55:16.925 UTC [1] LOG:  listening on IPv6 address "::", port 5432
temporal-postgresql   | 2024-05-01 04:55:16.932 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
temporal-postgresql   | 2024-05-01 04:55:16.944 UTC [63] LOG:  database system was shut down at 2024-05-01 04:55:16 UTC
temporal-postgresql   | 2024-05-01 04:55:16.952 UTC [1] LOG:  database system is ready to accept connections
temporal              | PostgreSQL started.
temporal              | Setup PostgreSQL schema.
temporal              | /etc/temporal/auto-setup.sh: line 234: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
temporal              | /etc/temporal/start-temporal.sh: line 16: /usr/local/bin/temporal-server: cannot execute binary file: Exec format error
temporal              | /etc/temporal/start-temporal.sh: line 16: /usr/local/bin/temporal-server: No error information
temporal exited with code 1

Minimal Reproduction

I am using an Azure linux/arm64 VM, but I believe this is not an Azure specific issue.

On a linux/arm64 machine, run the docker-compose-postgres.yaml on the v.1.23.1 tag of the docker compose samples.

View that the auto-setup container fails to start, with the following logs:

TEMPORAL_ADDRESS is not set, setting it to 172.21.0.3:7233
Waiting for PostgreSQL to startup.
Waiting for PostgreSQL to startup.
PostgreSQL started.
Setup PostgreSQL schema.
/etc/temporal/auto-setup.sh: line 234: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
/etc/temporal/start-temporal.sh: line 16: /usr/local/bin/temporal-server: cannot execute binary file: Exec format error
/etc/temporal/start-temporal.sh: line 16: /usr/local/bin/temporal-server: No error information

Environment/Versions

• OS and processor: [e.g. M1 Mac, x86 Windows, Linux]

 Linux 6.5.0-1018-azure #19~22.04.2-Ubuntu SMP aarch64 aarch64 aarch64 GNU/Linux linux/arm64

Architecture:           aarch64
  CPU op-mode(s):       32-bit, 64-bit
  Byte Order:           Little Endian
CPU(s):                 4
  On-line CPU(s) list:  0-3
Vendor ID:              ARM
  Model name:           Neoverse-N1
    Model:              1
    Thread(s) per core: 1
    Core(s) per socket: 4
    Socket(s):          1
    Stepping:           r3p1
    BogoMIPS:           50.00
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
Caches (sum of all):    
  L1d:                  256 KiB (4 instances)
  L1i:                  256 KiB (4 instances)
  L2:                   4 MiB (4 instances)
  L3:                   32 MiB (1 instance)
NUMA:                   
  NUMA node(s):         1
  NUMA node0 CPU(s):    0-3
Vulnerabilities:        
  Gather data sampling: Not affected
  Itlb multihit:        Not affected
  L1tf:                 Not affected
  Mds:                  Not affected
  Meltdown:             Mitigation; PTI
  Mmio stale data:      Not affected
  Retbleed:             Not affected
  Spec rstack overflow: Not affected
  Spec store bypass:    Not affected
  Spectre v1:           Mitigation; __user pointer sanitization
  Spectre v2:           Mitigation; CSV2, BHB
  Srbds:                Not affected
  Tsx async abort:      Not affected

• Temporal Version: [e.g. 1.14.0?] 1.23.1
• Are you using Docker or Kubernetes or building Temporal from source? Docker

Additional context

Inspecting the docker images of temporalio/server:1.23.0 and temporalio/server:1.23.1, it looks like the build was changed as part of this pull request: #190

1.23.0

1.23.1

[Julien4218]

I'm noticing the same issue with 1.23.1 on aws/graviton, 1.23.0 was working fine.