Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package request: X11 Utility Apps, X.Org Server, and Remaining X11 Libraries #26241

Open
2 of 14 tasks
ehfd opened this issue May 4, 2024 · 16 comments
Open
2 of 14 tasks

Comments

@ehfd
Copy link
Member

ehfd commented May 4, 2024

Package name

More to be added.

Package version

Newest

Package website

https://www.x.org/releases/individual/app/

https://www.x.org/releases/individual/util/

Package availability

https://www.x.org/releases/individual

https://gitlab.freedesktop.org/xorg/app

https://gitlab.freedesktop.org/xorg/util

https://salsa.debian.org/xorg-team/app

Additional comments

These are many numbers of command-line or GUI applications from X.org which are typically packaged in Debian or RHEL as a group of packages, such as x11-apps, x11-utils, x11-xserver-utils, and xfonts-utils.

Instead of creating >73 different feedstocks for iceauth, xrandr, xhost, xset, etc... (of which none are currently in feedstocks) it might be a good idea to build them all in one shot together and package them in categories like how Debian or RHEL does, reducing the number of feedstocks to <10.

Moreover, it would be less demanding to first limit the builds of utils and apps to Linux or Mac, unlike libs.

https://salsa.debian.org/xorg-team/app
https://src.fedoraproject.org/repo/pkgs/xorg-x11-server-utils/

Looking for help from experts for this complex recipe.

Package is not available

  • The package is not available on conda-forge.

No previous issues or open PRs

  • No previous issue exists and no PR has been opened.
@ehfd
Copy link
Member Author

ehfd commented May 4, 2024

CC @xhochy @hmaarrfk @pkgw and others.

@ehfd
Copy link
Member Author

ehfd commented May 4, 2024

Additional CC @epruesse @zklaus @scopatz @erykoff @ccordoba12 @mingwandroid @n-elie

I apologize for the tag bomb.

@ehfd
Copy link
Member Author

ehfd commented May 4, 2024

Moreover, it's possibly the opportunity to port the Windows X11 stack from MSYS2/MinGW to VC++ like how https://github.com/marchaesen/vcxsrv did it, or at least try to update the MSYS2 compiler.

I will contribute a subset of the X11 apps packages, typically packaged standalone by Debian or RHEL (such as xorg-libcvt, xauth), which I immediately need.

@ehfd ehfd changed the title Package request: x11-apps, x11-utils, x11-xserver-utils, xfonts-utils, etc. Package request: x11-apps, x11-utils, x11-xserver-utils, xfonts-utils, x11-xserver-xvfb, etc. May 4, 2024
@hmaarrfk
Copy link
Contributor

hmaarrfk commented May 5, 2024

I understand the maintenance challenge with X11 are somewhat tiring, but I'm rather hesitant to adding to our maintenance burden unless absolutely necessary.

We don't have a good way to 'transition' packages at conda-forge, so a new package name isn't necessarily an "easy swap".

Can you point us to conda-forge feedstock you need help with?

@ehfd
Copy link
Member Author

ehfd commented May 5, 2024

@hmaarrfk
None of the packages here I listed are currently in a non-CDT feedstock. Xvfb and xorg-utils (possibly others) are CDT packages. x11-util-macros does not overlap here.

These packages are utility executables (some are important utility tools which control the interaction of an X application with X11 servers where applications depending on X11 are crippled without) with dependencies to currently packaged X11 libraries. We are not changing existing feedstocks.

And my suggestion is simply to merge >30 small feedstocks if done separately into ~5 larger feedstocks similar to Debian or RHEL.

@hmaarrfk
Copy link
Contributor

hmaarrfk commented May 5, 2024

Hmm. Ok. My inclination is to say that unfortunately this is quite troublesome to execute.

conda-forge/conda-forge.github.io#1894

Discusses the reverse problem, but ultimately we don't have a way of "renaming" packages and thus this is really a difficult excercise.

We ignored problem for jpeg and jpeg-turbo and it left many I knew with no option to update. Only to tear down the environment to rebuild it.

@ehfd
Copy link
Member Author

ehfd commented May 5, 2024

Then, this would be a long-term thing, I'll leave this issue up for now.

We're not merging existing packages, but this might be a decision that should be made while creating the initial versions of such packages.

@ehfd
Copy link
Member Author

ehfd commented May 8, 2024

Also in the wishlist: xorg-libxv, xorg-libxxf86dga1

@hmaarrfk
Copy link
Contributor

hmaarrfk commented May 8, 2024

I think the first step in the "long term project" would be to create a table of the new package names, and which existing conda-forge package they "amalgamate".

Typically it helps to edit the first comment to keep the list alive.

@ehfd ehfd changed the title Package request: x11-apps, x11-utils, x11-xserver-utils, xfonts-utils, x11-xserver-xvfb, etc. Package request: x11-apps, x11-utils, x11-xserver-utils, xfonts-utils, ... etc. May 8, 2024
@ehfd
Copy link
Member Author

ehfd commented May 8, 2024

@ehfd ehfd changed the title Package request: x11-apps, x11-utils, x11-xserver-utils, xfonts-utils, ... etc. Package request: X11 Utility Apps May 8, 2024
@ehfd ehfd changed the title Package request: X11 Utility Apps Package request: X11 Utility Apps and Remaining Libraries May 8, 2024
@ehfd ehfd changed the title Package request: X11 Utility Apps and Remaining Libraries Package request: X11 Utility Apps and Remaining X11 Libraries May 8, 2024
@ehfd ehfd changed the title Package request: X11 Utility Apps and Remaining X11 Libraries Package request: X11 Utility Apps, X.Org Server, and Remaining X11 Libraries May 10, 2024
@h-vetinari
Copy link
Member

Out of curiosity, what's the use-case for having the binaries (as opposed to just the libraries)? In the bigger picture, while the Wayland future is still going to take a good while, it's coming inexorably, so I'm not sure how much of our scarce resources the X11 ecosystem still merits. 🤔

@ehfd
Copy link
Member Author

ehfd commented May 29, 2024

@h-vetinari

In principle, utilizing the same HPC environment (Conda) researchers use for ML and AI, to deliver rootless, isolated, and portable graphical workloads (Robotics including ROS, Carla, OpenCV, PyMOL, VMD, ParaView, OpenDroneMap, etc.).

HPC facilities now mostly use either Conda or Apptainer (known as Singularity). Other rootless environments such as LinuxBrew (HomeBrew), Pkgsrc, Nix, Spack, and EasyBuild (where the latter two are basically Conda wrappers adding some more packages on their sides) exist.
One example: https://hpc.nmsu.edu/discovery/software/modules/

Out of these, LinuxBrew, Pkgsrc, and Nix have X11 server binaries and the above CLI tools, but only Conda (with Apptainer coming close) is universal to all global HPC facilities and supercomputer centers, and most importantly, considered a first-class citizen of NVIDIA and/or ROCm, either through pip or conda.

The above binaries are required for functional usage of the xorg-xserver binary (needed for XDummy) and xorg-xvfb, as well as VirtualGL (if we are stretching a bit more for headless OpenGL support).

While Wayland is coming fast, the vast majority of HPC facilities are still optimized for X11, and despite Conda already having Wayland libraries (much like X libraries), people are, as of now, not going to start a Wayland server solely inside Conda.

In that sense, Wayland and X11 are similarly limited in Conda.

Moreover, Wayland does not mean that X11 will completely disappear. There will still be legacy applications still on X11 (very prevalent in academic situations), and thus XWayland will be required. Since XWayland is another X server, it comes back to still needing the X.Org CLI tools.

Speaking of which, unlike the libraries, the X.Org CLI tools, as you've said, aren't mandatory in essence, as these feedstocks don't have explicit dependence on other feedstocks. So conversely, they don't need to be built through all sorts of patching for VC or MSYS2 in Windows.

VcXsrv and Xming already do this well (after some meticulous code patches for the whole X11 stack on their sides) for everyone in Windows, and unlike Mac or Linux, Windows doesn't have the ABI and glibc hell which otherwise requires Conda for portability.

Thus, the maintenance strain would be much less if it was only targeted for Mac and Linux because the only things probably required include autoconf boilerplate and a script for automatic build recursion (in my experience writing new feedstocks, >90% of the time spent making X11 feedstocks pass the test comes from patching Windows).
I'm also simply omitting Windows for application feedstocks that depend on X11 when they don't build and have some weird Windows incompatibilities such as fork() (see the PRs for xclip, xsel, xdotool, etc.).

@h-vetinari
Copy link
Member

Thanks for the detailed reply! In principle it's a good idea to provide these things of course, but it might be a substantial effort, and that would need a champion. It sounds like that could potentially be you? 🙃

You might also want to join one of the core calls (biweekly Wednesday 18h UTC, next one is today, agenda not too crammed yet) to discuss the idea and how to approach it.

@ehfd
Copy link
Member Author

ehfd commented May 29, 2024

It sounds like that could potentially be you?

It's not immediate for me in the current sense, but in a year or so if people can wait. It will be in demand eventually. If someone's interested before that, they can do it. Else, I'll do it eventually.

For now, I'm just fully focused on the depending library structures that should land for the Alma8 sysroot launch.
The future structure of mesalib and X11 libraries should be decided before Alma8, then it will benefit this cause some time later.

From conda-forge/cdt-builds#66:

This seems to be the feedstock for libxcomposite: https://github.com/conda-forge/xorg-libxcomposite-feedstock

And libxshmfence is a critical package I forgot that isn't in Conda. This is a CDT package, so should be done ASAP. @hmaarrfk

@h-vetinari
Copy link
Member

I brought this up in the core call to get a feel for what people think about this. Overall there's no issue with doing this, though:

  • we may not be able to sanely ship all binaries, in case where running certain things needs elevated privileges
  • we don't yet really see use-cases in the form of other feedstocks that would want/need to use these
  • to do this at the scale proposed would need long-term commitment from >=2 contributors due to the scope of the maintenance effort

Not discussed directly in the call but from my POV: any of the missing lib* packages should be added and PRs are very welcome for that.

@ehfd
Copy link
Member Author

ehfd commented May 30, 2024

@h-vetinari Great, thank you!

I am also waiting for more potential contributors to contribute to this job.
Perhaps from my own project.

We can wait and see. Meanwhile, I'd thank the core team if they can focus on the CDT refactoring roadmap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

3 participants