Should GPU shaders be considered firmware?

[Hakan Bayındır]

Hi Stephan,

In first blush, I don’t think shaders should be considered firmware. They are not used to enable the hardware, but they’re just programs which work on pixels to perform some functions. These functions work like “suggested/recommended” packages which enable more functions with the hardware, but do not render it inoperable when they’re absent.

GPU firmwares used by AMD/NVIDIA are pushed to the card, or stay there indefinitely and enable the hardware, initialize it let it function as a graphics card. Similar to other firmware (printer, scanner, WiFi, Ethernet, etc.).

But shaders are not them, and they can be considered as software. Like CUDA or “compute” Kernels which run on the GPU directly (preferably after compilation) and consume input and generate output.

Of course this is my two cents,

Best regards,

H.

Hello everyone
The “drivers” for hardware video decoders on Intel GPUs have been split
into free and non-free packages.
https://packages.debian.org/en/sid/intel-media-va-driver
https://packages.debian.org/en/sid/intel-media-va-driver-non-free
The difference between the two is that the Intel source code contains
blobs (byte arrays in the C source) with pre-compiled GPU shaders.
The source code for the pre-complied shaders is not included and does
not appear to be available under a free license.
media_driver/agnostic/Xe_M/Xe_HPM/codec/kernel/XE_HPM_VC1_OLP.c
media_driver/agnostic/Xe_M/Xe_HPM/codec/kernel/Xe_Hpm_Film_Grain.c
media_driver/agnostic/Xe_M/Xe_XPM/codec/kernelisa/Xe_XPM_Film_Grain.c
media_driver/agnostic/Xe_M/Xe_XPM/codec/kernelisa/Xe_XPM_VC1_OLP.c
media_driver/agnostic/Xe_M/Xe_XPM/vp/kernel/cmfcpatch/igvpkrn_xe_xpm_cmfcpatch.c
media_driver/agnostic/Xe_M/Xe_XPM/vp/kernel/igvpkrn_isa_xe_xpm.c
media_driver/agnostic/Xe_M/Xe_XPM/vp/kernel/igvpkrn_xe_xpm.c
media_driver/agnostic/Xe_M/Xe_XPM_plus/codec/kernel/Xe_XPM_plus_Film_Grain.c
media_driver/agnostic/Xe_M/Xe_XPM_plus/vp/kernel/cmfcpatch/igvpkrn_xe_xpm_plus_cmfcpatch.c
media_driver/agnostic/Xe_M/Xe_XPM_plus/vp/kernel/igvpkrn_xe_xpm_plus.c
media_driver/agnostic/gen11/codec/kernel/igcodeckrn_g11.c
media_driver/agnostic/gen11_icllp/codec/kernel/igcodeckrn_g11_icllp.c
media_driver/agnostic/gen11_icllp/vp/kernel/igvpkrn_g11_icllp.c
media_driver/agnostic/gen11_icllp/vp/kernel/igvpkrn_isa_g11_icllp.c
media_driver/agnostic/gen12/codec/kernel/igcodeckrn_g12.c
media_driver/agnostic/gen12/codec/kernelisa/*.c
media_driver/agnostic/gen12_tgllp/vp/kernel/cmfc/igvpkrn_g12_tgllp_cmfc.c
media_driver/agnostic/gen12_tgllp/vp/kernel/cmfccmlpch/igvpkrn_g12_tgllp_cmfccmlpch.c
media_driver/agnostic/gen12_tgllp/vp/kernel/cmfcpatch/igvpkrn_g12_tgllp_cmfcpatch.c
media_driver/agnostic/gen12_tgllp/vp/kernel/igvpkrn_g12_tgllp.c
media_driver/agnostic/gen12_tgllp/vp/kernel/igvpkrn_isa_g12_tgllp.c
media_driver/agnostic/gen12_tgllp/vp/kernel/swsb/igvpkrn_g12_tgllp_swsb.c
media_driver/agnostic/gen8/codec/kernel/igcodeckrn_g8.c
media_driver/agnostic/gen8/vp/kernel/igvpkrn_g8.c
media_driver/agnostic/gen9/codec/kernel/igcodeckrn_g9.c
media_driver/agnostic/gen9/vp/kernel/igvpkrn_g9.c
media_driver/agnostic/gen9/vp/kernel/igvpkrn_isa_g9.c
media_driver/agnostic/gen9_bxt/codec/kernel/igcodeckrn_g9_bxt.c
media_driver/agnostic/gen9_cml/vp/kernel/igvpkrn_g9_cml.c
media_driver/agnostic/gen9_cml/vp/kernel/tgp/igvpkrn_g9_cml_tgp.c
media_driver/agnostic/gen9_cmpv/vp/kernel/igvpkrn_g9_cmpv.c
media_driver/agnostic/gen9_kbl/codec/kernel/igcodeckrn_g9_kbl.c
media_softlet/agnostic/Xe_R/Xe_HPG_Base/vp/kernel/cmfcpatch/igvpkrn_xe_hpg_cmfcpatch.c
media_softlet/agnostic/Xe_R/Xe_HPG_Base/vp/kernel/igvpkrn_isa_xe_hpg.c
media_softlet/agnostic/Xe_R/Xe_HPG_Base/vp/kernel/igvpkrn_xe_hpg.c
media_softlet/agnostic/Xe_R/Xe2_HPG/vp/kernel/cmfcpatch/igvpkrn_xe2_hpg_cmfcpatch.c
media_softlet/agnostic/Xe_R/Xe2_HPG/vp/kernel/igvpfc_420PL3_input_xe2.cpp
media_softlet/agnostic/Xe_R/Xe2_HPG/vp/kernel/igvpfc_444PL3_input_xe2.cpp
media_softlet/agnostic/Xe_R/Xe2_HPG/vp/kernel/igvpfc_common_xe2.cpp
media_softlet/agnostic/Xe_R/Xe2_HPG/vp/kernel/igvpfc_fp_xe2.cpp
media_softlet/agnostic/Xe_R/Xe2_HPG/vp/kernel/igvpkrn_xe2_hpg.c
Should they be considered non-free firmware rather than non-free
software?
1. The shaders are loaded into the GPU, very much like firmware.
2. They are not linked into any libraries or applications.
3. They are not loaded into the kernel (like kernel module or eBPF).
4. They are not running on the CPU, neither in kernel space, user
space or in a bytecode interpreter.
5. They are required by the operating system or applications to make
use of hardware capabilities.
1. The shaders are mixed up with library code which load them into
the GPU.
2. That library code is linked into libraries and applications and
running on the CPU. (How ever, there appears to be consensus that
the library code is free, only the shaders are not.)
3. GPU shaders can have various purposes and origins other than the
hardware vendor.
Regards
Stephan

[Marco d'Itri]

In first blush, I don’t think shaders should be considered firmware.
They are not used to enable the hardware,

I am not taking a position about shaders at this point, but I will note
that this has never been the definition of firmwares.
Firmwares are software which runs outside of the OS on a different CPU.

[Hakan Bayındır]

Post by Hakan Bayındır
In first blush, I don’t think shaders should be considered firmware. They are not used to enable the hardware,

I am not taking a position about shaders at this point, but I will note that this has never been the definition of firmwares.
Firmwares are software which runs outside of the OS on a different CPU.

You’re right, but many if not all firmware runs all the time the unit is powered on. Shaders are transient software which run on the GPU for a limited time, e.g. when you’re encoding a video.

If we think that a shader is a firmware, any software running on the second socket on a system is also a firmware, since the program is running on a different CPU w.r.t. to Kernel.

--
ciao,
Marco

Cheers,

H.

[Marco d'Itri]

Post by Hakan Bayındır
If we think that a shader is a firmware, any software running on the
second socket on a system is also a firmware, since the program is
running on a different CPU w.r.t. to Kernel.

This is not how Linux actually works.

[Hakan Bayındır]

Sent from my iPhone

Post by Marco d'Itri

Post by Hakan Bayındır
If we think that a shader is a firmware, any software running on the second socket on a system is also a firmware, since the program is running on a different CPU w.r.t. to Kernel.

This is not how Linux actually works.

Of course, this is the point. So the shaders.

Post by Marco d'Itri
--
ciao,
Marco

Cheers,

H.

[Stephen Kitt]

Post by Hakan Bayındır

Post by Marco d'Itri

Post by Hakan Bayındır
If we think that a shader is a firmware, any software running on the
second socket on a system is also a firmware, since the program is
running on a different CPU w.r.t. to Kernel.

This is not how Linux actually works.

Of course, this is the point. So the shaders.

I think Marco’s point is that the kernel runs on all CPUs in a system, not
just the first one.

Regards,

Stephen

[Bastian Blank]

1. The shaders are loaded into the GPU, very much like firmware.

Shaders are not what makes the device function. You need the firmware
first.

2. They are not linked into any libraries or applications.

Actually they are. They provide a specific interface that the library
expects, so they are linked to this library.

(Okay, so do the firmwares for the proprietary nvidia driver, which are
distinct to the "free" counterparts)

5. They are required by the operating system or applications to make
use of hardware capabilities.

Nope, they are not required to run the hardware. They implement a
specific capability on top of the existing hardware. And also you can
run multiple instances of it (pseudo) concurrently on the same hardware.

You can not replace the firmware without resetting the hardware
completely to a known state.
4. Shaders are "general purpose" execution on the GPU hardware.

We also had a (in Debian only partially supported) case in the past: the
PS3 contained two distinct and incompatible PowerPC core setups. Just
like this case of the GPU, you need special support to start code on
those separate CPU cores, called SPE. With the arguments brought up
here, code run on those SPE could also be firmware, which makes no
sense.

But if you think further: what about FPGA programs. Would that be
firmware? FPGA are "general purpose" devices and those programs can do
everything in reason.

What about an alternative "firmware" for a GPU that just outputs
the Mandelbrot set, but does not longer make it a usable GPU?

So firmware is a piece of code that
- runs exclusively on a piece of hardware,
- provides an agreed on interface to a kernel oder user mode driver.

Bastian

[Stephan Verbücheln]

GPU shaders should be clearly respected as software (and not
hardware).  Therefore, `intel-media-va-driver-non-free` is rightly in
the Debian archive `non-free`.

First of all, I completely agree that:
1. shaders are computer programs
2. the shaders in that package are unfree

They are not unfree because of license restrictions but because of
missing sources for some blobs.

Please note that Debian does not only require sources for computer
programs. Generated data should also preferably include all sources
that allow developers to study, modify and generate their own versions
of that data.

There are many edge cases. What about JavaScript executed in the web
browser? What about file formats which are de facto executable programs
in some kind of virtual machine such as PostScript or TrueType fonts?

    Machine/Computer
   ,-----------------------------------------,
   | optional non-free/proprietary software  |
   |-----------------------------------------|
   | Free and open-source software           |
   |-----------------------------------------|
   | Hardware (maybe non-free/proprietary)   |
   '-----------------------------------------'

I do not agree with that. Shaders do not run “on top” like apps or
JavaScript in the web browser. They are loaded into the GPU just like
the firmware and are executed there.

The question must be: Is firmware to be considered as hardware or
   * Should be GPU shaders considered as hardware or software?

That is not the question. Firmware is not hardware either.

   1. Firmware needs to be developed by the chip-vendor itself.
      (Hardware from companies point of view)
   1. GPU shaders will not be developed exclusively by the chip
vendor
      itself.  In contrast, they providing SDKs for development.

This is also true for these GPU shaders.
1. They are developed by the hardware vendor and its community.
2. They are not developed by a software company like shaders for a
game engine.
3. The whole purpose is to enable hardware capabilities, in this
case video encoders and decoders.
4. The same shaders developed by the GPU vendor can be used by any
video player or library.
5. The available libraries in user space are free.

The answer is: The path is the goal.  The 3-layer machine/computer
model we agree all, hopefully.  We can map the companies-/logistic-
point of view very well to that 3-layer model, imho.

That model does not make sense to me. I want firmware and shaders to be
free just as software. But there is a practical difference between
software which we can replace by alternatives and firmware which is
required by the hardware.

The only way to achieve the end goal is by only buying hardware which
works without non-free firmware. This is the long-term goal but not
always possible in the short term.

There are libreboot, coreboot etc., but they all require unfree blobs
from hardware vendors. This might change with more RISC-V options
coming to market, but the situation is as it is.

In the meantime, we have non-free-firmware for people who want/need it.
It will exists regardless of whether it is included in the installation
media or not.

From the information security and privacy perspective, those shaders
are also comparable with firmware.
1. Your wireless firmware runs on the network device. It does not
have access to your kernel or any other thing running in main CPU
and memory. It can only see the network traffic that you route
through that interface.
2. Your GPU firmware runs on the graphics card. It does not have
access to your kernel or any other thing running in main CPU and
memory. It can only see information that you send to this
graphics card for display.
3. The shaders run on the GPU. They do not have access to your
kernel or any other thing running in main CPU and memory. They
can only see information that you send to this graphics card for
decoding and encoding.

My point is that these particular shaders could be argued to belong to
non-free-firmware and not non-free.

Regards
Stephan