@uecker @tauon @mntmn So when you communicate, the reason that's bad is partially due to the bandwidth, but mostly due to the huge delays you incur in order to wait for execution to complete on the GPU, and then to start back up again once you produce more GPU commands. Readback at a fixed latency which exceeds that queuing delay is totally fine, and bandwidth-wise you can yeet something like 20 4k uncompressed images across the pcie link per 60Hz frame, if you really really wanted to.

@uecker @tauon @mntmn nitpick aside: The reason that these are remote buffer management protocols is actually subtly different from bandwidth concerns, it's more about queuing and throughput. the latency between the GPU and the CPU is actually pretty low, and the bandwidth quite high, however we deliberately introduce extra latency in order to allow the GPU to run out-of-lockstep from the CPU to increase throughput. It prevents us from starving the GPU, by ensuring a continuous stream of work.

@uecker @tauon @mntmn Right, I think I see your meaning now. I was just confused because X (as well as some other products) did of course allow you to send actual rendering commands to the remote machine, for them to be executed on the remote rather than the host. And tauon was talking specifically about forwarding gl commands. But if you just consider the direct rendering paths, then they're pretty similar between something like X and waypipe, it's just copying and forwarding the output frames.

@tauon @mntmn @uecker If I have a tool like blender which might have GPU memory requirements of say, 10GB, for a particular scene. Then in order to remote that very first frame, I need to send those entire 10GB all to the client! And then every frame the data in the GPU working set changes, and all that data needs to be sent as well.

@tauon @mntmn @uecker Let me put it this way, when I render frame 1 of a GPU program, in order to execute the GPU commands to produce that frame, I need to have the *entire up to date contents of vram* in the 'client' gpu's vram (or at least memory accessible to the 'client' gpu). That's really hard, and is bounded only by the GPU memory that the application wants to use. But using the 'server' gpu to render a frame, and sending it to the remote, is a bounded, and much smaller amount of work.

@tauon @mntmn @uecker No that's my point, it's *extremely* hard!

@uecker @tauon @mntmn I'm not sure I even understand what you're saying in that case, waypipe does just stream the rendered buffer from the host to a remote client. It just doesn't serialize and send the GPU commands required to render that buffer on the remote client. The latter is very hard, the former is very practical.

@uecker @tauon @mntmn The reason it's flawed imo is that while it will work fine in restricted situations, it won't work in many others. Comparatively, streaming the output always works (modulo latency and quality), and you have a nice dial to adjust how bandwidth and CPU heavy you want to be (and thus latency and quality). If you stream the command stream you *must* stream all the data before rendering a frame, and you likely need to stream some of it without any lossy compression at all.

@uecker @tauon @mntmn Similar, but likely at a narrower scale of latency tolerance. The issue is just the bandwidth v.s. the size of the working set, the GPU is remote (well, unless it's integrated) but PCIe 4 bandwidth is ~300 times greater than you get with a dedicated gigabit link. and vaguely ~15000 times greater than what you might have used to stream a compressed video.

@uecker @tauon @mntmn But like I hinted at before, there's also just issues like applications which just map all the GPU memory into the CPU address space and write it whenever they like (with their own internal synchronization of course). That's *really* hard to deal with, even for tools which trace GPU commands straight to disk. Doing it transparently over the internet is really really really hard.

@uecker @tauon @mntmn The PCIe bus lets us move hundreds of megabytes of data between VRAM and RAM every frame. And so we do that. Our engine also relies on CPU read-back of the downsampled depth buffer from the previous frame, so that's a non-starter, however that's not something you'd run into outside of games, probably.

@uecker @tauon @mntmn we keep gigabytes of constantly changing data in GPU memory. so yes, but unless you want to stream 10GB of data before you render your first frame, then no. (obviously blender is less extreme here, but cad applications still deal with tremendous amounts of geometry, to say nothing of the online interactive path tracing and whatnot)

@uecker @tauon @mntmn me too, i make aaa video games

@uecker @tauon @mntmn For very simple GPU programs you can make it work, but more advanced programs just do not work under a model with such restricted bandwidth between the GPU and the CPU. Plus, as was mentioned up-thread, you still need to somehow compress and decompress those textures online, which is itself a complex task. Plus you still need the GPU power on the thin client to render it. It's very much easier to render on the host, and then compress and transfer the whole framebuffer.

@uecker @tauon @mntmn Unfortunately that's really not how the GPU works at all in the present day, it made more sense back in OpenGL 1.1 when there were pretty straightforward sets of "commands" and limited amounts of data passing between the GPU and the CPU. Nowadays with things like bindless textures and gpu-driven rendering, and compute, practically every draw call can access practically all the data on the GPU, and the CPU can write arbitrary data directly to GPU VRAM at any time.

@uecker @tauon @mntmn it's not really obvious with the default scene, but a 3d program like blender requires a pretty hefty GPU to run the UI (see also any CAD tool, or a game)

@uecker @tauon @mntmn remote rendering for a program which is heavily reliant on the GPU like blender is the exact opposite of why you'd want remoting though. (Plus none of those virtualization things really work so well in the modern day, it's not gl1.1 anymore, the model just doesn't fit)

@uecker @tauon @mntmn for applications like blender there's neither a toolkit, nor any good way to handle forwarding that isn't streaming.