I have 384TB of ECC DDR4 across two blades with 4 CPUs for a combined core count of 96.
-
@GerardThornley @cursedsql @bob_zim @SecurityWriter
Or you want to place it in an environment where it has to deal with heavy vibrations. Like on a moving trolly or in a vehicle or ... there are multiple reasons for this. It may even just be because you need high random IO speeds...
And the sizing also depends on what you're using it for. Like e.g. if you get your data in to the system in infrequent busts but at multiple TB/s and you've to cache it until it is synced even to SSDs, well
@GerardThornley @cursedsql @bob_zim @SecurityWriter
(The later was an example from scientific environments. I think it was CERN but I'm not sure...)
-
@agowa338 Cards like that exist, but they don’t hold thousands of DIMMs.
@bob_zim But PCIe lane splitters and extenders also exist. And I don't know what the highest archivable density of these cards currently is.
I so far have only had one old one in my hands and seen them in slides in class at my job training about 10 years ago (they were mentioned as accelerator cards primarily used for things like MS Dynamics and SAP databases)...
-
@GerardThornley @cursedsql @bob_zim @SecurityWriter
Or you want to place it in an environment where it has to deal with heavy vibrations. Like on a moving trolly or in a vehicle or ... there are multiple reasons for this. It may even just be because you need high random IO speeds...
And the sizing also depends on what you're using it for. Like e.g. if you get your data in to the system in infrequent busts but at multiple TB/s and you've to cache it until it is synced even to SSDs, well
@agowa338 @cursedsql @bob_zim @SecurityWriter Yep, those are also possibilities. I described what I think is most probable given the information available and scenarios I've seen, but yeah, there are reasons it might be a less typical setup, or my knowledge might be out of date.
-
@GerardThornley @cursedsql @bob_zim @SecurityWriter
(The later was an example from scientific environments. I think it was CERN but I'm not sure...)
@agowa338 @cursedsql @bob_zim @SecurityWriter Yeah, that's sounds pretty plausible for things like the LHC experiments.
-
@agowa338 @cursedsql @bob_zim @SecurityWriter Yeah, that's sounds pretty plausible for things like the LHC experiments.
@agowa338 @cursedsql @bob_zim @SecurityWriter For the vehicle scenario, I know modern trains have a lot of sensors on, and I don't know the sampling rate, but they do only have short windows to upload the data they're in their terminal station. I'm not sure I can believe them needing TBs, though.
On the other hand, a F1 team would probably combine transport (though presumably powered down) with need for high bandwidth and low latency.
-
@bob_zim But PCIe lane splitters and extenders also exist. And I don't know what the highest archivable density of these cards currently is.
I so far have only had one old one in my hands and seen them in slides in class at my job training about 10 years ago (they were mentioned as accelerator cards primarily used for things like MS Dynamics and SAP databases)...
@agowa338 It’s more about the physical space for the cards. Most hold eight DIMMs. Holding 3072 would take 384 cards. That’s nearly a full rack just for the RAM cards, not counting the persistent drives. No way would you run that with even two entire blade frames, let alone two blades.
-
@agowa338 It’s more about the physical space for the cards. Most hold eight DIMMs. Holding 3072 would take 384 cards. That’s nearly a full rack just for the RAM cards, not counting the persistent drives. No way would you run that with even two entire blade frames, let alone two blades.
Hmm true. Makes it quite unlikely.
However I want to push back on one thing. It would still be theoretically possible, as PCIe fabrics and such also exist.
But yea, probably not then. I didn't do the math for the space requirements....
-
Don't forget to mention AI
️@stekopf @SecurityWriter I don't think you have to, it's all they can think of anyway nowadays
-
I have 384TB of ECC DDR4 across two blades with 4 CPUs for a combined core count of 96.
It powers a fully populated 192 disk solid state SAN.
I was told it was old and in need of replacing, but apparently now it’s worth more than the GDP of the UK.
Can’t afford to run it (or hear my thoughts when in the vicinity)… but I can sit atop it like a fucking dragon.
And I will.
@SecurityWriter 384TB of RAM is a lot. Perhaps you meant GB?
-
Don't Breath Fire !!!! you evil dragon !!!
@moppi @SecurityWriter we're not all like that
-
Hmm true. Makes it quite unlikely.
However I want to push back on one thing. It would still be theoretically possible, as PCIe fabrics and such also exist.
But yea, probably not then. I didn't do the math for the space requirements....
@agowa338 Also worth noting 128 GB DDR4 DIMMs are almost universally “load-reduced”, or LR-DIMMs. Those have significantly higher latency than normally expected from RAM, though still lower than persistent storage. For low-latency RAM, you’re generally limited to 64 GB per DIMM, so double everything.
Then you have the PCIe cabling. Each card needs at least two dedicated lanes, otherwise contention wrecks your latency and you may as well use flash. That’s 768 or 1536 twinax bundles.
Lots of RAM is possible, but it takes *bonkers* amounts of space and some very exotic hardware.
-
I have 384TB of ECC DDR4 across two blades with 4 CPUs for a combined core count of 96.
It powers a fully populated 192 disk solid state SAN.
I was told it was old and in need of replacing, but apparently now it’s worth more than the GDP of the UK.
Can’t afford to run it (or hear my thoughts when in the vicinity)… but I can sit atop it like a fucking dragon.
And I will.
@SecurityWriter I never thought about quantifying the perching potential of my computer hardware ...
I have a physical treasure trove, of a sort!
-
I have 384TB of ECC DDR4 across two blades with 4 CPUs for a combined core count of 96.
It powers a fully populated 192 disk solid state SAN.
I was told it was old and in need of replacing, but apparently now it’s worth more than the GDP of the UK.
Can’t afford to run it (or hear my thoughts when in the vicinity)… but I can sit atop it like a fucking dragon.
And I will.
@SecurityWriter I'm not gonna lie, I'm a little turned on.....
More than a little even.
-
I have 384TB of ECC DDR4 across two blades with 4 CPUs for a combined core count of 96.
It powers a fully populated 192 disk solid state SAN.
I was told it was old and in need of replacing, but apparently now it’s worth more than the GDP of the UK.
Can’t afford to run it (or hear my thoughts when in the vicinity)… but I can sit atop it like a fucking dragon.
And I will.
@SecurityWriter@infosec.exchange king of the ram
-
E elbekai@norden.social shared this topic
