It&#x27;s been annoying to me as an end user that so many basic things require nightly. I use nightly as my main toolchain, but enabling unstable features makes a project nightly-only, which is undesired for crates that don&#x27;t already revolve around the unstable feature.I most often encounter unstable features when I reach for a basic common-sense utility method and discover that it&#x27;s not stable. Like just earlier today I would have reached for bool::toggle which not only is unstable, but is also newly added as of like a month ago! but some unstable methods have been sitting around for years.And now that IntelliJ-Rust is proprietary, I can&#x27;t even make a feature request anymore for the ability to exclude unstable features from the autocomplete. So they will taunt me forever, perfect little helpers just locked away.

Pretty common for Rust to cook things in nightly for a very long time; I wouldn&#x27;t consider it a bad thing, tbh.

The author mentions Rust&#x27;s portable SIMD library [0]. The only issue with portable SIMD is it&#x27;s only available on nightly. I used it in my FFT crate, but we had to switch to the fearless_simd crate in order to get a portable SIMD solution that works on stable [1].[0] <a href="https:&#x2F;&#x2F;doc.rust-lang.org&#x2F;std&#x2F;simd&#x2F;index.html" rel="nofollow">https:&#x2F;&#x2F;doc.rust-lang.org&#x2F;std&#x2F;simd&#x2F;index.html</a>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;linebender&#x2F;fearless_simd" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;linebender&#x2F;fearless_simd</a>

&gt; GPUs aren&#x27;t really SIMD, they&#x27;re SIMT (single instruction multiple thread)False. If they were threads they&#x27;d have their own PC. They do not - only the warp has a PC.&gt; You just write plain scalar code and the hardware model dispatches it efficiently to SIMD execution units.Absolutely not. If you don&#x27;t write coalesced loads, bank-conflict free, predication-free, cooperative code you will get worse than CPU performance.

It&#x27;s not really obvious unless you go in depth of the details on modern GPU architecture. GPUs aren&#x27;t really SIMD, they&#x27;re SIMT (single instruction multiple thread). The silicon looks a lot like SIMD, but the programming model is different.If you go look at AMD&#x27;s ISA docs (they&#x27;re public) you&#x27;ll see you don&#x27;t have the equivalent of a __mm256 register like on x86. Each &#x27;thread&#x27; just deals with single scalar values like int32 of float32. The hardware, however, groups 32 or 64 threads together which all run the same program and runs them together. Each &#x27;thread&#x27; loosely maps to a SIMD lane. The SIMD is implicit, not explicit.The main difference is that the &#x27;SIMD&#x27; execution is somewhat opaque to the program. You just write plain scalar code and the hardware model dispatches it efficiently to SIMD execution units. It&#x27;s not really an abstraction because to extract maximum performance you have to understand how it works. You can use this kind of programming model on a CPU too, Intel did it with [0] ISPC. It&#x27;s a C-like language that has execution semantics similar to GPU shader languages but compiles to regular CPU code, and maps threads to your CPUs SIMD lanes like a GPU.[0] <a href="https:&#x2F;&#x2F;ispc.github.io&#x2F;" rel="nofollow">https:&#x2F;&#x2F;ispc.github.io&#x2F;</a>

64 what? Bits&#x2F;bytes&#x2F;something bigger?

GPU &quot;cores&quot; are basically what a CPU would call SIMD lanes. So a GPU with 1024 &#x27;CUDA cores&#x27; might be structured as 16 relatively independent pieces that a CPU might call a core, each with a 64 wide SIMD unit.

GPUs work on vectors and matrices very often, that&#x27;s what they are good at, so it makes a lot of sense that they can operate with SIMD I think!

There is something very SIMD-coded in GPU programming which is coalesced stores&#x2F;loads, if a warp (32 threads) handles contiguous memory, it will create ~4 transactions instead of 32.

Welcome to the lucky 10,000! SIMD is actually a pretty integral part of how GPUs are able to work efficiently, it&#x27;s part of why there&#x27;s such a strong focus on branchless programming in the field.

My heard hurts - i was stupid enough to think that SIMD was a CPU only thing - I don&#x27;t understand why it would be ported to GPU - huge kudos to managing to surprise me

This is basically the goal of fearless_simd, but of course achieving the same level of maturity will take time.

I would love to have an open source Rust SIMD library with the scope and maturity that
<a href="https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;highway" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;highway</a> brings to C++.

The capabilities of various SIMD ISAs don&#x27;t have enough intersection to be portable outside of relatively trivial cases. Many of the somewhat unique capabilities are load-bearing, so you want to use them on architectures that support them. Taken in whole, someone who cares about performance would be using different data structures and algorithms depending on the specific SIMD architecture and that is nearly impossible to abstract in a library. Too many important but complex details are idiosyncratic to the implementation.Another way of looking at it is that our programming environments are not sufficiently powerful and expressive to create the necessary abstractions to make SIMD truly portable.

.NET and Java have three levels of SIMD support, Go&#x27;s ongoing efforts, and does the upcoming C++ standard.Autovectorization, depending on compiler&#x27;s cleverness, really portable SIMD operations, and then the CPU specific SIMD ones.So this should be perfectly doable in crate that advertises as portable, while leaving the non portable stuff to another crate.

It should really be read&#x2F;advertised as &quot;portabler SIMD&quot;. It beats hoping the compiler autovectorizes everything well forever or writing architecture specific code manually again but is going to compromise on average performance vs platform specific SIMD.

Except you can&#x27;t use this in actual code, because either, as is the case in this example with f32x32, you run out of registers and spill all over the place.
Or you aren&#x27;t using your full vector register or could&#x27;ve gotten better performance by &quot;unrolling&quot; more often for the larger vectors.If you use f32x16 (the avx-512 wisth), SSE now effectively has 4 registers to work with and will spill when doing anything beyond the most simple stuff.The default should imo be relative to the native register width, so you can do 1x, 2x or sometimes 4x the native width, depensing on your register preasure.

&gt; They specifies a constant SIMD width so it&#x27;s non-portable.This is incorrect, you can use vectors wider than native SIMD width and the compiler will break them down to register size of the target cpu.In fact it&#x27;s sometimes better to used wider than native width, in some applications I see 20% better throughput with f32x16 (512 bits) on an AVX2 CPU (256 bits). It is kinda like loop unrolling it.

Sure but there&#x27;s no real way to use that in a portable way, at least not a way that maximises performance on every CPU you run it on. That&#x27;s pretty much impossible at the moment.

Go&#x27;s implementation is vector size independant <a href="https:&#x2F;&#x2F;pkg.go.dev&#x2F;simd@master" rel="nofollow">https:&#x2F;&#x2F;pkg.go.dev&#x2F;simd@master</a>

The create is called portable_simd.There is no reason a portable_simd relu_dot implemention should need to specify the SIMD width.But the design and documentation of portable_simd makes the fixed size syntactically easy&#x2F;the default and the width agnostic code harder.

Counter question: why shouldn&#x27;t it be portable?It&#x27;s definitely a 80% solution where you occasionally need to drop down to intrinsics (at zero runtime perf cost) for CPU specific instructions.But just having vector types, arithmetic, swizzling, loads and stores will go a long way for basic tasks.And with generics you can write code that is type and width agnostic. No need to rewrite your code of you want to go from SSE to AVX512, just change from f32x4 to f32x16 (or use generics) and you are done.

Why should it be portable? Honest question.SIMD seems to me, to be very platform specific. Maybe there are times one SIMD unit is not anothers&#x27; SIMD unit?

I love how ever example of portable SIMD isn&#x27;t portable.They specifies a constant SIMD width so it&#x27;s non-portable. Well, not performance portable, but why are we using SIMD again?

Do you have examples of complex algorithms running on the gpu with rust with competative performance? Radix sort might be a good one to start with

The tentative plan is to open source all the compiler and `std` bits with our products built on top (compilers are not good businesses). More about our products coming in the next couple of months!

What is vectorware&#x27;s business model? Are you planning to sell support&#x2F;consulting to companies using your stack? Or are you looking to sell licenses to your tool? Or something else?

Didn&#x27;t want to go into crazy detail in the post.Each family of operations is a trait parameterized by the operation itself:<pre><code> pub trait EvaluateReduction&lt;Operation, T&gt;: LaneEvaluator {
 &#x2F;&#x2F;&#x2F; Reduce one distributed definition to an ordinary uniform scalar.
 fn evaluate_reduction(&amp;self, value: LaneValue&lt;Self, role::Distributed, T&gt;) -&gt; T;
 }

</code></pre>
Call sites name the operation:<pre><code> let one = evaluator.splat::&lt;Splat, _&gt;(1_u32);
 let two = evaluator.splat::&lt;Splat, _&gt;(2_u32);
 let three = evaluator.binary::&lt;Add, _&gt;(one, two);

 let total = evaluator.reduce::&lt;Sum, u32&gt;(three); &#x2F;&#x2F; a uniform u32
 let running = &lt;Executor as EvaluateScan&lt;Scan&lt;Sum, Exclusive&gt;, u32&gt;&gt;::scan(&amp;evaluator, three);

</code></pre>
Operations like Sum, Max, ReduceXor, Inclusive, and Exclusive are all distinct types.As mentioned in the post, execution shape is typed too. A static shuffle takes its control as a type-level constant, and the shuffle mode constrains which controls are expressible:<pre><code> &#x2F;&#x2F; Shift down one lane, keeping our own value where the source is inactive.
 let down = &lt;Executor as EvaluateShuffle&lt;Shuffle&lt;Down&gt;, DownOrSelf&lt;1&gt;, u32&gt;&gt;::shuffle(&amp;ev, v);
 &#x2F;&#x2F; Broadcast from lane zero.
 let bcast = &lt;Executor as EvaluateShuffle&lt;Shuffle&lt;Broadcast&gt;, WarpLane&lt;0&gt;, u32&gt;&gt;::shuffle(&amp;ev, down);
 &#x2F;&#x2F; Butterfly exchange with the neighbor one bit away.
 let bfly = &lt;Executor as EvaluateShuffle&lt;Shuffle&lt;Xor&gt;, Butterfly&lt;1&gt;, u32&gt;&gt;::shuffle(&amp;ev, bcast);

</code></pre>
For an example of errors caught, a warp-scoped executor for a device-scoped barrier is a compile error:<pre><code> &lt;ScopedWarpExecutor&lt;&#x27;_, WarpUniform&gt; as EvaluateBarrier&lt;Barrier&lt;Device&gt;&gt;&gt;::barrier(evaluator)
 &#x2F;&#x2F; error[E0277]: the trait bound `Device: NvptxBarrierScope` is not satisfied
 &#x2F;&#x2F; help: the trait `NvptxBarrierScope` is implemented for `Warp`

</code></pre>
Strip mining is typed on the amount of work and the lane capacity, and it hands back one chunk at a time along with the predicate saying which lanes live in that chunk:<pre><code> &#x2F;&#x2F; Six work items across four active lanes: two chunks, based at 0 and 4.
 &lt;Executor as EvaluateStripMine&lt;StripMine, (WorkItems, ActiveLanes&lt;StripMined&lt;4&gt;&gt;), i32&gt;&gt;::
 for_each_strip_mined(
 &amp;evaluator,
 (WorkItems::new(6)?, ActiveLanes::new(4)?),
 |index, active| {
 &#x2F;&#x2F; ...
 },
 );

</code></pre>
Hopefully that gives the flavor of it.

I&#x27;m confused too. How does this fit between these approaches for paraellization:<pre><code> - CUDA kernels and Tiles (e.g. Cudarc, cuda-oxide, rust-gpu etc) - SIMD on the GPU. (E.g. as in the title...)
 - CPU SIMD using avx or SSE instructions (And probably thin wrappers for vectors so you can have sane syntax). Or the maybe-upcoming core simd which should abstract over architecture-specific instructions. Magic floats etc which do 4-16 computations at once, but are a bit clumsy to work with
 - Rayon thread pools - arbitrary parallel computations, including SIMD, one per CPU core.
</code></pre>
It looks like from the code samples like maybe a cleaner syntax for writing code on the GPU than CUDA kernels? E.g. without mucking with serialization, host and device by abstracting over it? And inspired by core::simd. (Good choice if so, in the interest of standardizing on syntax; I did this for my x86 SIMD vector&#x2F;quaternion lib as well)

The post is kind of vague on the IR you&#x27;re targeting. Can you give some examples of what the SIMD-ized IR looks like, and how it maps to the target PTX?

It is not currently available but we intend to make it available after we launch our products.

This is really cool! It sounds like y&#x27;all have a compiler fork that you are using to make this work. I wanna tinker with this, is your compiler available?

The goal of this work is to run existing unmodified CPU libraries (which may use core::simd) on the GPU. If you are manually writing ML-shaped workloads, it doesn&#x27;t add any value over writing with tech like torch&#x2F;tensorflow&#x2F;jax which are custom built for those use-cases (except maybe familiarity if you are a CPU programmer).

If you have to express your computation using an &quot;array programming DSL&quot; with things like scan and gather anyways - why not opt to use torch&#x2F;tensorflow&#x2F;jax or anything else that targets MLIR? An example of writing a relu using an embedded array DSL is really not helping your case either - that&#x27;s exactly the problem that these other solutions mentioned above are successfully solving for the past ~15y (starting with theano etc). Not sure what this brings to the table - doing that AoT instead of at runtime?

Part of our thesis is that decent GPUs are in every shipping device and most software doesn&#x27;t use them and should.

Given the massive demand for GPUs for LLMs, what sorts of work do you expect to economically benefit from utilizing GPUs more?

The goal is to use similar abstractions and code across both the CPU and GPU where it makes sense.

Hm is the intent to one day replace the CPU?

Any thoughts about SIMD-related crates?

Very interesting. 
But GPU programming gets complicated when you start doing 3d computation on very large data, will be interesting to see how tensor abstraction is built on top of this. Another point is that this is using fixed-width SIMD vectors; unless there is a way to compute this statically based on available GPU info, performance will always be left on the table.

The main problem with SIMD is it is a complex subject, even if not that good, autovectorization wins over what most common devs know about SIMD.Well now you could in theory AI generate SIMD, which will be vibe coded, as those devs have no idea of its correctness.

I did see the article about that too. Don&#x27;t know about &quot;always&quot; since there are applications like web backends where you&#x27;re never going to add arrays of floats or something. Even if it&#x27;s data science stuff, if that&#x27;s in Python, Numpy is doing the SIMD for you.

SIMD is actually underrated still. Programmers should always be thinking about it. It&#x27;s a free 4x in a lot of cases

I’m wondering if AI has made SIMD intrinsics much more approachable for many and so there are just more people working on abstractions for their workflow of choice right now. There’s probably a lot of code out there that could benefit from SIMD but the effort to actually use it was too high for the return.

Yes, this kind of thing seems to happen quite often. Popular posts spurring further posts on a theme.

Maybe things being on the front page reminds others? After seeing something, sometimes you can have ideas relating to it for a while.

I&#x27;ve noticed a lot of articles about SIMD on the HN front page. That&#x27;s cool, but just wondering, is there some reason this is more in focus lately?

Really exciting work and great write up, thanks a lot and all the best to your startup!`core` instead of `std` is great too!This will become useful in one of my sideproject where I use bitmaps to speed up pathfinding, exited to try it out!

Congrats to the Rust-GPU folks! Nice to see the good work flowing.

It&#x27;s just a way for us to add minutia and details that most don&#x27;t care &#x2F; need to know about. There are three audiences we try to make the posts accessible for: Rust people who don&#x27;t know about GPUs, GPU people who don&#x27;t know about Rust, and non-Rust non-GPU people. The toggle lets knowledgable readers go &quot;wait, what about...&quot; and hopefully the toggle answers it.

I don&#x27;t get what&#x27;s the value of it not being enabled by default what does the toggle get us, really? Maybe I don&#x27;t understand web design and it makes it harder to read for some, I am dyslexic and never had any issues.

Love the pendantic mode setting on the website

Could also just be trolls attracted by the Rust topic.

No idea, but it seems HN needs POW challenges.

Hey - this is probably off-topic&#x2F;meta, but what is going on with the comments here? Is it bots?

Rust SIMD on the GPU