Jonathan Berte’s Post

Unlocking GPU & Multi-Core Performance on Edge Devices: Experimenting with 𝗢𝗽𝗲𝗻𝗔𝗖𝗖 Are you 𝗺𝗼𝗱𝗲𝗿𝗻𝗶𝘇𝗶𝗻𝗴 𝗹𝗲𝗴𝗮𝗰𝘆 𝗖/𝗖++ 𝗰𝗼𝗱𝗲 on 𝗲𝗱𝗴𝗲 𝗱𝗲𝘃𝗶𝗰𝗲𝘀 like the 𝗝𝗲𝘁𝘀𝗼𝗻 AGX Orin? It is not always necessary to go ahead with a brute-force CUDA rewrite! Even a 𝘁𝗿𝗶𝘃𝗶𝗮𝗹 𝗯𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸𝗶𝗻𝗴/𝗽𝗿𝗼𝗳𝗶𝗹𝗶𝗻𝗴 is surprisingly inline with what we commonly see in production-grade codebases. Here are three common approaches that get profiled/benchmarked: 1. 𝗖𝗣𝗨 𝗢𝗻𝗹𝘆 (optimizations without ACC or CUDA) 2. Decorating code with 𝗢𝗽𝗲𝗻𝗔𝗖𝗖 based pragma directives 3. Rewrite as a 𝗖𝗨𝗗𝗔 program Obviously OpenACC ain’t 𝗻𝗼 𝘀𝗶𝗹𝘃𝗲𝗿 𝗯𝘂𝗹𝗹𝗲𝘁; however our results show that 𝗢𝗽𝗲𝗻𝗔𝗖𝗖 can provide a significant speedup of ~𝟱× compared to CPU-only serial computation, acting as the ideal 𝗺𝗶𝗱𝗱𝗹𝗲 𝗴𝗿𝗼𝘂𝗻𝗱 migration path. Of course a brute-force CUDA rewrite did come up with a ~10x speedup. OpenACC allows you to leverage GPU or multi-core ARM capabilities using simple pragmas, 𝗼𝗳𝗳𝗲𝗿𝗶𝗻𝗴 𝘀𝗰𝗮𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗰𝗼𝗺𝗽𝗹𝗲𝘅𝗶𝘁𝘆. OpenACC works on both 𝗱𝗶𝘀𝗰𝗿𝗲𝘁𝗲 𝗚𝗣𝗨𝘀 and 𝗖𝗣𝗨-𝗼𝗻𝗹𝘆 𝗔𝗥𝗠 𝘀𝘆𝘀𝘁𝗲𝗺𝘀 (e.g. Jetson AGX Orin), making it an invaluable tool to 𝘂𝗻𝗹𝗼𝗰𝗸 𝗺𝘂𝗹𝘁𝗶-𝗰𝗼𝗿𝗲 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 with minimal code restructuring/refactoring (before you decide and go-ahead with a complete CUDA rewrite or the 𝘀𝘁𝗱::𝗲𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻::𝗽𝗮𝗿 approaches (unsupported by nvc++ as on date)). Read the full document to see how OpenACC can accelerate Sensor Data Processing, Image Processing, and numerical code with 𝗦𝗶𝘅 𝗘𝗮𝘀𝘆 𝗣𝗶𝗲𝗰𝗲𝘀. https://lnkd.in/g4H4cyht #sixeasypieces #gpu #acc #openacc #pragma #cplusplus #c++ #nvcc #nvc++ #hpcsdk #nvidiahpcsdk #nvidia #onaquest #Quest1

Microsoft has made it possible to 𝗿𝘂𝗻 𝗹𝗮𝗿𝗴𝗲 𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗺𝗼𝗱𝗲𝗹𝘀 𝗼𝗻 𝗖𝗣𝗨𝘀 (not just GPUs) with 𝗕𝗶𝘁𝗡𝗲𝘁, its 1-bit inference framework that delivers multi-fold speedups and significant energy savings. BitNet (bitnet.cpp) is designed for fast inference of 1.58-bit models on both CPUs and GPUs. It achieves up to 𝘀𝗶𝘅 𝘁𝗶𝗺𝗲𝘀 𝗳𝗮𝘀𝘁𝗲𝗿 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 and reduces energy use by as much as 𝟴𝟬%. Impressively, it can even run a 100-billion-parameter model on a single CPU at near human reading speeds. BitNet marks an important step toward more efficient and accessible AI. It shows that high-performance language models can run effectively on local or edge devices, without the need for heavy GPU infrastructure.  👉 Check it out here: https://lnkd.in/eSe2eQBb #AI #ArtificialIntelligence #DeepLearning #GenerativeAI #LLM #LargeLanguageModels #CPUComputing #GPUs #BitNet #MicrosoftAI #AIOptimization

🤯 Mind Blown! Running LLMs Locally Just Got a CPU Upgrade! 🤯🚀 Microsoft's open-sourcing of BitNet.cpp is revolutionizing how we run Large Language Models! Now, you can run powerful LLMs locally on your CPU with unprecedented efficiency. This C++ inference engine for 1-bit LLMs means: ✅ Significantly faster inference speeds compared to traditional methods. ✅ Drastically reduced energy consumption, making AI more sustainable. ✅ The ability to run large models on standard CPUs, including laptops and edge devices, without needing expensive GPUs. BitNet.cpp enables local, accessible, and efficient AI inference for everyone. This is a massive step towards democratizing access to powerful language models and fostering innovation. Link: https://lnkd.in/g4miAEf2 #AI #LLMs #BitNet #CPUInference #OpenSource #TechInnovation #MachineLearning #DeepLearning

sn-news: #ai #ml #llm #hmc BitNet - Inference framework for 1-bit LLMs offering a suite of optimized kernels supporting fast and lossless inference of 1.58-bit models on CPU and GPU (NPU coming next) https://lnkd.in/d5MEaSvk

GPUs don’t play by Kubernetes’ rules. They can’t be overcommitted like CPUs, lack true memory isolation, and often retain state between workloads. When clusters go multi-tenant, these gaps turn into bottlenecks for utilization, security, and performance. That mismatch is now one of the toughest challenges in platform engineering for AI. Daniele Polencic and Saiyam Pathak have written a new technical eBook that breaks down how GPUs are surfaced in Kubernetes, why CPU-style scheduling fails, and which architectures actually work in production. From time-slicing and NVIDIA MIG to vGPU and Dynamic Resource Allocation, it’s a hands-on blueprint for building reliable GPU platforms at scale. If you’re operating shared GPU infrastructure, this is the starting point. Download and register: https://lnkd.in/gyVGCyQJ #Kubernetes #GPUs #PlatformEngineering #MLOps #CloudNative

Using CRIU and NVIDIA's cuda-checkpoint, we attempted to snapshot containers with pre-loaded models in GPU memory, preserving GPU memory, CUDA contexts, and process state. During restoration, we bypass weight loading (18s), CUDA initialization (8s), and kernel compilation, reducing startup time from 32s to 6s. This approach accelerates serverless ML operations by enabling quick shutdown and instant reactivation of costly GPUs. https://lnkd.in/gxDmPuUC

Microsoft has open-sourced bitnet.cpp, a blazing-fast 1-bit LLM inference framework optimized for CPUs — and it’s a big deal for local AI compute. This could redefine how we think about running large models without expensive GPUs or cloud dependencies. Key highlights: * Up to 6x faster inference with 82% lower energy consumption * 100B parameter models running directly on x86 CPUs (via kernel throughput demo) * Ternary weights (-1, 0, +1) + 8-bit activations for huge memory savings Alongside this, Microsoft also released BitNet b1.58 2B4T, the first open-source model using just 1.58 bits per weight — and it still performs impressively on benchmarks. If you care about efficient AI at scale, this is absolutely worth a look. The efficiency gains are real, though the “100B on CPU” demo was with dummy parameters (~5–7 t/s). The currently usable model is 2B4T — but the direction is clear. The era of efficient, low-bit AI might be closer than we think. GitHub: https://lnkd.in/gi6R8ptP Paper: https://lnkd.in/gzASgUaQ #AI #LLM #BitNet #OpenSource #EdgeAI #EfficientAI #Microsoft #MachineLearning #DeepLearning #AIResearch #GPU

One of the key advantages of using macOS is its integration with Apple’s Metal framework, which enables video frame processing to be efficiently parallelized across the GPU’s multiple cores. This approach reduces reliance on CPU cores and greatly enhances real-time performance through hardware accelerated parallel execution. However, the process is not literally “multithreading each video frame” in the traditional CPU sense. Instead, Metal uses command buffers and parallel GPU kernels (threads on shader cores) to distribute the computational workload. The CPU still manages scheduling and synchronization, but the heavy computations such as frame processing, filtering, and machine learning inference are executed in parallel on the GPU, making the overall system more efficient and responsive. I have yet to find a working solution for the Windows-based setup using NVIDIA CUDA. At the moment, it is still failing to detect the GPU through TensorFlow and CUDA, which prevents proper GPU acceleration from being utilized.

Learn when to use CPUs vs. GPUs for AI inference. Compare performance, cost, and energy efficiency to choose the right hardware for your AI workloads. Read more. #CloudComputing https://ow.ly/4leu50X84pz

Learn when to use CPUs vs. GPUs for AI inference. Compare performance, cost, and energy efficiency to choose the right hardware for your AI workloads. Read more. #CloudComputing https://ow.ly/4leu50X84pz