Build your own video pipeline with PYNQ composable overlays

PYNQ Composable Overlay gives FPGA users more functional flexibility at runtime

By Ivo Bolsens

Xilinx has developed an open-source, composable overlay system for the Python-based PYNQ FPGA development platform. Traditional FPGA overlays, often called hardware libraries, are a good way to start using FPGAs because they provide access to hardware-speed processing through software programmability, within a well-defined scope. PYNQ already embraces these types of hardware overlays and they have clearly reduced the barriers to entry into the FPGA world for software developers. Typically, these hardware overlays are designed in Xilinx Vivado IPI (IP Integrator) and then imported into PYNQ. The newly developed composable overlay for PYNQ takes the overlay idea one step further by allowing users to compose overlay instances at runtime directly from within the PYNQ environment without using additional development tools.

The PYNQ Composable Overlay combines three abstraction layers, shown in the figure below. Developers can harness these layers by composing an overlay at runtime, and they can alter functionality on the fly using Python APIs. These Composable Overlay features boost FPGA runtime flexibility significantly when compared to traditional overlays.

Figure 1: The new PYNQ Composable Overlay consists of three layers

The new PYNQ Composable Overlay implements a 3-layer stack. Each layer builds on the layer below to greatly ease development of new overlays at runtime. The bottom-most Composable layer is based on an AXI4-Stream Switch, a block of configurable hardware IP, and is used to define how streaming data flows among hardware function blocks instantiated in the FPGA. The multi-port, AXI4 Stream Switch can connect as many as sixteen AXI4 slave ports and sixteen AXI4 master ports in a fully populated crossbar matrix. This hardware switch enables composable datapaths,  so that the dataflow between the hardware functional blocks can be set at runtime.

The Dynamic layer is based upon the Dynamic Function eXchange (DFX), built into the Xilinx Vivado ML tool suite. The Dynamic layer and DFX provides a dynamic hardware reconfiguration capability that allows you to swap out functions on the fly without disturbing the rest of the hardware overlay. DFX augments the FPGA’s inherent hardware overlay functionality by allowing you to load the appropriate logic for the application at hand at any given time.

The API layer provides a set of PYNQ-based APIs that permit overlay composition at runtime. This is how PYNQ users interact with the composable overlay. The API builds on top of and extends PYNQ’s DefaultHierarchy object to expose the underlying Composable Overlay functionality to the users. This object extension makes the Composable Overlay easy to use and easy to extend. More details of using the API are discussed at the composable video pipeline.

These three layers sit on top of the FPGA’s programmable-logic fabric. Each of these layers brings new dimensions, more capabilities, and more flexibility to the design space, which allows developers at all skill levels to explore and unleash the full potential of Xilinx’s Adaptive technologies.

To demonstrate the capabilities of PYNQ’s new Composable Overlay , Xilinx has developed a Composable Video Pipeline that shows  how to create a variety of hardware video-processing pipelines, all from within a Jupyter Notebook. The Composable Video Pipeline is available on GitHub. Preconfigured video pipeline components  include six static hardware components (Color thresholding, Filter2d, Gray2rgb, LUT (color lookup table), rgb2gray, and rgb2hsv) that are always present on the FPGA as hardware instantiations, and twelve dynamic hardware components (absdiff, add, bitwise_and, cornerHarris, dilate, duplicate, erode, fast, fifo, filter2d, rgb2xyz, and subtract) that can be swapped in and out of the FPGA’s programmable fabric as needed to build specific video-processing pipelines. Although the Composable Video Pipeline was developed as a demonstration vehicle with three specific video applications in mind – corner detection, difference of Gaussians, and color detection – it can be used to implement a variety of other video-processing applications.

For more information and for a more complete technical discussion of the Composable Overlay system and the Composable Video Pipeline, watch the 28-minute “PYNQ Composable Overlay overview and demo” video on YouTube. This demo uses a low-cost Xilinx Kria KV260 Vision AI Starter Kit for advanced vision development as the target hardware platform. It shows how you can use the PYNQ Composable Overlay to create a variety of video pipelines for an FPGA using only Python and an interactive Jupyter Notebook as development tools.

However, please don’t get the idea that the PYNQ Composable Overlay methodology is limited to constructing video pipelines. That’s merely one example to showcase its capabilities. The same underlying approach  enables you to quickly develop a variety of custom datapaths whenever many discrete hardware functional blocks can be identified and then combined to create very complex, high-performance processing systems using adaptive, programmable-logic hardware and development tools from Xilinx.