Xilinx AXI Chip2Chip for multi-FPGA design

by Gilad Krupsky Reisman

The AXI Chip2Chip IP from Xilinx allows the designer to connect two or more FPGAs using an AXI bus implemented using transceivers running the Aurora64/66 protocol. While there is also an option to use regular FPGA pins if you don’t have transceivers, in my experience, it takes up too many pins to be relevant.

What is it good for?

If you have a design with a SOC (Zynq 7000, MPSoC, or Versal) and at least one more FPGA, you might want to have a single master processor that can communicate and control the other FPGAs with high throughput. Since the native bus for the ARM processors on the SOC devices is AXI, it makes sense to use it to connect to the other FPGAs. If you know the AXI bus protocol (if you don’t, read this link first – press download on the left corner), you know that it sometimes contains hundreds of bits. Connecting the bus as-is is not feasible and that is where the AXI Chip2Chip IP comes in.

How does it work?

As described in the user guide in this link (Xilinx PG067):

The AXI Chip2Chip core functions like a bridge to seamlessly connect two devices over an AXI interface. The core transparently bridges transactions in compliance with AXI protocol specifications.

The bridging function allows all AXI channels to operate independently by forwarding per-channel data and control information in compliance with AXI per-channel Valid-Ready handshake.

The AXI Chip2Chip (C2C) IP is divided into two parts, The C2C Master and the C2C Slave, and is a point-to-point connection so that a separate master is needed per slave. As seen in the image below, the C2C master has an AXI4 slave interface. Any AXI event on the C2C master AXI interface is packetized and transmitted over the transceiver to the C2C slave.

Xilinx AXI Chip2Chip Master and Slave

The AXI Chip2Chip IP needs to be manually connected to an Aurora 64/66 connection and this is not the most trivial connection I’ve seen but if you go through some examples, you will get it…

AXI Chip2Chip and Aurora64b66b IP connection

At this point, I find it important to emphasize that when you connect an AXI C2C master in your Block Diagram, you assign an AXI address for it using the address editor, as you would for any other AXI IP added to the design. When writing or reading to that AXI address from any AXI master, the AXI operation reaches the AXI C2C master, is transferred to the AXI C2C slave, and is outputted using the exact same address used to write on the master (!!!).

This means that when you assign a memory range on one device, you are automatically assigning it on the other devices as well. If you have a memory region on the slave device that needs to be accessed from the master device, the master device must assign the AXI C2C master’s address range to match the slaves’ address range.

For example, if you have 64kB of register space on slave FPGA 1 mapped to address 0xA0010000 up to 0xA001FFFF, you must map the AXI C2C master IP to 0xA0010000.

AXI Memory maps on master and slave devices

There is, however, a way around this…

You can write a small AXI IP of your own with an AXI slave and an AXI master interface that remaps any AXI command on the slave port to a new address on the master port by overwriting the MSB of the address.

I found this workaround useful for example when working with more than one MPSoC device and needing the slave MPSoC device to write to the master MPSoC’s DDR through the AXI C2C connection. This is a problem because the AXI DDR_LOW address range (0x00000000 to 0x7FFFFFFF) exists on both devices and so the AXI C2C cannot be assigned to this address on the memory editor. But, if you use the workaround you can, for example, assign the address 0xB0000000 to the AXI C2C master and on the AXI C2C slave add your address remap IP changing all incoming transactions from 0xB0000000 to 0x00000000 which is the DDR_LOW address required.

FPGA resources utilization

The AXI Chip2Chip is relatively not a large resource user but does require a single/dual transceiver per master-slave connection.

You can see below an example of resource utilization for a single channel over SerDes using Aurora 64/66.

AXI Chip2Chip resource utilization using Aurora 64/66

Additional notes

• The AXI Chip2Chip IP supports a single AXI4 Full master to slave and an additional AXI Light master in the opposite direction on a single IP.
  If, however, you require AXI4 Full interfaces in both directions, you need to implement an additional AXI Chip2Chip connection in the opposite direction (with additional resources and transceivers).
• The AXI Chip2Chip IP supports interrupt signals in both directions in the IP.
• (Updated at 12/09/2922) – I was asked on LinkedIn what is the expected throughput of this block, and this was my answer: It depends on the AXI parameters and the Aurora parameters. The AXI parameters are width, frequency, and muxing ratio (how many masters share the link). The Aurora parameters are encoding (8/10, 64/66) and link speed in Gbps. PG067 has a detailed explanation (for the Chip2Chip side of the calculation, on the Throughput and Latency section). You have to refer also to your Aurora IP UG to calculate its throughput.