Building a 32-Channel System with the QR210

https://www.ettus.com/kb/detail/building-a-32-channel-system-with-the-qr210

Overview

The Ettus Research QR210 Quadradio is a four-channel, high-performance receiver that can be used to build scalable phased-array systems. Up to eight units can be integrated into a single system with 32 coherent channels. Several things must be taken into consideration when building high-channel count systems:

• The local oscillators and calibration signals must be distributed to each QR210 in a way that ensures coherency.
• 10 MHz and pulse-per-second (PPS) signals must be distributed for time and sample clock alignment.
• A 32-channel system can generate an aggregate sample rate of 1920 MS/s, or a total of 7.68 GB/s of data. Therefore, a flexible data interface architecture and processing scheme must be included.

Local Oscillator and Calibration Source Distribution

As described in QR210 Architecture document, each QR210 contains a low phase noise, YIG-tuned local oscillator and an internal calibration source. In a single-QR210 system, the output of the LO and Cal. source is distributed to four internal receive channels with passive splitters and matched length traces. This ensures accurate phase and amplitude alignment across all four channels and allows the user to eliminate minor errors with rapid calibration. The approach to achieving this in larger, multi-unit systems is similar. The QR210 was designed so the output of the local oscillator and calibration source can be routed to an external connector and fed to a distribution system. This external distribution system is used to split the LO/CAL signals eight ways. The output of this system is fed back to the eight QR210s with matched-length cables. Each QR210 can then be configured to accept the external LO/Cal signals, and distribute those signals to the four internal receive channels. An illustration of this distribution architecture can be seen in Figure 1.

32-Channel QR210-based System

Figure 1 - Typical Multi-QR210 Architecture

10 MHz and 1 PPS Distribution

While a common local oscillator and calibration source are used to align the RF frontends, the 10 MHz and PPS signals are used to align the digital components. The 10 MHz signal acts as a clock reference for the system. A PLL is used to multiply the 10 MHz reference to generate a 120 MHz clock, which is used for ADC sampling and other functions. Deriving the 120 MHz clock for each QR210 from a common 10 MHz reference assures that the ADC sample clocks for all 32 channels are aligned - this is important for coherent processing.

The QR210 FPGA provides temporal alignment functionality. The 120 MHz clock serves as the timebase of the system. However, the PPS signal provides a mechanism to set the internal time of the QR210. In typical operation, the developer will use the QR210 programming API to set the time of all QR210s in the system on the rising edge of the PPS signal. In this process, a 64-bit counter in each set of receiver logic is programmed to a specified time. This time can be an absolute time, derived from GPS, or can be set to something arbitrary, like "0.0s". This 64-bit counter is driven by the 120 MHz sample clock, which means that the system timing precision is one sample clock period, or 8.33 ns. Many operations, such as tuning, receive stream initialization, and gain controls can be synchronized to this precise time reference. The time reference is also embedded in the VITA-49 frames that carry the received samples, which can be used to align samples from all channels for coherent DSP operations.

SuperMIMO Hub

To support the distribution of LO/CAL, 10 MHz, and PPS signals, Ettus Research has developed the SuperMIMO (S-MIMO) hub. This device accepts and splits the LO and CAL signals with an in-line amplifier/filter subassembly and a passive splitter. The filters were designed to reduce harmonic distortion across the wide frequency range of the QR210.

S-MIMO accepts external 10 MHz and 1PPS signals. It also contains an internal GPS Disciplined Oscillator (GPSDO), which can serve as an alternate, high-accuracy source for these signals. Active circuits are used to split the selected 10 MHz and PPS signals. Sensing circuitry and an internal microcontroller provide automatic switch-over capability, which is useful for situations that require auto-redundant operation.

While not required, the S-MIMO hub provides an option for a high-performance FPGA computation engine. This is based on a large Virtex-6 FPGA, and provides four 10 Gigabit Ethernet interfaces for control and data streaming.

SuperMIMO Hub

Figure 2 - Ettus Research SuperMIMO Hub

Data Interface and Streaming Considerations

A system of eight QR210s can produce 1920 MS/s. This is a high volume of data, and the user must give careful consideration to the interface and processing architecture. Multi-QR210 systems will typically use a rack-mounted 10 Gigabit Ethernet Switch (10 GigE) to provide connectivity between the receivers, a host controller, and one or more processing elements. These processing elements can be FPGAs, PCs, GPUs, or any other device that can accept VITA-49 frames on an Ethernet interface. Ettus Research can provide host-based and FPGA-based IP to read these frames.

In practical situations, each port on the 10 GigE switch can provide 80% throughput capability, which translates to 8 Gbps or ~250 MS/s (complex-int-16) in each direction. This is enough to stream data from four 60 MS/s channels. Reducing the sample rate can allow the system designer to interleave the data from multiple QR210s onto a single 10 GigE interface. For example, if a processing element has 4 10GigE ports, reducing the sample rate of the system to 30 MS/s would allow these interfaces to support the throughput of all 32 channels.

Figure 3 - VITA-49 Data Streaming

The QR210 design includes a new software feature that allows the user to specify a destination IP address for the streaming data. A single host-based computer can act as the control and configuration master, while offloading DSP operations to other platforms. For example, in the Super-MIMO reference design, a host PC can configure the QR210s to stream to an FPGA processing engine. This processing engine accepts data from all 32 channels at 30 MS/s and feeds the data through 40 parallel beamformers. The output of this DSP logic is 40 beamformed channels that are in turn streamed to another destination through the 10 GigE switch. This architecture, combined with the VITA-49 standard used to stream the data, makes building distributed processing systems more straightforward.

32-Channel Reference Architecture

To demonstrate the scalability of a QR210-based system, Ettus Research has developed a 32-channel reference architecture, which fulfills the requirements discussed in this document. The components of the system are:

Component	Quantity
Ettus Research QR210 Quadradio	8
SuperMIMO Hub w/ Virtex-6 Computation Engine	1
IBM RackSwitch G8124 10 Gb Switch	1
PC – 8 Core, i7-3770 @ 3.40 GHz 8 MB Cache 8 GB RAM 2x 10 GigE PCIe Card	1
Model SS-18 Power Supply	6
Matched-length cables, Ethernet cables, SFP adapters, etc.	n/a

Table 1 - 32-Channel Reference System Components A photograph of a 16-channel version of this reference system is shown in Figure 4.

Figure 4 - 16-Channel Reference System

Data Connectivity

Eight QR210s are connected to the FPGA computation engine in the SuperMIMO hub, and a host computer via the 10 GigE Switch. In this design, all devices are on the same subnet, but have different IP addresses. The FPGA computation engine maintains one IP address for each SFP+ (10 GigE) port connected to the system. This allows the user application to split the high volume of traffic into four separate streams.

The host computer is used to configure the eight QR210s and the computation engine. Aside from setting frequency and other RF parameters, the host computer also specifies the destination for each data stream in the system. In this reference design, the host configures the QR210s to stream to the FPGA computation engine. After accepting and processing data, the computation engine also produces 40 beamformed streams. Each beamformer output can stream to any arbitrary IP address. In the examples shown here, they are configured to stream back to the host PC for simple processing. They could just as easily be configured to stream to additional processing elements.

Figure 5 illustrates the flow of information in this system.

Error creating thumbnail: File missing

Figure 5 - 32-Channel Beamforming Reference System - Data Paths

FPGA Computation Engine – 40-Beamformer Example

To illustrate the power of distributed and parallel processing, this reference design includes an FPGA processing engine that implements 40 beamformers. Each beamformer operates on all 32-channels of the system. A block diagram of this DSP architecture is shown in Figure 6.

When processing channels coherently, it is important that the DSP chains operate on samples that are aligned in time. The Ethernet interface introduces variations in latency across the system, and there is no way to guarantee that samples taken from two receivers will arrive at the destination IP address at the same time. Therefore, the first processing stage of this implementation examines the timestamp inserted into the VITA-49 frames. This timestamp is used to align samples from all 32 channels before they are passed along to the actual beamformer entities. The beamformers multiply the sample from each channel with a complex tap, and sum all channels to produce a beamformed output.

The beamformed I/Q samples are then passed along to framers that add VITA-49, UDP/IP, and Ethernet headers for transport to the next processing element. All stages of this process are flexible and configurable. For example, each beamformer output can be enabled/disabled, and configured to stream to a specific IP address. Replacing the beamformers with some other DSP elements is also relatively straightforward.

Figure 6 - FPGA Beamforming Architecture

Host-Driver and GNU Radio Integration

An extension of UHD (USRP Hardware Driver) and GNU Radio can be used to configure the entire system and process streamed samples. This reference system contains several examples. A GNU Radio block with many parameters is used to configure all of the QR210s. The software also configures one or two beamformers to stream their results to the host computer. When receiving these sample streams, the host computer can perform additional DSP operations such as spectral estimation and display, blind signal detection, etc. A simple demo application which shows an FFT and graphical controls for many of the system parameters can be seen in Figure 7 and Figure 8. Ettus Research has also developed other examples for direction finding using the MUSIC algorithm.

GNU Radio Flow Graph

Figure 7 - QR210 Multi-Unit GNU Radio Flowgraph w/ VITA-49 Integration

2 Channel Receptio

Figure 8 - Diversity Reception of Two Signals - Two-Different Beamformer Outputs

Conclusion

This document provides an overview of the key considerations to be made when building a high-channel count, coherent distribution architecture built around the Ettus Research S-MIMO hub, which allows the developer to easily distribute LO, CAL, 10 MHZ, and PPS signals. The VITA-49 standard and 10 GigE interface facilitates data streaming and distributed processing. All of these capabilities are accessible using an open source driver and the GNU Radio environment, which provides a way to rapidly develop systems in a graphical environment. For more information about the QR210, please visit the product page or contact us: info@ettus.com.

Additional Resources

Link: Overview of QR210 Architecture Link: QR210 Product Page