Torus interconnect

The following images are 1D, and 2D torus. 1D torus is a simple circle, and 2D torus has the shape of a doughnut. The animation below illustrates how a 2D torus is generated from a rectangle by connecting its two pairs of opposite edges. Here the concept of torus is used to describe essentially the beginning and ending of a sequence of nodes are connected, like a doughnut. To better illustrate the concept, and understand what the topology means in network interconnect, we give 3 examples of parallel interconnected nodes using torus topology. At one dimension, a torus topology is equivalent to a ring interconnect network, of a shape of a circle. At 2D, it is equivalent to a 2D mesh, but with extra connection at the edge nodes, which is the definition of 2D torus.

• 
  1D torus example, a circle.
• 
  2D torus example, a donut.
• 
  Generating a 2D torus from a 2D rectangle.

We can generalize the rule from the figures above. Torus interconnect is a switch-less topology that can be seen as a mesh interconnect with nodes arranged in a rectilinear array of N = 2, 3, or more dimensions, with processors connected to their nearest neighbors, and corresponding processors on opposite edges of the array connected.[1] In this lattice, each node has 2N connections. This topology got the name from the fact that the lattice formed in this way is topologically homogeneous to an N-dimensional torus.

Sandeep Palur and Dr. Ioan Raicu from Illinois Institute of Technology conducted experiments to simulate 3D torus performance. Their experiments ran on a computer with 250GB RAM, 48 cores and x86_64 architecture. The simulator they used was ROSS (Rensselaer’s Optimistic Simulation System). They mainly focused on three aspects:

• Varying network size
• Varying number of servers
• Varying message size

They concluded that throughput decreases with the increase of servers and network size. Otherwise, throughput increases with the increase of message size.[3]

Fujitsu Limited developed a 6D torus computer model called "Tofu". In their model, a 6D torus can achieve 100 GB/s off-chip bandwidth, 12 times higher scalability than a 3D torus, and high fault tolerance. The model is used in the K computer and Fugaku.[4]

Higher speed, lower latency

Because of the connection of opposite edges, data have more options to travel from one node to another which greatly increased speed.

Better fairness

In a 4×4 mesh interconnect, the longest distance between nodes is from upper left corner to lower right corner. Each datum takes 6 hops to travel the longest path. But in a 4×4 Torus interconnect, upper left corner can travel to lower right corner with only 2 hops

Lower energy consumption

Since data tend to travel fewer hops, the energy consumption tends to be lower.

Complexity of wiring

Extra wires can make the routing process in the physical design phase more difficult. If we want to lay out more wires on chip, it is likely that we need to increase the number of metal layers or decrease density on chip, which is more expensive. Otherwise, the wires that connect opposite edges can be much longer than other wires. This inequality of link lengths can cause problems because of RC delay.

Cost

While long wrap-around links may be the easiest way to visualize the connection topology, in practice, restrictions on cable lengths often make long wrap-around links impractical. Instead, directly connected nodes—including nodes that the above visualization places on opposite edges of a grid, connected by a long wrap-around link—are physically placed nearly adjacent to each other in a folded torus network.[5][6] Every link in the folded torus network is very short—almost as short as the nearest-neighbor links in a simple grid interconnect—and therefore low-latency.[7]