Facebook Open Sources Time Appliance For Data Center Networks

Article By : Nitin Dahad

Open source time appliance overcomes limitations of existing solutions for network synchronization infrastructure, and is a tenth of the cost.

Facebook engineers have built a time appliance for network synchronization and open-sourced the entire specification, which should drive the price of the function down significantly. The Open Compute project is based on a time card the company invented in a PCI Express (PCIe) form factor that can turn almost any commodity server into a time appliance. With the help of the OCP community, it established the Open Compute Time Appliance Project and open-sourced every aspect of the Open Time Server.

Open time server concept
Open time server concept: the general idea is the time card is connected via PCIe to the server and provides time of day (TOD) via /dev/ptpX interface. Using this interface phc2sys continuously synchronizes PHC (PTP hardware clock) on the network card from the atomic clock on the time card. This provides precision < 1us. For the extremely high precision 1PPS output of the time card will be connected to the 1PPS input of the NIC, providing <100ns precision. (Source: Open Compute Time Appliance Project)

Time appliances are critical elements of much of modern timing infrastructure from 5G and automotive to financial services and television broadcasting. All of these rely heavily on reliable distribution of time and frequency synchronization across packet networks. The big challenge with current off-the-shelf time appliances is that while they work well and are tried and tested, they are often outdated, vulnerable to software security concerns, and feature closed-source software and proprietary hardware, making them difficult to service, repair and update; and they are also expensive.

In comparison, every aspect of the design of the open compute time appliance released today — including the specifications, schematics, mechanics, bill of material (BOM), and the source code — is open sourced and available on the GitHub repository, under the Open Compute Project umbrella.

In addition, a Facebook spokesperson told EE Times that the new time card can be 10 times cheaper, three times more accurate, and more compact than any commercial solution available today. He indicated that the new time card might cost in the range of $1,500 to $2,000, as compared to the $50,000 range for current solutions.

Facebook time appliance diagram
Time appliance system overview. The time card (light green block area) comprises GNSS receiver, atomic clock and FPGA. The time card allows any x86 machine with NIC to be turned into a time appliance. (Source: Facebook)

The new time card allows any x86 machine with a network interface card (NIC) capable of hardware time-stamping to be turned into a time appliance. This system is agnostic to whether it runs for NTP, PTP, SyncE, or any other time synchronization protocol, since the accuracy and stability provided by the time card is sufficient for almost any system.

The time card built by Facebook consists of a GNSS receiver, and a miniaturized atomic clock (MAC). This is significant as it removes the dependencies on internet connectivity and satellite signals which, if lost, could result in outages or drift in timing for any dependent systems. With the integrated GNSS receiver and clock, users of the time appliance can keep accurate time, even in the event of GNSS connectivity loss. That’s because the atomic clock kicks in and can maintain accuracy for several hours until a signal is re-established.

To implement the time engine in the time card, Facebook used an onboard MAC, a multiband GNSS receiver, and an FPGA. The time engine’s job is to interpolate in nanoseconds the granularity required between consecutive PPS (pulse per second) signals. The GNSS receiver also provides a ToD (time of day) in addition to a 1 PPS signal. In the event of the loss of GNSS reception, the time engine relies on the ongoing synchronization of the atomic clock based on an average ensemble of the consecutive PPS pulses.

The processing blocks of the time engine on the FPGA include various filtering, synchronization, error checking, time-stamping, and PCIe-related subsystems to allow the time card to perform as a system peripheral that provides precision time for the open time server.

Facebook Time card
Facebook built its time card in a PCIe form factor, which makes it possible to turn almost any commodity server into a time appliance. (Source: Facebook)

Facebook said that building a device that is very precise, inexpensive, and free from vendor lock was an achievement on its own. Writing in an article on the development of the time card, the company’s network hardware engineers for the project, Ahmad Byagowi and Oleg Obleukhov, said, “We wanted to truly set it free and make it open and affordable for everyone, from a research scientist to a large cloud data center. That’s why we engaged with the Open Compute Project (OCP) to create a brand-new Time Appliance Project (TAP). Under the OCP umbrella, we open-sourced at the Time Appliance Project GitHub repository, including the specs, schematics, mechanics, BOM, and the source code. Now, as long as printing the PCB and soldering tiny components does not sound scary, anyone can build their own time card for a fraction of the cost of a regular time appliance. We also worked with several vendors such as Orolia who will be building and selling time cards ,and Nvidia who are selling the precision timing- capable ConnectX-6 Dx (and the precision timing-capable BlueField-2 DPU).”

They added, “We published an Open Time Server spec at www.opentimeserver.com, which explains in great detail how to combine the hardware (time card, network card, and a commodity server) and the software (OS driver, NTP, and/or PTP server) to build the time appliance. Building an appliance based on this spec will give full control to the engineers maintaining the device, improving monitoring, configuration, management, and security.”

For more details on the development of the open source time card, including discussion on accuracy and source code, see the article on embedded.com, “Open-sourcing a more precise time appliance“.

Nitin Dahad is a correspondent for EE Times, EE Times Europe and also Editor-in-Chief of embedded.com. With 35 years in the electronics industry, he’s had many different roles: from engineer to journalist, and from entrepreneur to startup mentor and government advisor. He was part of the startup team that launched 32-bit microprocessor company ARC International in the US in the late 1990s and took it public, and co-founder of The Chilli, which influenced much of the tech startup scene in the early 2000s. He’s also worked with many of the big names—including National Semiconductor, GEC Plessey Semiconductors, Dialog Semiconductor and Marconi Instruments.


Subscribe to Newsletter

Leave a comment