Time Card installation and usage
To use the time card, you'll a system with a PCIe slot
I typically use Centos for my setup, so my instructions will be for Centos.
- Enable the I/O virtualization of your CPU on the BIOS:
- For Intel CPUs: Enable Vt-D or Vt-x
- For AMD CPUs: Enable IOMMU
In addition, use the following commands to get certain info about your system:
Use the following command to get info about your motherboard:
sudo dmidecode -t baseboard
In order to build the latest ocp_tap driver and use all the features on the Time Card, I typically build the kernel from scratch. If you want to avoid this step (patching the kernel), you can use Ubuntu 20.04.3 LTS or 22.04.3 LTS or anything newer by Canonical and jump directly to step 11.
- Make sure system clock is current. To update your system clock from the internet use the following command:
sudo date -s "$(wget -qSO- --max-redirect=0 google.com 2>&1 | grep Date: | cut -d' ' -f5-8)Z" ; hwclock -w
- Copy latest github
cd ~
git clone https://github.com/opencomputeproject/Time-Appliance-Project
- Install the latest kernel keys and reboot for CentOS
yum -y install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm
yum config-manager --set-enabled powertools
yum install epel-release
dnf --enablerepo=elrepo-kernel install kernel-ml
- Install prereqs for CentOS
yum install -y ncurses-devel make gcc bc bison flex elfutils-libelf-devel openssl-devel grub2 i2c-tools patch git dwarves
Install prereqs for Ubuntu
sudo apt-get install libncurses-dev flex bison openssl vim libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf zstd
- Get full kernel source from online.
for CentOS
cd /usr/src/kernels
wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.19.16.tar.xz
tar -xvf linux-5.19.16.tar.xz
rm linux-5.19.16.tar.xz
mv linux-5.19.16/ 5.19.16/
cd 5.19.16/
for Ubuntu
cd /usr/src/
sudo wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.19.16.tar.xz
sudo tar -xvf linux-5.19.16.tar.xz
sudo rm linux-5.19.16.tar.xz
cd linux-5.19.16/
- Copy existing kernel config to kernel source, edit the /boot/ file to whatever is the latest
cp -v /boot/config-5.19.16-1.el8.elrepo.x86_64 .config
vim .config
if you are adding TGPIO support, make sure you patch the source as well
sudo patch -p1 < ~/Time-Aware-GPIO-Support-main/linux-kernel-patch/tgpio-kernel-5_19_16-static.patch
then run make prepare and press enter on all the selections
sudo make prepare
- Change these kernel build parameters in .config
CONFIG_I2C_XILINX=m
CONFIG_MTD=y
CONFIG_MTD_SPI_NOR=m
CONFIG_SPI=y
CONFIG_SPI_ALTERA=m
CONFIG_SPI_BITBANG=m
CONFIG_SPI_MASTER=y
CONFIG_SPI_MEM=y
CONFIG_SPI_XILINX=m
CONFIG_I2C=y
CONFIG_I2C_OCORES=m
CONFIG_IKCONFIG=y
CONFIG_EEPROM_AT24=m
CONFIG_SYSTEM_TRUSTED_KEYS=""
if you are enabling TGPIO and PTM, select the following in addition
CONFIG_PTP_INTEL_PMC_TGPIO=y
CONFIG_PCIE_PTM=y
additionally for for Ubuntu
CONFIG_SYSTEM_TRUSTED_KEYS=""
CONFIG_SYSTEM_REVOCATION_KEYS=""
CONFIG_DEBUG_INFO_BTF=n
run make prepare again and select "y" for CONFIG_IKCONFIG_PROC and select default (just press Enter) for any other question
- Build and install this kernel
make -j5 bzImage modules; make INSTALL_MOD_STRIP=1 modules_install; make install;
- Update grubby for the new kernel, make sure version matches here.
for CentOS
grub2-mkconfig -o /boot/grub2/grub.cfg
grubby --set-default /boot/vmlinuz-5.19.16
grubby --update-kernel=ALL --args=8250.nr_uarts=8
for Ubuntu Modify the GRUB_DEFAULT in /etc/default/grub to the following
GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.19.16"
then run
sudo update-grub
- Reboot
reboot
- Now kernel is up to date. Make sure this is actually picked up by the kernel next time you boot
cat /proc/cmdline
[root@localhost Binaries]# cat /proc/cmdline
BOOT_IMAGE=(hd1,gpt2)/vmlinuz-5.19.19 root=/dev/mapper/cs-root ro crashkernel=auto resume=/dev/mapper/cs-swap rd.lvm.lv=cs/root rd.lvm.lv=cs/swap rhgb quiet 8250.nr_uarts=8
- Copy latest github for most up to date driver and build and install it
cd ~/Time-Appliance-Project/Time-Card/DRV/
./remake
modprobe ptp_ocp
In addition, you can add ptp_ocp as well as i2c_xiic to the /etc/modules to make them automatically load on boot
Now you can check dmesg to see if things are in a good shape. This is what you should expect:
[ 1491.440223] 0000:02:00.0: ttyS5 at MMIO 0x70161000 (irq = 180, base_baud = 3125000) is a 16550A
[ 1491.440278] 0000:02:00.0: ttyS6 at MMIO 0x70171000 (irq = 181, base_baud = 3125000) is a 16550A
[ 1491.440322] 0000:02:00.0: ttyS7 at MMIO 0x70181000 (irq = 182, base_baud = 3125000) is a 16550A
[ 1491.440362] 0000:02:00.0: ttyS0 at MMIO 0x70191000 (irq = 187, base_baud = 3125000) is a 16550A
[ 1491.441702] xilinx_spi xilinx_spi.1024: at [mem 0x70310000-0x7031ffff], irq=186
[ 1491.442678] spi-nor spi1024.0: n25q128a13 (16384 Kbytes)
[ 1491.447310] pps pps1: new PPS source ptp2
[ 1491.449374] ptp_ocp 0000:02:00.0: Version 1.2.0, clock PPS, device ptp2
[ 1491.449388] ptp_ocp 0000:02:00.0: Time: 1647548640.775286647, in-sync
[ 1491.449390] ptp_ocp 0000:02:00.0: version 8005
[ 1491.449392] ptp_ocp 0000:02:00.0: regular image, version 32773
[ 1491.449393] ptp_ocp 0000:02:00.0: GNSS: /dev/ttyS5 @ 115200
[ 1491.449394] ptp_ocp 0000:02:00.0: GNSS2: /dev/ttyS6 @ 115200
[ 1491.449396] ptp_ocp 0000:02:00.0: MAC: /dev/ttyS7 @ 57600
[ 1491.449399] ptp_ocp 0000:02:00.0: NMEA: /dev/ttyS0 @ 115200
https://www.raspberrypi.com/documentation/computers/linux_kernel.html
CONFIG_NET_DEVLINK=y CONFIG_SERIAL_8250_NR_UARTS=8 CONFIG_IRQ_POLL=y
changed Kconfig in net to have DEVLINK enabled
make oldconfig before doing build but after manually adding DEVLINK
Time Card is interfaced with via the sysfs. Newer cards can have any SMA be an input or output.
cd /sys/class/timecard/ocp0/
Serial ports and PHC number can be found via dmesg or ls -l
All possible outputs are available via available_sma_outputs
cat available_sma_outputs
10Mhz PHC MAC GNSS1 GNSS2 IRIG DCF GEN1 GEN2 GEN3 GEN4 GND VCC
- Output FPGA PPS on SMA
echo OUT: PHC >> sma1
- Output Atomic clock PPS on SMA
echo OUT: MAC >> sma1
- Output GPS Module's PPS on SMA
echo OUT: GNSS1 >> sma1
All possible inputs are available via available_sma_inputs
cat available_sma_inputs
10Mhz PPS1 PPS2 TS1 TS2 IRIG DCF TS3 TS4 FREQ1 FREQ2 FREQ3 FREQ4 None
- Use an SMA for Timestamping incoming signals
echo IN: TS1 >> sma1
- To read back Timestamps, use testptp.
- Source available at these locations
https://www.mjmwired.net/kernel/Documentation/ptp/testptp.c
https://github.com/torvalds/linux/blob/master/tools/testing/selftests/ptp/testptp.c
https://unix.stackexchange.com/questions/725969/does-testptp-c-compile-for-anyone-else -> seems to be an issue soon
wget https://raw.githubusercontent.com/torvalds/linux/master/tools/testing/selftests/ptp/testptp.c
cp /usr/src/kernels/5.17.4/include/uapi/linux/ptp_clock.h /usr/include/linux/ptp_clock.h
gcc testptp.c -lrt -o testptp
- Read infinite timestamps (-1 , use a positive number for fixed event count) from Timestamper 1 (-i 1) from the Time Card (/dev/ptp1)
./testptp -d /dev/ptp1 -e -1 -i 1
- Use gpsd toolset (recommended)
stty -F /dev/ttyS5 speed 115200
gpsd /dev/ttyS5
cgps
- Use tio to make sure the GPS is sending messages
tio -b 115200 /dev/ttyS5
- Make sure the FPGA time is correct by checking NMEA
tio -b 115200 /dev/ttyS0
- Use a third party program to configure and interact with GNSS module, requires monitor or VNC
yum install python3
yum install python3-tkinter
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade Pillow
python3 -m pip install pygpsclient
pygpsclient
- For a factory default SA5x you must run the following commands at default baud rate of 57600
\{set,Disciplining,1}
\{set,PpsWidth,80000000}
\{set,TauPps0,10000}
\{set,PpsOffset,-30}
\{set,DisciplineThresholdPps0,20}
\{store}
- Use timetickler.py
cd ~/Time-Appliance-Project/Time-Card/MAC/SA53
python3 timetickler.py
- Use devlink to flash the firmware.
- Hack cards use images from https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Time-Card/SOM/FPGA/Binaries
cp TimeCard.bin /lib/firmware
devlink dev flash pci/0000:11:00.0 file TimeCard.bin
The FPGA binaries with PTM support require an updated Linux driver. The mainstream Linux kernel driver won't work.
For most Time Card versions FPGA images with PTM support are available:
- SOM with 100T FPGA: https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Time-Card/SOM/FPGA/Binaries/LitePCIe
- SOM with 200T FPGA: https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Time-Card/SOM/FPGA/Binaries/LitePCIe200T
- Production/Celestica: https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Time-Card/FPGA/Binary/Production/Binaries/LitePCIe
For PTM support, the driver included in the mainstream Linux kernel will not work (host may not boot). Therefore, it makes sense to update the driver before updating the FPGA binaries.
If the system no longer boots, blacklisting ptp_ocp will help the system boot without the mainstream Linux kernel driver:
- Remove the Time Card and boot the system
- open /etc/modprobe.d/blacklist.conf
- add the entry blacklist ptp_ocp
- Install the Time Card again
With an updated and correctly loaded driver, PTM is supported:
- Get the latest version from git: https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Time-Card/DRV
- Change to the directory of the driver
cd ~/Time-Appliance-Project/Time-Card/DRV/ - Compile the latest ptp_ocp driver with
./remake - Load the driver with
insmod ptp_ocp.ko
To test PTM, the host must also support PTM.
Also make sure you have a newer version of linuxptp installed. (e.g. v4.3 https://github.com/richardcochran/linuxptp/releases/tag/v4.3).
Other time services should be stopped.
With the lspci command you can see the Capabilities: [1ac v1] Precision Time Measurement:
01:00.0 Memory controller: Facebook, Inc. Device 0400 (rev 02)
Subsystem: Xilinx Corporation Device 0007
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Region 0: Memory at 74000000 (32-bit, non-prefetchable) [size=64M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr+ FatalErr+ UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis-, NROPrPrP-, LTR-
10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-, TPHComp-, ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00001808
Capabilities: [100 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [1ac v1] Precision Time Measurement
PTMCap: Requester:+ Responder:- Root:-
PTMClockGranularity: 8ns
PTMControl: Enabled:+ RootSelected:-
PTMEffectiveGranularity: 4ns
Kernel driver in use: ptp_ocp
Kernel modules: ptp_ocp
Following commands can be used to synchronize the host:
sudo systemctl stop systemd-timesyncd.service
sudo phc_ctl /dev/ptp2 set
sudo phc2sys -c CLOCK_REALTIME -s /dev/ptp2 -O 0 -R 16 -u 8 -m
With correct working PTM the print out of phy2sys should show delay 0 +/-.
Example with PTM:
phc2sys[4503.063]: CLOCK_REALTIME rms 26 max 40 freq -17250 +/- 17 delay 0+/- 0
phc2sys[4503.564]: CLOCK_REALTIME rms 46 max 85 freq -17346 +/- 17 delay 0+/- 0
phc2sys[4504.066]: CLOCK_REALTIME rms 25 max 54 freq -17316 +/- 25 delay 0+/- 0
phc2sys[4504.567]: CLOCK_REALTIME rms 29 max 53 freq -17277 +/- 24 delay 0+/- 0
phc2sys[4505.069]: CLOCK_REALTIME rms 29 max 63 freq -17219 +/- 20 delay 0+/- 0
phc2sys[4505.570]: CLOCK_REALTIME rms 27 max 53 freq -17253 +/- 18 delay 0+/- 0
phc2sys[4506.072]: CLOCK_REALTIME rms 24 max 40 freq -17247 +/- 21 delay 0+/- 0
phc2sys[4506.573]: CLOCK_REALTIME rms 32 max 49 freq -17284 +/- 22 delay 0+/- 0
phc2sys[4507.075]: CLOCK_REALTIME rms 18 max 32 freq -17280 +/- 16 delay 0+/- 0
phc2sys[4507.577]: CLOCK_REALTIME rms 28 max 48 freq -17289 +/- 26 delay 0+/- 0
phc2sys[4508.078]: CLOCK_REALTIME rms 31 max 54 freq -17252 +/- 27 delay 0+/- 0
phc2sys[4508.579]: CLOCK_REALTIME rms 20 max 40 freq -17268 +/- 14 delay 0+/- 0
phc2sys[4509.081]: CLOCK_REALTIME rms 30 max 48 freq -17243 +/- 24 delay 0+/- 0
phc2sys[4509.583]: CLOCK_REALTIME rms 31 max 52 freq -17291 +/- 25 delay 0+/- 0
phc2sys[4510.084]: CLOCK_REALTIME rms 24 max 37 freq -17275 +/- 20 delay 0+/- 0
phc2sys[4510.586]: CLOCK_REALTIME rms 21 max 37 freq -17273 +/- 19 delay 0+/- 0
phc2sys[4511.087]: CLOCK_REALTIME rms 27 max 62 freq -17235 +/- 19 delay 0+/- 0
phc2sys[4511.589]: CLOCK_REALTIME rms 23 max 36 freq -17272 +/- 19 delay 0+/- 0
Tests below were executed with the following driver version of following commit: bdbde83
After loading the driver following commands were used:
sudo systemctl stop systemd-timesyncd.service
sudo phc_ctl /dev/ptp2 set
sudo phc2sys -c CLOCK_REALTIME -s /dev/ptp2 -O 0 -R 16 -u 8 -m
Following measurement setup was used:
| Chipset/Device | lspci printout | Time Card Version | Results |
|---|---|---|---|
| Intel® Z690 Chipset (ROG STRIX Z690-I) |
00:00.0 Host bridge: Intel Corporation Device 4630 (rev 05)00:01.0 PCI bridge: Intel Corporation Device 460d (rev 05)
|
v31 Time Card LitePCIe |  |
| Intel® Z690 Chipset (ROG STRIX Z690-I) |
00:00.0 Host bridge: Intel Corporation Device 4630 (rev 05)00:01.0 PCI bridge: Intel Corporation Device 460d (rev 05)
|
v15 Time Card Production LitePCIe |  |
| Intel® WM590 Chipset (ASUS NUC 11 Extreme Kit) |
00:00.0 Host bridge: Intel Corporation 11th Gen Core Processor Host Bridge/DRAM Registers (rev 05)00:01.0 PCI bridge: Intel Corporation 11th Gen Core Processor PCIe Controller #1 (rev 05)
|
v31 Time Card LitePCIe | NO TGPIO on the Host |
| Intel® WM590 Chipset (ASUS NUC 11 Extreme Kit) |
00:00.0 Host bridge: Intel Corporation 11th Gen Core Processor Host Bridge/DRAM Registers (rev 05)00:01.0 PCI bridge: Intel Corporation 11th Gen Core Processor PCIe Controller #1 (rev 05)
|
v15 Time Card Production LitePCIe | NO TGPIO on the Host |