Copyright (C) 2023 Intel Corporation SPDX-License-Identifier: MIT
Introduction SW Genlock
SW Genlock is a software solution designed to synchronize the display outputs of multiple computer systems to single digit microsecond precision, enabling seamless visual outputs across an array of screens. This technology is particularly beneficial in settings that require highly synchronized video outputs, such as video walls, virtual reality setups, digital signage, and other multi-display configurations.
Key Features of SW Genlock
- High Precision Synchronization: Ensures that vertical sync (vblank) intervals occur simultaneously across all connected systems, maintaining the integrity of visual displays.
- Flexible Integration: Offers both dynamic (.so) and static (.a) linking options to accommodate various application needs.
- Comprehensive Tools: Includes a suite of tools and libraries that facilitate the synchronization process, such as obtaining vblank timestamps and adjusting the phase-locked loop (PLL) frequencies.
- Extensive Compatibility: Supports systems equipped with Precision Time Protocol (PTP), allowing for precise clock synchronization over network infrastructures.
- User-Centric Utilities: Provides reference applications that demonstrate effective usage of the library, assisting users in integrating Genlock capabilities into their own projects.
The SW Genlock system operates by aligning the internal clocks of all systems involved, ensuring that every display refresh is triggered simultaneously. This alignment is achieved through the utilization of industry-standard protocols and our advanced proprietary algorithms, which manage the synchronization over ethernet or direct network connections.
Known Issues
Users of this SW Genlock libraries may need to adjust the value for SHIFT. In testing, on various displays a valaue that is too large can cause flickering on screen when synchronized. To resolve this, adjust the value of SHIFT. The adjustment can be made by:
1) Editing the default value in lib/common.h 2) Providing a new value for SHIFT when running the sample application:
Installation
Installing SW Genlock involves a few straightforward steps. This section guides through the process of setting up SW Genlock on target systems, ensuring that user have everything needed to start synchronizing displays.
System Requirements
Before the user begin the installation, ensure the systems meet the following criteria:
The build system assumes the following, prior to executing the build steps:
1) The system has an Ubuntu OS 1) Libraries installed: sudo apt install -y git build-essential make 1) Disable secure boot 1) MUST apply the PLL kernel patch. Compile the kernel and update either the entire kernel or just the i915.ko module, depending on the Linux configuration. Note: In some setups, the i915.ko module is integrated into the initrd.img, and updating it in /lib/module/... might not reflect the changes.
- Following the application of the patch, edit the grub settings: sudo vi /etc/grub/default# add i915.share_dplls=0 to the GRUB_CMDLINE_LINUX options:# GRUB_CMDLINE_LINUX="i915.share_dplls=0"# Save and exit, then update grubupdate-grub
1) Apply the monotonic timstamp patch to Linux kernel i915 driver which allows it to provide vsync timestamps in real time instead of the default monotonic time.
Note that the monotonic timestamp patch is generated based on Linux v6.4 and has been tested on Linux v6.4 and v6.5.
Please follow the steps to disable monotonic timestamp after installing the local built Linux image on a target. Compile the kernel and update either the entire kernel or just the drm.ko module based on the installed Linux configuration.
- Add
drm.timestamp_monotonic=0option to GRUB_CMDLINE_LINUX in /etc/default/grub - Update GRUB_DEFAULT in /etc/default/grub to address the local built Linux image
- Run
update-grubto apply the change Reboot a target 1) Turn off NTP time synchronization service by using this command:
1) All the involved systems must support PTP time synchronization via ethernet (e.g ptp4l, phc2sys)To verify if an ethernet interface supports PTP, the `ethtool` linux tool can be used. Run the following command, replacing `eth0` with the relevant interface name:```consoleethtool -T eth0If the interface supports PTP, the output will display specific PTP hardware clock information. To ensure proper synchronization, the Ethernet interfaces on the involved systems should be directly connected either via a crossover cable or through a PTP-supported router or switch. If the systems only have a single network interface, it may be necessary to add a separate network adapter, such as a USB-to-Ethernet or USB-to-WiFi dongle, to maintain SSH access and connectivity to external networks. 1) Displays must be at same refresh rate (e.g 60Hz)
Build steps
1) Git clone this repository 1) sudo apt install libdrm-dev libpciaccess-dev 1) If the directory /usr/include/drm does not exist, it may be necessary to run the commandsudo apt install -y linux-libc-dev 2) Type make from the main directory. It compiles everything and creates library and executable binaries in respective folders along with code which can then be copied to the target systems.
Software Components
Vsync Library
The Vsync library is distributed in both dynamic (.so) and static (.a) binary formats, allowing users to choose between dynamic and static linking depending on their application requirements. This library includes essential functions such as retrieving vblank timestamps and adjusting the vblank timestamp drift by specified microseconds. Additionally, all operations related to Phase-Locked Loop (PLL) programming are encapsulated within the library, providing a comprehensive suite of tools for managing display synchronization.
Reference Applications
Accompanying the library, three reference applications are provided to demonstrate how to utilize the library effectively. These include:
- A
vsync testapp that runs in either primary mode or secondary mode. primary mode is run on a single PC whereas all other PCs are run as secondary mode with parameters pointing to primary mode system ethernet address. The communication is done either in ethernet mode or IP address mode.
- A
vbltestapp to print average vblank period between sync. This is useful in verification and validation.
- A
synctestapp to drift vblank clock on a single display by certain period such as 1000 microseconds (or 1.0 ms).
synctest sample output:
Generating Doxygen documents
Please install doxygen and graphviz packages before generating Doxygen documents:
Building of this program has been successfully tested on both Ubuntu 2x and Fedora 30.
Synchronization between two systems
Synchronizing displays across two systems involves a two-step process. Firstly, the Real Time Clocks of both systems need to be kept in synchronized state using the ptp4l Linux tool. Subsequently, the vsync test app should be run in primary mode on one system and in secondary mode on the other, ensuring that the vblank of the secondary system remains synchronized with that of the primary system.
Synchronizing Clocks between two systems
The SW Genlock feature requires that all participating systems' clocks be synchronized to the nanosecond level. Kernel patches included with the software return vblank timestamps based on the system clock rather than the kernel's monotonic clock, which tracks time from when the kernel is loaded. Accurate synchronization is crucial because SW Genlock shares vblank timestamps from the primary system with secondary systems. This synchronization is achieved using the Linux ptp4l tool. There should be a network cable directly connecting the ethernet ports of the two systems.
Clock Configuration in PTP-Supported Systems
Systems supporting PTP typically have two clocks: the system's real-time clock and another clock on the Ethernet card. The synchronization process involves:
- Primary System Real-Time Clock Synchronization:
- Initially synchronize the primary system's real-time clock with an NTP server.
- Then sync the Ethernet PTP clock with the primary system's real-time clock.
- Secondary Systems Synchronization:
- Sync the secondary systems' Ethernet PTP clocks with the primary's Ethernet PTP clock.
- Sync their system real-time clocks with their Ethernet PTP clocks, which are already synchronized with the primary system.
This setup ensures all secondary system's real-time clocks are synchronized with the primary system's real-time clock. Synchronization within a system (system real-time clock to Ethernet PTP clock) is managed using the phc2sys tool, while inter-system synchronization (across Ethernet interfaces) utilizes the ptp4l tool.
<figure>
<figcaption>PTP4L Setup.</figcaption></figure>
It's also important to turn off NTP time synchronization service by using this command:
- phc2sys:
Secondary System Commands:
- ptp4l:
- phc2sys:
The output of ptp4l on the primary system after running the above command should look like this:
The output of ptp4l on the secondary system after running the above command should look like this:
Please make it sure that the secondary sytem refers to the primary's precision time by checking if there is the **"new foreign master"** message in a log
Note that the rms value should be decreasing with each line and should go down to single digits.
For user convenience, an automation Bash script is available in the scripts folder to facilitate ptp4l synchronization on both primary and secondary systems. Users simply need to update the config.sh file with relevant details such as the secondary system's address, username, and Ethernet address, among others. After these updates, run the scripts/run_ptp.sh script from the primary system. This script will automatically execute the ptp4l and phc2sys tools on both the primary and secondary systems in their respective modes. Note that the script requires the secondary system to enable passwordless SSH to execute ptp4l commands through the SSH interface.
Since the ptp4l and phc2sys tools require root privileges or sudo access, users need to provide their sudo password in the config.sh file. As an alternative, setting up passwordless sudo can streamline this process. Here's how to enable passwordless sudo:
Open a terminal and type sudo visudo to edit the sudoers file. Add the following line to the file at the end, replacing username with actual username:
Save and close the file to apply the changes. This adjustment allows the automation scripts to run without requiring a sudo password each time, simplifying the setup for ptp4l and phc2sys synchronizations.
Synchronizing Display VBlanks
Once the system clocks are synchronized, user can proceed to run the vsync test tool on both the primary and secondary systems. Installing and running the programs:
- Copy the release folder contents to both primary and secondary systems.
- On both systems, go to the directory where these files exist and set environment variable: export LD_LIBRARY_PATH=.
- On the primary system, run it as follows: ./vsync_test -m pri -i [PTP_ETH_Interface]
- On the secondary system, run it as follows: ./vsync_test -m sec -i PTP_ETH_Interface -c [Primary_ETH_MAC_Addr] -d [sync after # us of drift]
This program runs in server mode on the primary system and in client mode on the secondary system.
<figure>
<figcaption>VBlank synchronization setup.</figcaption></figure>
If using PTP protocol to communicate between primary and secondary, there are some extra parameters required. The primary system must identify the PTP Interface (ex: enp176s0). The secondary system also requires its PTP interface as well as this port's ethernet MAC address. An example of PTP communication between primary and secondary looks like this: On the primary system, run it as follows:
On the secondary system, run it as follows:
In the above examples, we were just sync'ing the secondary system once with the primary. However, due to reference clock differences, we see that there is a drift pretty much as soon as we have sync'd them. We also have another capability which allows us to resync as soon as it goes above a threshold. For example: On the primary system, run it as follows:
In the above example, secondary system would sync with the primary once but after that it would constantly check to see if the drift is going above 100 us. As soon as it does, the secondary system would automatically resync with the primary. It is recommended to select a large number like 60 or 100 since if we use a smaller number, then we will be resyncing all the time. On the other hand, if we chose a really big number like 1000 us, then we are allowing the systems to get out of sync from each other for 1 ms. On a Tiger Lake system, 100 us difference is usually happening in around 60-70 seconds or about a minute. So this program would automatically resync with the primary every minute or so.
Similar to ptp4l, an automation script is provided to setup vsync primary and secondary on both systems. see ./scripts/run_vsync.sh. Make sure to update config.sh accordingly unless if it's already updated for run_ptp.sh.
Similar to ptp4l, For user convenience, an automation Bash script is available in the scripts folder to facilitate vsync synchronization on both primary and secondary systems. Users simply need to update the config.sh file with relevant details such as the secondary system's address, username, and Ethernet address, among others. After these updates, run the scripts/run_vsync.sh script from the primary system. This script will automatically copy vblank_sync to secondary under ~/.vblanksync directory and execute the vblank_sync tool on both the primary and secondary systems in their respective modes. Note that the script requires the secondary system to enable passwordless SSH to execute ptp4l commands through the SSH interface.
The console will display logs from both systems, but each process, including ptp4l and phc2sys, will also generate its own separate log file. These files can be found in the ./logs directory at the root level.
Running vsync_test as a Regular User without sudo
While the ptp synchronization section provides methods for configuring passwordless sudo, it may be preferable to operate without using sudo or root privileges. The vsync_test application, which requires access to protected system resources for memory-mapped I/O (MMIO) operations at /sys/bus/pci/devices/0000:00:02.0/resource0, can be configured to run under regular user permissions. This can be achieved by modifying system permissions either through direct file permission changes or by adjusting user group memberships. Note that the resource path might change in future versions of the kernel. The resource path can be verified by running the application with the strace tool in Linux as a regular user and checking for permission denied errors or by examining /proc/processid/maps for resource mappings.
Option 1: Modifying File Permissions
Altering the file permissions to allow all users to read and write to the device resource removes the requirement for root access. This method should be used with caution as it can lower the security level of the system, especially in environments with multiple users or in production settings.
To change the permissions, the following command can be executed:
This command adjusts the permissions to permit all users (o) read and write (rw) access to the specified resource.
Option 2: Configuring User Group Permissions and Creating a Persistent udev Rule
A more secure method is to assign the resource to a specific user group, such as the video group, and add the user to that group. This method confines permissions to a controlled group of users. To ensure the changes persist after a reboot, a udev rule can be created.
Step 1: Add the Resource to the Video Group
Change the group ownership of the resource file to the video group using the following command:
Step 2: Modify Group Permissions
Set the group permissions to allow read and write access with:
Step 3: Add the User to the Video Group
Add the user to the video group using this command (replace username with the actual user name):
Log out and log back in, or start a new session, for the group changes to take effect.
Step 4: Create a udev Rule for Persistent Permissions
To make the permissions persist after reboot, create a udev rule by adding a new rule file in /etc/udev/rules.d/:
This rule ensures that the permissions are correctly set when the system boots.
Step 5: Reload udev Rules
After creating the rule, reload the udev rules to apply them immediately without rebooting:
Verifying the Changes
After applying one of the above methods, confirm that the user has the necessary permissions by running the vsync_test application without elevated privileges. If the setup is correct, the application should run without requiring sudo or root access.
Running ptp4l as a Regular User without sudo
Similarly, the ptp4l command can be executed by a regular user. This can be done by either altering the permissions or modifying the group for /dev/ptp0, or by creating a udev rule to ensure persistent permissions.
Reload udev Rules
However, phc2sys will still need to be run with root or sudo privileges as it involves modifying system time. Running it under a normal user account could be possible if permissions to update the system clock are granted to the application.
Debug Tools
When performing debugging, it is helpful to have the following libraries installed:
Debug Tool Usage
When encountering unexpected behavior that requires additional debugging, the installed tools can assist in narrowing down the potential problem.
intel-gpu-tools Provides tools to read and write to registers, it must be run as the root user. Example:
edid-decode is a tool used to parse the Extended Display Identification Data (EDID) from monitors and display devices. EDID contains metadata about the display's capabilities, such as supported resolutions, manufacturer, serial number, and other attributes.
To operate edid-decode, it is typically necessary to supply a binary EDID file or data. Here's an example of how to use it:
