Tracing packet processing latency

Job ID: 36300634

Budget: $30 – $250 USD

I want to be able to accurately test packet processing latency throughout the linux kernel. This means the amount of time that is spent in the linux kernel. For example, I have already written a program that uses socket programming (in C) and uses the SO_TIMESTAMP option in setsocket() to require a timestamp when the kernel receives a UDP packet headed for a specific port. The user space server code then compares this time stamp timeval object with the current time using gettimeofday. The difference between the two should be the packet processing latency.

However, I would like to also test the other way around. How long it takes a packet to arrive to the network interface card, or just right before leaving the linux kernel. In my current understanding, I can use eBPF and embed a code somewhere there (not totally sure) to have event based triggers based on recieving a specific udp packet headed for a specific port send by user space code.

This is to trace a single kernel linux stack. However, I would like to expand on this technology to trace the total packet processing latency with the addition of containers. Usually, such as in overlay networks (multi host system with docker containers on different hosts), packets have to travel through two kernel stacks, the host machines and the containers. I would like develop a method to be able to trace the whole packet processing latency, from the user space of one container to the user space of another container.

I am not sure if this requires host time synchronizing, using NTP and related technologies. But this is what I want to achieve. I want to be able to accurately this latency. In the end, I want C code (implemented with eBPF if possible), and can accept python code to implement eBPF)