Packet loss

If the receiver isn't in the same room as the transmitter, some packets will get lost. Let's see how this affects our video. LQ(link quality) tells us how many frames reach the receiver.

LQ

90.00%

I frame interval

30 frames

00:00 / 00:30

So despite only losing 10% of the frames, the video is mostly useless since losing a frame prevents the decoding of future P frames until the next I frame.

Now I still want my video stream to work even with 10% of packet loss. So if you did some testing, the issue can mostly be resolved by reducing the I frame interval, then frame drops don't matter much. But this brings us back to our JPEG codec which performs terribly...

Forward error correction

Fortunately there is a simple solution: send each frame twice. Then there is only 1% chance of a frame loss at 90% LQ. Easy win right? Well... Let's take a look.

LQ

90.00%

00:00 / 00:30

This approach is called 1+1 forward error correction. But while the video is much more stable, the quality seems to have degraded? The I frame interval is still 1s, but if you send each frame twice, your budget drops from 15.6kBps to 7.8kBps. And you pay this penalty even at 100% LQ!

At this point in time there is a decision to make. We can either build our system with a two way link where the receiver tells the car that the frame 20 was lost and it should send it again, or we can use forward error correction and I frames and a few other tricks we'll see later.

It turns out two way link is in a certain sense optimal since you can get great video quality when LQ is high while tolerating even very low LQ levels by resending frames multiple times. If reverse communication is free and instant, there is no reason to use FEC.

However the implementation of a two way link turns out to be quite difficult(simple example: what if the packet requesting the frame again gets lost?) so we decided to work on a better one way link instead.

Surviving 90% LQ

So our 1+1 FEC has two main issues we'd like to fix: it reduces the bitrate by 50% and it still lets a few frames get lost.

Now the theory of forward error correction(and linear codes) is interesting, but we won't go into details. For now let's just say that we can "easily" integrate something like 8+4 FEC which sends 8 frames and then 4 more "repeats" that manage to fix any 4 dropped frames(I'm simplifying here since we haven't yet split the frames into equally sized packets). Let's see how that helps us.

LQ

90.00%

00:00 / 00:30

So here, to get a failure, 5 or more out of 12 packets must be lost, at 90% LQ this is very unlikely to occur(Nice binomial distribution exercise) with the probability of 0.4% of the block having enough damage to not be fully fixable.

8+4 here wasn't chosen randomly, it is another OpenIPC default.

Now we still have to pay 33% bitrate cost, but you most likely didn't see any I frame waits. so definitely much better than the 1+1 FEC right?

So why not use 100+20 FEC and save even more bitrate with even lower odds of failure? Well there is one issue we are neglecting: latency.

What if the first packet in the 100+20 FEC block is lost? Well we have to wait for 100 frames to be sent(we need data about all 100 frames to compute the repair blocks) to compute what the missing block was. This will take about 4 seconds. So we might as well just wait for the I frame instead!

And even if we have no failures, we spend 20 frames sending FEC repairs causing a latency spike (that we don't simulate yet).

This is not usable for a low latency system so we'll need to be smarter about how we approach things. Still, 8+4 FEC turns out to be quite a useful baseline to work from allowing us to retain the video link even with some error rate though with modest bitrate cost. We'll take this as our new baseline.

Dealing with the real world

With the packet loss simulations we were operating in a world without latency, where every frame can be put into a single packet and forward error correction is easy and just handles varying frame sizes.

Now we actually have to fit our frame data into lora packets, decide how to do that, which parameters to use to set up the communication, how to deal with packet loss on the level where it actually occurs and how to deal with latency.

Where to even start...

LoRa

So semtech promises us 15.6kBps (but hey, if you are from Semtech by any chance, how do I unlock the 25kBps mode? 1.6MHz BW would be really nice to try :) ). Well that is the theoretical bitrate we can achieve, but it doesn't include LoRa overhead.

Let's see what's actually possible. The main two parameters of each LoRa packet are the spreading factor and the bandwidth. To achieve the 15.6kBps data rate we need to use the lowest spreading factor which is 5 for the LR2021 and the 1000kHz bandwidth. These parameters do reduce our range, they in fact represent the shortest range LoRa mode that LR2021 supports, but they still allow for very long range communication especially on lower frequencies.

The problem with 15.6kBps however, is, that it includes the preamble, packet header and CRC check. We'll go through each part

Each lora packet requires a preamble, we get to decide how long it is, but for our setup, the general recommendation is to use a 12 symbol preamble. At SF5 and 1000kHz BW, one symbol takes 32 microseconds. Including the additional 6.25 synchronization symbols, the preamble portion takes about 0.584ms.

Header contains the length of the packet and some lora parameters, it also includes a checksum for itself and uses stronger 4/8 error correction. Its 20 information bits are 2.5B, but at SF5 the header takes 8 symbols, or 0.256ms of airtime.

CRC (cyclic redundancy check) is sent after the data, it can be thought of as a hash of the data and is used to ensure there are no errors in the data stream. It is 2B long.

In a lora packet we can send at most 255B of data. With these settings, including the explicit header and CRC, the packet takes 17.319ms on air. That leaves about 14.72kBps of payload before any spacing, about 5.8% below the nominal 15.625kBps. If we allow 0.7ms to prepare the lora module for the next packet, we get about 14.15kBps.

With this setup we send a packet once every 18ms or 55.5 packets/s.

I'm skipping over a lot of details, but let's say this is enough info for now.

Dealing with the latency

For now we'll ignore packet losses and FEC to keep things simple.

h265 encoder takes in a frame from the camera and gives us a payload that we then have to split into 255B packets (or smaller, but then we lose bitrate) to send over LoRa.

Another simplification we'll make, is to always use 255B packets, this reduces the FEC complexity and simplifies frequency hopping (we'll cover later), if the video payload is too small we'll simply use padding to fill the packet(essentially zero bytes).

Now we can simulate latency under these assumptions.

Bitrate

10.0 kB/s

00:00 / 00:30

Frame merging

Two things stand out in that simulation.

First, latency jumps once per second. Those are the I frames. They can't borrow anything from the previous frame and the encoder tries to keep quality approximately constant, so at 14kBps an average I frame is about 1600B against about 420B for a P frame.

Second, and worse, latency runs away long before the link should be full. Slowly raise the bitrate and somewhere around 9.5kBps the delay stops recovering, even though our link can carry 14.15kBps. So where did the other third go?

Into padding. Each frame must end with a full 255B packet, so on average we waste about 155B per frame. At 30 frames per second that's about 4.7kBps: add the 9.4kBps of video and the link is full.

The fix is simple: if the next frame is already waiting when we start a packet, put its bytes in the remaining space. We are in a fixed packet schedule so this improves our utilization without any latency cost.

With variable packet sizes we could sometimes save a bit of latency without packet merging, but again at a cost of significant code complexity.

Video bitrate

12.0 kB/s

One frame per packet

Camera

With frame merging

Whole 30 s clip

	Padding per frame	Average RF delay	Frames over 200 ms
One frame per packet	–	–	–
With frame merging	–	–	–

Scroll sideways through the whole clip. Each line joins a camera frame to the start of its first packet, so the more a line leans, the longer that frame waited. A red line means the frame finished arriving more than 200 ms after the camera produced it. Each column is one 255 B packet: colors identify frames, orange is an I frame, pale grey is padding. A fully grey column is an empty packet. Hover or tap a packet to inspect it.

Move the bitrate up, scroll along the clip and compare the two. The top row keeps wasting space and falls further behind the camera, the bottom one only pads when it has caught up. At 14kBps average RF delay in this simulation drops from about 3.2s to about 95ms.

Bitrate

14.0 kB/s

00:00 / 00:30

We can now use nearly the whole link, but the once per second spikes are still there, reaching about 500ms at 14kBps. Merging makes better use of the packets, it doesn't make I frames any smaller.

Putting it all together

We've ignored link quality in our packet scheme. So let's bring back our 8+4 FEC, this time on actual LoRa packets. Every 12 packet slots now carry 8 packets of video and 4 repair packets, and each packet can be lost on its own. Any 8 out of the 12 are enough to restore the block.

That 8+4 FEC leaves us with about 9.4kBps for video.

Bitrate

9.0 kB/s

00:00 / 00:30

We can see that we get very frequent latency spikes. They are caused by sending the 4 error correction packets consecutively, this takes 72ms during which the receiver doesn't receive any video frames(assuming no packet loss). This is also nicely visible in the RF latency chart as the sawtooth pattern.

So this is our baseline: 360p at 30 frames per second over LoRa, in a simulation that survives 10% packet loss without a return link.

We are not done with the tuning yet though. The I frames still cause spikes of up to 700ms, the repair bursts add a sawtooth to every frame's latency, a lost block can still freeze the picture until the next I frame, and a third of our link goes to repairs even when nothing is lost. That's what we'll be working on from here.