Problem with long chain - framerate drops after different time

[mzdunek92]

EDIT: I have reinvestigated my problem and now can describe it better.

Hello,
My display is built from 2 parallel chains of 64x32 P4 modules, 11 modules long. I am using these parameters:

• chain 44
• rows 16
• transformer
• PWM bits 3
• PWM nano 130
• GPIO slowdown 2
• brightness 50
• subpanels 2

Everything works fine when I run app from rpi's console.
Unfortunately when I run your application from watchdog or crontab -e, I got problem with framerate and brightness drop. They drops about 50% (from ~160Hz to 80Hz, brightness drops twice too). It happens with different time intervals, sometimes 2 minutes all is good, 1 minut bad, 5 minutes good...
I use minimal distro of Raspbian Lite Stretch (Raspberry Pi 3 v 1.2 model B).

Why is this happening?

Thank you for your help

[zzaurak]

I have the same problem, I thought it is the Raspberry CPU throttling because of overheating. But I couldn't confirm it yet.

[hzeller]

I'd make sure that the crontab program does not set some nice-settings that are then inherited (something I'd suspect a cron daemon to do).
Check out if the update thread shows up as RT (realtime) thread or if it couldn't set these settings.

Also I'd use the --led-daemon flag in the program to put it in the background, so that you don't have to rely on the crontab to get that done correctly.

Other than that, for such a long chain I suggest to set the disable RT throttle setting, i.e. remove the comment in #DEFINES+=-DDISABLE_RT_THROTTLE in lib/Makefile and recompile.

[hzeller]

Did you figure it out ? I suspect your cron is just running things in a very low CPU setting and does not allow the program to use all the CPU and realtime settings it needs.

[mzdunek92]

I would like to reopen this issue, as I have tested your suggested actions.

Disabling throttling and setting nice parameter to -19 did not remove this framerate dropping effect.
Update thread is always marked as RT on htop preview.

Do you have any other suggestions? We have used this software in other projects with smaller displays (like 200-300 px width max instead of 700) and there were no such problems.

[hzeller]

The longer the display chain is, the more you are susceptible to the operating system interrupting you creating changes in the time it takes to refresh a whole screen. If in a time period there are less refreshes per second, you see this as a brightness deviation.

A chain of 44 is very long. Typically you start seeing drop in framerate after about 8 or so, so 44 is definitely stretching it (ususally, you'll get a couple of hundred Hz refresh rate with 16 high panels). You are also only using 2 parallel chains. You should consider using all 3 chains as they don't cost any extra CPU to use, but now you only need about 30 panels in each chain, which would definitely help.

There are a few potential reasons why the variations happen:

• Either it is just that the kernel is interrupting the task more often even though it requests not to. There is probably not much that can be done with the generic stock kernel that is optimized for interactive user experience. In that case, I'd suggest to play with a realtime kernel. @russdx found a ready made kernel configuration in https://github.com/guysoft/RealtimePi but I haven't tried that yet. Might be worth an experiment, in particular since you have such a long chain.
• Your display is pretty big, so this means a lot of data. The display works best if the data can be kept in CPU cache, because the RAM is very slow on the Raspberry Pi. Reading data from the RAM alone would not be fast enough to send to the display, so if there is any other process that competes for the CPU cache, you will have issues. So avoid running anything else on the Pi you don't need.
  An example for processes that mess up things: someone found the reason for their brightness glitches every minute because there was a cron running updating the ntp (I guess you read the README for the details).
  Also check if there is anything running in cron: it wakes up every minute to look for things to do, so will also mess with your timing every minute.
  If you use programming languages that use garbage collection, they will have a random impact on the cache.
• It could be some other hardware limitation in the Pi: whenever there is something else using the memory bus (e.g. networking), it might slow down the GPIO.

I don't understand how you run the application from the crontab ? Are you just displaying something shortly and then exit ? I would rather suggest to have the application running all the time, only started at boot time of the Pi and build a way to trigger it showing things you'd like to show. The problem with starting and stopping is that you don't keep the cache warm, which will not work well in your situation (see above).

Finally, you can try to smooth out the framerate fluctuations by setting it to some fixed value; there is an option in a more recent version of the library, FIXED_FRAME_MICROSECONDS in lib/Makefile. It helped people that have seen slight fluctuations in brightness to get an entirely smooth output. However with this procedure you'll fix it at the lowest framerate observed, which is pretty low already because you have a very long chain and only use two parallels.

So suggestions in sequence of difficulty

• Make your application run continuously, not per cron. Start it at boot time (simplest way: put it in /etc/rc.local) Disable anything in cron or stuff that runs regularly. Avoid programming languages that do garbage collection. Preprocess all your image data if calculation takes too long. Maybe even pre-process image data, store on disk and play from there (e.g. using the Content Streamer set of APIs (see example how to use in utils/led-image-viewer.cc)).
• Smooth out irregularities with the FIXED_FRAME_MICROSECONDS setting.
• Try a realtime kernel to run on your Pi.
• Utilize all three chains. They don't cost any extra CPU, but requires to push 30% less data for the same display. This will require you to change the wiring and write some PixelMapper to map the pixels to the same physical layout, but it will be worthwhile.