Rpi Developer says "slimm(ing) down your application". How ?

[NahanniV]

OpenCPN on Rpi is working very well aboard, except for what seems like one problem: it sometimes runs out of video memory.

This only seems to happen after running for a long time and active navigation.

I'm using the new VC4 driver for OpenGL on the Rpi since this seems to give the best performance, uses the least resources and is the least bugy.

The problem gets reported in the system log indicating that the VC4 driver can not get memory. The CMA memory is already set to 256M and the developer says increasing that will not likely help, instead he suggests slimming down the APP.

There are some configuration parameters available to limit memory usage, but I'm not sure that they are working:

GPUMemorySize=128 ; is this even implemented, could not find it in the code ?

MEMCacheLimit=256
NCacheLimit=10

Hi Eric,

I am getting an error message from your VC4 code.
It happens infrequently but approximately once a day if the system is in use.
The distribution is specific for marine navigation: OpenPlotter http://sailoog.com/openplotter 
The application that I think is causing the problem is a marine charting program called OpenCPN.
It displays a moving map display and other information, sometimes quilting together several charts to create the display. There is also rotation for course up modes and display of other information such as routes and tracks.

The error that I see is:
  pi@openplotter:~ $ dmesg
[21965.710804] [drm:vc4_bo_create [vc4]] *ERROR* Failed to allocate from CMA:
[21965.710817] [drm]                            V3D: 176852kb BOs (345)
[21965.710821] [drm]                     V3D shader:    412kb BOs (100)
[21965.710824] [drm]                           dumb:   3088kb BOs (2)
[21965.710833] vc4_v3d 3fc00000.v3d: Failed to allocate memory for tile binning: -12. You may need to enable CMA or give it more memory.
pi@openplotter:~ $

I am not suggesting that this is an error in your code, I may simply be running out of Graphics memory.
It appears that the memory is already set at 256M.

pi@openplotter:~ $ dmesg | grep cma
[    0.000000] cma: Reserved 256 MiB at 0x1ec00000
[    0.000000] Kernel command line: 8250.nr_uarts=0 cma=256M bcm2708_fb.fbwidth=1024 bcm2708_fb.fbheight=768 bcm2708_fb.fbswap=1 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1 root=/dev/mmcblk0p7 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
[    0.000000] Memory: 685600K/970752K available (8192K kernel code, 612K rwdata, 2148K rodata, 1024K init, 821K bss, 23008K reserved, 262144K cma-reserved)
[    3.600272] vc_sm_cma: module is from the staging directory, the quality is unknown, you have been warned.
[    3.602447] bcm2835_vc_sm_cma_probe: Videocore shared memory driver
pi@openplotter:~ $ 


I see that you have implemented a patch to allow more than 256M, but I'm not sure if that is in the Kernel version I am using ( 4.19.30-v7+) ?

I am also using the Fake KMS version vc4-fkms-v3d since my monitor would sometimes be blank at startup with the other version.

What exactly do I need to do in order to increase the CMA memory size ?

Also there is quite a bit of confusion about GPU memory. It seems that this parameter is not used by your driver but many people still think it is.

Perhaps you could point me to a clear description of how to configure memory for your driver; or, if no description exists, perhaps you would be the person to write it.

Cheers,
John Middleton.


Eric Anholt
Attachments
Apr 3, 2019, 6:14 PM
to me


It's already in.  You can hack your kernel build to increase the value
in the overlay, but honestly if you're running out at 256, another 128
or so (you can't go too high or linux will just fail) likely isn't going
to help you, and you probably need to work on slimming down your
application.

[seandepagnier]

Are you using vector or raster charts?

[NahanniV]

Originally Posted by boat_alexandra

Are you using vector or raster charts?

Raster Charts, that's all I have for this area.

No Texture Cache.

[transmitterdan]

Try the following one at a time then in combinations:

Turn off/on texture cache. See if one is better than the other.

Reduce graphics memory from 256MB to 64MB. I know it’s counterintuitive but try it anyway.

Set swap file size to zero. On an SD card memory swapping is worse than useless. You may have to set a non-zero swap file size to compile O. I run O with zero swap file size.

Get a real flash hard drive if you want to have a swap file.

Don’t run unnecessary background services that you don’t use such as SSH and the like.

[NahanniV]

Originally Posted by transmitterdan

Try the following one at a time then in combinations:

Turn off/on texture cache. See if one is better than the other.

I don't think the Rpi supports texture cache, so I have it off.
If this is the case, it would be nice if the option was greyed out when Rpi hardware is detected.

Originally Posted by transmitterdan

Reduce graphics memory from 256MB to 64MB. I know it’s counterintuitive but try it anyway.

I had the same error with 16M and 256M. It is CMA memory that seems to run out.

Originally Posted by transmitterdan

Set swap file size to zero. On an SD card memory swapping is worse than useless. You may have to set a non-zero swap file size to compile O. I run O with zero swap file size.

Get a real flash hard drive if you want to have a swap file.

Don’t run unnecessary background services that you don’t use such as SSH and the like.

All good suggestions, but in normal use I am not using any swap.
If I start Cromium then I have a problem with slowing down due to swaping.

[bdbcat]

NahanniV...


re:
"I don't think the Rpi supports texture cache, so I have it off."


There is no reason why rPI should NOT support texture cache. What happens when you turn it on?
Also, is the memory fault present if you leave the device running with literally no user input for extended time?

Thanks
Dave

[NahanniV]

Originally Posted by bdbcat

NahanniV...


re:
"I don't think the Rpi supports texture cache, so I have it off."

There is no reason why rPI should NOT support texture cache. What happens when you turn it on?

OK, I thought I remembered Sean posting on this forum that the Rpi did not support Texture Compression with caching.

Perhaps he will chime in.

I can certainly try it.

Originally Posted by bdbcat

Also, is the memory fault present if you leave the device running with literally no user input for extended time?

Thanks
Dave

No, it never fails if it is just sitting there at anchor. It can run for a week or more.
It almost always has high precision tracking and the anchor alarm turned on.
Also the wind instrument on the dashboard.
When we were travelling down the ICW what happened on many occasions was that I would set-up a route in the morning and follow it all day without touching the Rpi, then just as we were entering an anchorage I would want to: pan, change the scale, switch to another chart, or switch from course-up to north-up and it would fail.
Usually it seems to be a single error from VC4 with a few lines in the system log, and the display mostly freezes except for the cursor and some pop-up windows (I can still ssh into the Rpi from another computer).

Once I saw OpenCPN actually crash (see Crash report thread).

For the most part the Rpi works surprisingly well. The VC4 driver (with latest updates) is much improved since I tried this a few years ago. Hopefully this issue can be resolved.

[NahanniV]

Originally Posted by bdbcat

NahanniV...


re:
"I don't think the Rpi supports texture cache, so I have it off."


There is no reason why rPI should NOT support texture cache. What happens when you turn it on?
Also, is the memory fault present if you leave the device running with literally no user input for extended time?

Thanks
Dave

I tried to turn on Texture Compression and Caching, but could not.

There are some messages in the log regarding compression:

12:44:07 EDT: OpenGL-> No Useable compression format found
12:44:07 EDT: OpenGL-> Not Using compression
12:44:31 EDT: OpenGL-> No Useable compression format found
12:44:31 EDT: OpenGL-> Not Using compression

This is a bit confusing, I tried to set it, then when I went back it was not set.
If the setting is not allowed, it would be best if it could be "greyed out".

Also the Frames Per Second display never shows ?

[bdbcat]

NahanniV...


Compression: Interesting....The GL driver does not support compression. I have never seen this before....


Is your logfile (GL section) posted anywhere already? If not, could you post it?


Thanks
Dave

[NahanniV]

Here is the logfile up to and including the OpenGL:

________
13:08:42 EDT: 2019-04-21
13:08:42 EDT:  ------- Starting OpenCPN -------
13:08:42 EDT: 5.0.0
13:08:44 EDT: wxWidgets version: wxWidgets 3.0.2 Linux 32 bit wxGTK
13:08:44 EDT: OS Version reports as:  4.14
13:08:44 EDT: MemoryStatus:  mem_total: 976 mb,  mem_initial: 0 mb
13:08:44 EDT: SData_Locn is /usr/share/opencpn/
13:08:44 EDT: PrivateDataDir is /home/pi/.opencpn
13:08:44 EDT: Using existing Config_File: /home/pi/.opencpn/opencpn.conf
13:08:44 EDT: Setting Viewpoint Lat/Lon 23.5329, -75.7731
13:08:44 EDT: Setting Ownship Lat/Lon 23.5329, -75.7743
13:08:44 EDT: Switch InlandEcdis mode Off
13:08:44 EDT: Styles loading from /usr/share/opencpn/uidata/styles.xml
13:08:44 EDT: No styles found at: /home/pi/
13:08:44 EDT: No styles found at: /home/pi/.opencpn/
13:08:44 EDT: Using SVG Icons
13:08:44 EDT: Detected display size (horizontal): 271 mm
13:08:44 EDT: Getting DefaultSystemLocale...
13:08:44 EDT: System default Language:  en_CA
13:08:44 EDT: Config file language:  en_US
13:08:44 EDT: Adjusted App language:  en_US
13:08:44 EDT: Opencpn language load for:  en_US
13:08:44 EDT: Loading catalog for opencpn core.
13:08:44 EDT: Getting DefaultSystemLocale...
13:08:44 EDT: Opencpn language set to:  en_US
13:08:44 EDT: Loading Configs catalog: /home/pi/.opencpn/Configs/configs.xml
13:08:44 EDT: Creating MyFrame...size(1024, 716)  position(-2, -2)
13:08:45 EDT: Creating glChartCanvas
13:08:45 EDT: Metrics:  m_display_size_mm: 271     wxDisplaySize:  1024:768   
13:08:45 EDT: Loading Tide/Current data source: /usr/share/opencpn/tcdata/HARMONIC.IDX
13:08:46 EDT: OpenGL-> Renderer String: Gallium 0.4 on VC4 V3D 2.1
13:08:46 EDT: OpenGL-> Version reported:  2.1 Mesa 13.0.6
13:08:46 EDT: OpenGL-> Texture rectangle format: de1
13:08:46 EDT: OpenGL-> Using Vertexbuffer Objects
13:08:46 EDT: OpenGL-> Using Framebuffer Objects
13:08:46 EDT: OpenGL-> Using FBO Stencil buffer
13:08:46 EDT: OpenGL-> Using Stencil buffer clipping
13:08:46 EDT: OpenGL-> Using Scissor Clipping
13:08:46 EDT: OpenGL-> Not Using compression
13:08:46 EDT: OpenGL-> Minimum cartographic line width:  1.0
13:08:46 EDT: OpenGL-> Minimum symbol line width:  1.0
13:08:46 EDT: ChartDB Cache policy:  Application target is 487 MBytes
13:08:46 EDT: Loading chart db version: V018

[Moo]

Originally Posted by bdbcat

NahanniV...


re:
"I don't think the Rpi supports texture cache, so I have it off."


There is no reason why rPI should NOT support texture cache. What happens when you turn it on?
Also, is the memory fault present if you leave the device running with literally no user input for extended time?

Thanks
Dave

I see this on my test linux box at home. It is running a gps puck providing NMEA 0183 and has an SDR attached with AISdeco running also providing 0183 via UDP. An old Nettop box.

It runs for over a week but eventually takes over with CPU ramping up over time to over 90% (as shown for OCPN with Top) where it grinds to a halt and opening say AIS target list just produces a blank window.

This is basically the same as I have seen at sea but the accumulating track distance causes the grinding halt to occur inside 24hrs (this on a RPi).

Anything I can run to help debug?

[NahanniV]

Originally Posted by bdbcat

NahanniV...


Compression: Interesting....The GL driver does not support compression. I have never seen this before....


Is your logfile (GL section) posted anywhere already? If not, could you post it?


Thanks
Dave

I can only understand this stuff well enough to probably sound foolish, but here goes:

Looks like the Rpi VC4 driver supports ETC compression.

OpenCPN seems to only support ETC compression for GLES (#ifdef ocpnUSE_GLES)

Is that true ? Why ? Is it worth changing ?

[bdbcat]

NahanniV....
Your comment not foolish. This may be true, and worth looking into.
However, if we have a texture leak causing memory exhaustion, that needs to be resolved first.
I'll dig into this a bit soon, when I get some bandwidth available.


Thanks
Dave

[NahanniV]

Originally Posted by bdbcat

NahanniV....
Your comment not foolish. This may be true, and worth looking into.
However, if we have a texture leak causing memory exhaustion, that needs to be resolved first.
I'll dig into this a bit soon, when I get some bandwidth available.


Thanks
Dave

FYI:

I also had problems with: no OpenGL, software OpenGL and OpenGL on the legacy driver, but in those cases the system just became extremely sluggish and sometimes the display was corrupted (black screens) same as reported by Moo, no errors were reported in the log.
OpenGL on the experimental driver with fake KMS works the best, but you need to update to the latest versions.

Also, when power allows (sunny, or motoring) I run OpenCPN in parallel on a Win10 system and have never had a problem.

[transmitterdan]

Just a quick note...

I updated my RPi to latest release of everything. Whenever I turn on the “real” GL driver (not fake) and then run O5 the CPU maxes out all cores. In htop it says there are 4 opencpn threads running flat out. I waited an hour and it stayed the same. Processing NMEA data was very sluggish.

If I run the “fake” GL driver it runs very low CPU usage. Screen updated much faster and NMEA processing did not tax the system.

So something seems to be kicking off several threads when the “real” GL driver is active. Does O sense some driver capability and then start several threads? If so, that’s a “questionable” thing to do on an RPi. Maxing our the CPU can exacerbate the power supply issue.