OCPN messaging

[jongough]

There are flyspray requests (#FS1881, FS#1587) for a Guard Zone (GZ) for watching AIS targets to see if they are getting too close to a position regardless of whether they are on a collision course or even moving towards the position. Currently the AIS code handles all the processing of information regarding AIS targets and displays the alerts if deemed needed. The concept of a Guard Zone requires drawing the zone on the chart and then watching to see if any AIS target enters this zone.

The OCPN_DRAW (OD) plugin can draw the zone and check to see if any given lat/lon is within the zone. This is currently being done for the Boundary alarms using the Watchdog plugin. The Watchdog plugin communicates with OD through the OCPN messaging process. This seems to work well for this type of interaction where a single lat/lon is checked to see if it is either inside or outside a particular or any boundary.

If the GZ were to be drawn by OD then either the AIS code would need to query OD for each AIS target to ascertain whether it was inside a GZ, or would need to broadcast a message for each target so that others could do the check. At best, if the AIS code does the query, there would be one message for each target sent to OD and a subsequent response message saying the lat/lon is inside or not the GZ Boundary. So for each target there would be two synchronous messages. The more structured case would be a broadcast message for each AIS target, the Watchdog plugin would pick this up and query the OD plugin to see if an alert is needed. This would result in 5 synchronous messages for every AIS target. As there appears to be certain locations that have large numbers of targets (hence the work on scaling the targets) this would result in a large number of messages per timer event in the AIS code.

As mentioned the current messaging in OCPN is using a synchronous model and there is a high probability that that the time taken to process the messages could exceed the timer events causing them. This would either cause the timer to be ignored or double events to be processed.

Is there a need to re-examine the messaging within OCPN to allow for large numbers of messages that, at times, may take longer to process than the time available.

At the moment there are only a few messages being sent around the system, so broadcasting is acceptable. Does this need to change to allow directed messages as well?

Should mainline OCPN code use messages that are sent to specific plugins or should the mainline remain unaware of the plugins? The reverse, i.e. plugins sending messages to the mainline, also needs to be answered.

Do we need to start documenting the messages being sent and the responses to these? Do we need a more structured approach to the message content or should it be left to the parties directly involved in the communications?

Thoughts?

Jon

[seandepagnier]

Originally Posted by jongough

As mentioned the current messaging in OCPN is using a synchronous model and there is a high probability that that the time taken to process the messages could exceed the timer events causing them. This would either cause the timer to be ignored or double events to be processed.

You can handle the messages asynchronously if you want.

Is it really that slow to check if a point is in a boundary?

Is there a need to re-examine the messaging within OCPN to allow for large numbers of messages that, at times, may take longer to process than the time available.

What is the "time available"?

At the moment there are only a few messages being sent around the system, so broadcasting is acceptable. Does this need to change to allow directed messages as well?

Maybe because currently each plugin must check each message against each type of message. If we used mailboxes and/or subscriptions it would eliminate a lot of this.

Should mainline OCPN code use messages that are sent to specific plugins or should the mainline remain unaware of the plugins? The reverse, i.e. plugins sending messages to the mainline, also needs to be answered.

It is already doing it like with the wmm plugin.

Do we need to start documenting the messages being sent and the responses to these? Do we need a more structured approach to the message content or should it be left to the parties directly involved in the communications?

Thoughts?

Jon

I don't see why you couldn't document the messages, it's just a question if you want to rewrite the documentation every time the message changes.

Sean

[jongough]

Sean,
The current plugin api 'SendPluginMessage' appears to be a synchronous process. Each message is sent to each plugin that is currently active. The internal message is sent to an event handler. It is not slow to check a point in a boundary, but if there are many boundaries and many points to check then this will add up. I raised this due to the work going on to scale the AIS targets. Some have mentioned thousands of targets and if there are checks to see if boundaries exist anywhere in a 360deg arc then that dramatically increases the amount of work that has to be done.

The time available is set by the event timers that are being used. Most I have seen are 1sec intervals, but there may be some that are shorter than that.

Running on a grunty desktop machine is all very well, but if you are using an Android tablet or an ARM machine the processing of messages may become an issue. If it does not, then all well and good. If it does it would be nice to have thought it through a little beforehand.

Directed messages is one way, mailboxes another and subscribe yet another. What would be the preferred method(s)? These types of message would really require a message handler that is message agnostic, but gives all parties a standard interface. This would require certain message contents, i.e. recipient address, return address, message type, message number, etc, so that there can be directed responses.

I have found where wmm is sent the message in OCPN to get the variation, but this message is sent to all plugins not just wmm, also wmm is a standard part of OCPN now (was it always?). Do we want each plugin to have code added to chart1.cpp? I thought Dave was trying to get everything out of the mainline to make it easier to maintain. If we are not careful

As to the documentation, at the moment I am currently busy making the OD plugin suitable for release and getting its user documentation up to date. I will then start on the developer documentation and cover the messages that OD will respond to and the content of the response.

At the moment you have to go hunting in the code to understand what messages are being used, how and when. I just thought it may be an idea to make it a bit easier. Message content 'should' be relatively static, but even if changes are made to the 'body' the 'header' information should not change much. In this case the 'body' is the message type and attributes needed to ascertain the information needed and the 'header' is the information needed to allow a 'two way' conversation.

Jon

[rgleason]

Jon
Tomorrow l'll try an Ais alarm with the very very big ais-vdr file, just to see what happens.
Is there a way to judge the load of messages on memory, cpu etc?

[jongough]

Rick,
I would not expect memory to change, but if you do manage to load it up then the CPU usage would be higher than normal and there may be delays in processing messages, i.e. AIS seems slow, or slow chart updates.

The new AIS messaging and Guard Zones are invoked for each AIS message at the time the message has been successfully decoded. The AIS decode sends a message to all plugins, the Watchdog plugin gets this message and checks for Guard Zone alarms. For each of these it sends a message to OD which then uses the information to look at the specified Boundary, based on the GUID, to see if the the Lat/Lon provided is inside the boundary. Then OD sends a message back to the Watchdog plugin to process. When the Watchdog plugin has finished it returns to OD, which then returns to Watchdog which then returns to AIS and the next AIS message can be decoded.

If other plugins start to make use of the AIS messages that will elongate the processing as well. I would not expect a desktop machine to have much of an issue, but low powered devices, i.e. tablets, may start to show slow downs. I just don't know at the moment.

Jon

[seandepagnier]

If it's a performance issue, we could send the ais info in a struct instead of a message and the watchdog plugin could query the boundary by calling a function directly in the draw plugin rather than with a message.

Currently weather routing will get function pointers from climatology via messages, then after that it can simply call the functions.

[jongough]

I don't really want to tie too much stuff together, so using messages decouples the code. There is extra processing for the message to go in and out of JSON format, but apart from that there should not be too much overhead. Its just that most of this is probably happening on a single thread and may overload that. If we could make the messaging occur on a separate thread it may help the mainline code. The problem is I just don't know at the moment. I just want to see if there is a better way to do messaging within OCPN so that we don't get these problems.

[seandepagnier]

you can stick the message in a queue then process it in a background thread in your plugin if needed.

[bdbcat]

jon...

Lets try some large data sets against the current synchronous implementation. This is what we will ship in O4.2, so we need to know what to expect.

Later, for O5, I think a background thread implementation makes good sense, possibly extended to allow more precise addressing, etc...

Please keep me posted.
Dave

[rgleason]

Here are two screenshots first with the smaller Hakefjord.txt 1mb file and then with the AIS-VDR-LeaseOnLife.7z (5.24mb compressed, unpacks to 33mb) also named vdr-ais-cpuload-reaching-100percent.log which is a huge AIS file from LeaseOnLife.

These are available OpenCPN Beta File Thingie rguser rgpass
look under the nmea directory.

SurfacePro3 with Haswell i5 cpu I believe.

OpenCPN readings using Task Manager
1. No Ais Cpu 0-0.7% Mem 139 mb Network 0%
2. Hakefjord.txt Cpu 2-4% Mem 146 mb Network 0%
3. Vdr-ais...log Cpu 25%-30% Mem 146 mb Network 0%

#3 was playing TDad VDRServer.py python script for both Hakefjord.txt and Vdr-ais...log

While these cpu loads certainly do not cause trouble on this machine, I think it might be a good idea to have in a future version of Opencpn, a structured messaging system, with perhaps critical external functions that can be called and shared by plugins to reduce the message load and cpu load as suggested by others. --background thread...bdcat.

However, Sean and Dave would be able to advise if these loads would be a problem on less powerful cpu's as currently implemented. Perhaps a warning is needed, but most of these users are quite sophisticated users, so the warning may not be needed.