Signal K

[Tehani]

Originally Posted by rgleason

Jose Louis

Would you provide a good link for this please? It is 404 page not found.
"Something like (https://github.com/SignalK/signalk-s...GINS.md#deltas):"
Thank you.


You bring up some good points here, (AIS for example), and you suggest a dedicated external device to process to 1 protocol to reserve the computer for use by OpenCPN and plugins. Maybe I am being thick here, but what is the protocol suggested? Some extension of nmea0183? nmea2000? signalK?

Rick,

Este es el link, que indicó Teppo anteriormente:
https://github.com/SignalK/signalk-s...GINS.md#deltas

I try to answer your question.
Evaluating the current situation, we see that the use of NMEA0183 has gone well, and that it is convenient to maintain it for some reasons:
1) Compatibility with previous versions.
2) Less development effort.
3) NMEA association continues to publish versions of NMEA0183. It seems then that NMEA0183 is not dead at all ...

But of course, as we know, we are reaching limitations:
1) It is difficult to expand with more data processing in the O core.
2) NMEA0183 transducers speak very slowly and use RS422 bus. Therefore, O must maintain its MUX concept in absence of external multiplexers.
3) The inclusion of other protocols may compromise the performance of O.
4) Communications in N2k go much faster, and more refresh rate.

Therefore, my proposal is to restructure the treatment of NMEA0183, so that it is totally external to the O core, but that shares the data table with O and all its plugins (Of course in binary format: address relative to structure-> value, never ASCII key-> value).
If we analyze communications at a low level, only systems such as the CAN bus can filter PGN's on the fly and at the hardware level. Other buses require CPU intervention. With this I intend to answer some intervention of Dave: This processing, even with the subscription method, always consumes CPU time and requires an associated task that performs the function of filtering and parsing.
I think that in this way, it will not be a problem to continue with NMEA0183 and its extensions. We can also establish tricks (Inside TwoCan for example) to access N2k data that has no equivalence in NMEA0183.
The philosophy is to consolidate and optimize before doing something new.

José Luis

[teppokurki]

I apologize, it seems I have failed to communicate my message so that peple understand it.

Originally Posted by Tehani

I can't give credit to what is being said here.
It seems incomprehensible that people do not understand that in all cases of signalK processing, calls to subscription functions involve starting a process that will analyze and compare the ASCII streams received by the node with the string "electric.batteries.1 .voltage ". In no case "electric.batteries.1.voltage" is a direct access address.

In the simplest case you are correct. But the API and implementation can be fashioned so that you only incur the overhead once.

Let's imagine I am parsing a csv file with headers where the first column contains values for "electric.batteries.1.voltage". Now I can call a function like

multiplexer.publish("electric.batteries.1.voltage" , theValue)

for each and every line. Then things are as you said.

But if the multiplexer exposes an API that allows me to look up a function to call for each data item when I process the header, like

firstColumnCallback = multiplexer.getCallback("electric.batteries.1.volt age")

then I can call that function for every row's first column, with no string comparisons. Then if the multiplexer is smart it won't do any string wrangling to dispatch the data to subscribers.

The equivalent can be done for NMEA 0183 and NMEA 2000 parsers, where the fields are (mostly) deterministic once you know the sentence/pgn.

The distinct benefit of using string keys is that new ones can be processed without prior knowledge. The multiplexer can pass any data, not just what was hardcoded into it.

Btw you can also eat and save the cake: you can have special shorthand methods for subscribing to well known data like position and speeds while keeping support for generic data handling and position and speed available over both APIs.

{
  path: 'navigation.position',
  value: { longitude: 24.7366117, latitude: 59.72493 },
  context: 'vessels.urn:mrn:signalk:uuid:a9d2c3b1-611b-4b00-8628-0b89d014ed60',
  source: {
    label: 'n2k-sample-data',
    type: 'NMEA2000',
    pgn: 129039,
    src: '43'
  },
  '$source': 'n2k-sample-data.43',
  timestamp: '2014-08-15T19:00:02.392Z'
}

I think it's an ASCII barbarity. And only to report a sad coordinate ... Can you count how many characters are in this message?

Well, in documentation it needs to be in textual format and when talking in JavaScript context object literals and JSON objects are close enough.

I meant that the structure of a single message should be something like this and I have tried very hard to say that do not take the JSON examples literally - the structure matters, not the presentation in JSON format!

So please squint your eyes and replace the JSON object with a C++ class properly typed and all, timestamps as proper timestamps (is there a C++ construct for millisecond accuracy timestamps?) etc etc.

Repeat after me: let's not use JSON strings for passing data around inside a single process. Even SK Node server doesn't do that, why should OpenCpn?

It is a complex task and subject to the NMEA2000 certification. I do not believe that through an NGT-1 we are exempt from that responsibility.

This is the reason devices like NGT-1 and iKonvert exist - they enable both read and write access to an NMEA 2000 bus without forcing every single piece of connected software to be N2K certified. They take care of low level protocol details and for example if our application goes crazy and tries to flood the bus with messages they limit the transmit rate to observe N2K specification limits.

What they can not do is check that the data written makes sense. If we start sending bogus values for for example wind speed the interface will pass those happily along and other equipment will treat them as legitimate data, as long as the pgn format is correct.

You should evaluate that to decide if it is safe to install an N2k node on the same host that runs everything else.

This is done routinely. Think about the navigation PCs that racing boats have - they use the PC for multiple purposes, including interfacing with the N2K bus.

As I said before, there are speed problems because some data in N2k is refreshed 10 times per second (position, wind, heading, speed, course, and attitude).

What speed problems? You mean that a value changes 10 times a second is a problem? People are using RPis to connect to very loaded N2K buses without any problems.

The main body of Ocpn also needs to operate with databases for the processing of various AIS targets, because they are not repetitive values. You have wondered what would happen if we are surrounded by >100 AIS devices, sending their VDMs, and those messages must go from NMEA0183 / N2k to SignalK, and then again to NMEA0183, and you can not miss any packet because they come from different ships and can also be multi-messages?

This is just fearmongering. AIS input is limited by the VHF bandwidth - even if you were in range with all the vessels on earth there is an upper limit to the data rate. AIS receivers typically have 38400 baud connections and the native AIS data encoding is 9.6 kbps. If the software is buggy and messes or misses data then it is a problem, but from a performance problem AIS is not a problem at all on any computer you would run OpenCpn on.

Please do not invent problems. Can you point me from the previous discussion a message from somebody else than you that says we should convert AIS from NMEA0183 to Signal K and back to NMEA0183? Even SK Node server explicitly keeps all NMEA0183 data available in original format without any conversions and in fact I use this feature on my own boat for AIS data access on my tablet.

[teppokurki]

Originally Posted by Tehani

(Of course in binary format: address relative to structure-> value).
I think that in this way, it will not be a problem to continue with NMEA0183 and its extensions. We can also establish tricks (Inside TwoCan for example) to access N2k data that has no equivalence in NMEA0183.

I don't understand what you mean. Use NMEA0183 (that is text) but in binary format - how? Address relative to what structure - instead of a preparsed double value a plugin would get an address "start parsing double here? Can you explain how my fancy pants gauge plugin would get updates for SOG and voltage for battery one - a single API or one for NMEA0183 and another for TwoCan data? What if I received the voltage not from N2K but from my Victron battery monitor via VE.direct and wanted to publish that to my gauge and all other dashboards and plugins, how do I do that?

[Tehani]

Thank you Teppo,

Originally Posted by teppokurki

In the simplest case you are correct. But the API and implementation can be fashioned so that you only incur the overhead once.

Let's imagine I am parsing a csv file with headers where the first column contains values for "electric.batteries.1.voltage". Now I can call a function like

multiplexer.publish("electric.batteries.1.voltage" , theValue)

for each and every line. Then things are as you said.

But if the multiplexer exposes an API that allows me to look up a function to call for each data item when I process the header, like

firstColumnCallback = multiplexer.getCallback("electric.batteries.1.volt age")

then I can call that function for every row's first column, with no string comparisons. Then if the multiplexer is smart it won't do any string wrangling to dispatch the data to subscribers.

The equivalent can be done for NMEA 0183 and NMEA 2000 parsers, where the fields are (mostly) deterministic once you know the sentence/pgn.

The distinct benefit of using string keys is that new ones can be processed without prior knowledge. The multiplexer can pass any data, not just what was hardcoded into it.

Btw you can also eat and save the cake: you can have special shorthand methods for subscribing to well known data like position and speeds while keeping support for generic data handling and position and speed available over both APIs.

I agree on the indexing that can be done for the subscription. But not for the parsing process. The parser must digest the entire incoming spagetti...
I program microcontrollers at a low level, I think of the code that compilers generate with these high level sources, and it scares me. Surely my problem is due to that.

Originally Posted by teppokurki

So please squint your eyes and replace the JSON object with a C++ class properly typed and all, timestamps as proper timestamps (is there a C++ construct for millisecond accuracy timestamps?) etc etc.

Repeat after me: let's not use JSON strings for passing data around inside a single process. Even SK Node server doesn't do that, why should OpenCpn?

I would completely agree if the transducers spoke in signalK, and using ethernet as a physical medium.
Then my efforts would not take place. The question is what does SignalK bring to Ocpn? Right now I think nothing for its internal consumption. That is why I do not understand another utility other than establishing data links with "third parties". Recall here that users depend on GPS, sensors and pilots who do not speak SignalK, therefore the "third parties" are not them.

Originally Posted by teppokurki

This is the reason devices like NGT-1 and iKonvert exist - they enable both read and write access to an NMEA 2000 bus without forcing every single piece of connected software to be N2K certified. They take care of low level protocol details and for example if our application goes crazy and tries to flood the bus with messages they limit the transmit rate to observe N2K specification limits.

What they can not do is check that the data written makes sense. If we start sending bogus values for for example wind speed the interface will pass those happily along and other equipment will treat them as legitimate data, as long as the pgn format is correct.

We agree in the case of contemplating only transducers information as a client. And Luckily, for transducer data, at the present N2k manufacturers are accepting data from generic gateways.
But I wonder how these devices will act before PGN's directed and / or requested. Do these devices allow you to customize the PGN "product information" for several concurrent sources, for example?
I intend to illustrate with this the gap that exists to transfer the identification of the messages in SignalK and N2k, or even between NMEA0183 and N2k.
In N2k you must identify yourself on the network. From that moment, your ID number on the network will be linked to the "product information". The NGT-1 will surely appear as "Internetwork device: 25, Bridge: 150".

Originally Posted by teppokurki

This is done routinely. Think about the navigation PCs that racing boats have - they use the PC for multiple purposes, including interfacing with the N2K bus.

What speed problems? You mean that a value changes 10 times a second is a problem? People are using RPis to connect to very loaded N2K buses without any problems.

This is just fearmongering. AIS input is limited by the VHF bandwidth - even if you were in range with all the vessels on earth there is an upper limit to the data rate. AIS receivers typically have 38400 baud connections and the native AIS data encoding is 9.6 kbps. If the software is buggy and messes or misses data then it is a problem, but from a performance problem AIS is not a problem at all on any computer you would run OpenCpn on.

Please do not invent problems. Can you point me from the previous discussion a message from somebody else than you that says we should convert AIS from NMEA0183 to Signal K and back to NMEA0183? Even SK Node server explicitly keeps all NMEA0183 data available in original format without any conversions and in fact I use this feature on my own boat for AIS data access on my tablet.

You will surely be right. I am not referring only to my concern about the mountain of code that the processor is running, but also that sometimes the execution of a critical process can be cut due to a system task switch. And this happens even in the best families if an appropriate hierarchy is not designed. It will also be my problem because I think at a low level and in real time. : banghead:

José Luis

[Tehani]

Originally Posted by teppokurki

I don't understand what you mean. Use NMEA0183 (that is text) but in binary format - how? Address relative to what structure - instead of a preparsed double value a plugin would get an address "start parsing double here? Can you explain how my fancy pants gauge plugin would get updates for SOG and voltage for battery one - a single API or one for NMEA0183 and another for TwoCan data? What if I received the voltage not from N2K but from my Victron battery monitor via VE.direct and wanted to publish that to my gauge and all other dashboards and plugins, how do I do that?

I think I understand you. It is clear that the method of publication and subscription to a data is managed by each program module that parses it. In my case, I use a system of semaphores and task control of the OP. I think the concept is similar, even if we talk at two different levels. The problem arises when there is more than one subscription to the same data. (In my low level language: when more than one thread depends on a bit of one semaphore, which thread resets it?)
I am also realizing that programming with subscriptions approximates work to human language.
This is a whole theory of set applications in mathematics ...Thank you.

José Luis.

[motu42]

Interesting thread!.

The benefit of Signal K to opencpn is not especially valuable when you consider it in the way we currently use our chartplotters. But you miss an important aspect.

We all understand the way the internet has changed our lives - this forum is an example. Its changed our lives because its a valuable way to share and communicate between individuals, directly, instantaneously. Before the internet we had letters (slow, one-to-one) or we used the media, if we could get their attention.

But we continue to use our boat instrumentation as an isolated onboard system, usually heavily proprietary (any-one remember the mini-computers (DEC, Wang, etc). Its a lot like that.

We all carry cellphones, mostly with powerful internet capabilities, and we are mostly in cell phone range but we dont share boat data. Why does your mates boats not pop-up on your chartplotter? Why cant you share a waypoint, route or chart with them directly in your chartplotter? Why cant you see the wind/seastate on their vessel, in fact right across your sailing area?

Its because we cant share information. That requires a common language, an internet friendly common data format that includes vessel identifiers, timestamps, sources, and considers security and filtering. Thats what Signal K is about, but its become tangled with implementations (node server, rpi), and mired in 'product' issues.

The compelling use cases are around sharing data outside the vessel, and the existing languages (NMEA0183, NMEA2000, etc) just dont have the features to do this.

Some of these can be done now stand-alone services (crowd-sourced charting, depth, APRS, weather, etc) but they rarely integrate with your other gear or systems. Each has its own custom language

Signal K provides the common language for OpenCPN to become a social chartplotter, to integrate with data from other vessels, shore services, crowd-sourced cruising notes, and who knows what?

[motu42]

**Warning - deep tech stuff here, sorry **

Another lesson from the Artemis server:

The various delta and other json formats are only to allow transmission between systems, and not internally as Tehani and Teppo have discussed.

The ideal internal format is simple key:value,using the full signalk path. We actually need vessel:key:value:src:timestamp. No problem that can be done in efficient binary structures. (The key can be hashed into a form that allows binary masking, just like IP numbers)

What proved very useful in the Artemis server was exposing these as a stream over a tcp/ip port.

Consumers of the stream (basically events) receive only the keys they have an interest in, they perform their task, and they send the resulting keys back into the server via the tcp/ip socket. The common binary format is efficient, can be filtered for each connection, and for security.

These tcp/ip consumers are essentially plugins, but they are not limited to fixed API's, and compatible languages. The plugins can now be written in any language (so no reliance on contributors knowing C++, or javascript, etc) and TCP-IP provides queuing and reliable delivery without allowing a single plugin bug to crash the whole server.

Since each plugin can be a separate program you get the same effect as multi-threading without the problem of lock contention. Its also the same pattern as callbacks, but without the code lock in. Of course you can have many modules in one plugin program, or all modules in opencpn, but you have the flexibility.

Examples:

• db storage subscribes to all and just writes them out to the db
• alarms subscribe to a key and send an alarm when a value is exceeded
• opencpn listens for your lat/lon, and moves your vessel across the chart
• opencpn listens for your mates lat/lon, and moves your mates vessel across the chart
• opencpn listens for ais lat/lon, and moves an ais vessel across the chart
• sharing plugin listens for your lat/lon and sends it to your mates.

[bdbcat]

Motu42...


I think we are on the same wavelength.
But tell me more about:


What proved very useful in the Artemis server was exposing these as a stream over a tcp/ip port.



Do I read that you did NOT use SK/JSON as the "language" of the stream, but defined a proprietary binary package for inter-process/inter-machine communication instead? If so, this seems a step backwards from the SK idea...


Dave

[motu42]

I did not use JSON for this stream, but I did use an internal object format (binary). That format could easily be made generic, although my experiment at the time didnt do that. Keep in mind this is for internal use by the signalk server, not for distribution to other servers.


The reason was efficiency. You need to distribute a large number of key:values to a number of 'plugins' very quickly (1000's sec) and redistribute their output. Plus you need to apply a filter to each key:value sent so you dont flood plugins with data they do not want, and a security filter to each reply so the plugins cant compromise your vessel. Parsing json strings is not the way to do that efficiently.


So I still support signalk json (as strings) for general communication from one system to another, and for device to signalk consumer.



As mentioned already its not an efficient format internally so all current implementations use their own internal formats and API's for plugins. That means functionality is duplicated in every implementation, and plugins are essentially proprietary to servers again.


What the tcp/ip port does is open that out again to share plugins and functionality as well as data.

[Axel_G]

Originally Posted by Tehani

Thank you Teppo,

I agree on the indexing that can be done for the subscription. But not for the parsing process. The parser must digest the entire incoming spagetti...
I program microcontrollers at a low level, I think of the code that compilers generate with these high level sources, and it scares me. Surely my problem is due to that.

I've done some Signal-K programming based on an 8 Bitter (Atmel) and yes, that has real problems with this type of data, even the memory size becomes an issue when you need to store a couple of frames. But switching to a nowadays equally expensive 32 Bit ARM processor with its much higher memory size easily solves these problems. And in the end it's all C-Code.

Originally Posted by Tehani

I would completely agree if the transducers spoke in signalK, and using ethernet as a physical medium.
Then my efforts would not take place. The question is what does SignalK bring to Ocpn? Right now I think nothing for its internal consumption. That is why I do not understand another utility other than establishing data links with "third parties". Recall here that users depend on GPS, sensors and pilots who do not speak SignalK, therefore the "third parties" are not them.

For those it is no problem to continue with NMEA. But for all the undefined stuff (Battery logging etc.) signal-k is a viable solution.

Originally Posted by Tehani

You will surely be right. I am not referring only to my concern about the mountain of code that the processor is running, but also that sometimes the execution of a critical process can be cut due to a system task switch. And this happens even in the best families if an appropriate hierarchy is not designed. It will also be my problem because I think at a low level and in real time. : banghead:

José Luis

For a Raspberry Pi it is no issue at all to run this mountain of code. It is vastly less than the code it has to run for the HMI.

But it was a big mess trying to adapt dashbord to show my battery data as it would only handle NMEA.



In the end it is mainly about displaying the data and Signal-K has all the information required to do this. Using NMEA or any other kind of internal structure always requires that OCPN is prepared for this structure. If your boat has any kind of information the programmers have not anticipated there is no way to display it.



Axel

[rgleason]

@Motu This thread is to benefit the development of OpenCPN for its users, considering the best way(S) to handle and program for future communication (nmea0183, nmea2000 and signalK) . Deep tech posts are just fine if it helps understanding these issues.

There may be two (or multiple) approaches that fit within the development framework chosen, and still address the communication problem. Fast effective software (main app + plugins) to reduce cpu load and offloading programmed hardware devices can coexist. This thread is intended to inform the way forward.


So please, everyone. Keep an open mind. Listen and try to understand what is being said. Contribute and build on the suggestions made. There are numberous ways to skin the cat, and we are trying to find the most effective ways.

[rgleason]

I believe Tehani wrote me directly, recently about a similar idea to this for a data structure, table, whatever, that is used by plugins and also can be used by hardware at the (extended nmea0183) at the binary level..data that is needed and used in Opencpn (an OpenCPN set of data). Translator plugins then provide Nmea2000, 0183 and SignalK connections. (Jose Louis please feel free to correct me.)

Motu I like the sound of this:

The ideal internal format is simple key:value,using the full signalk path. We actually need vessel:key:value:src:timestamp. No problem that can be done in efficient binary structures. (The key can be hashed into a form that allows binary masking, just like IP numbers)

What proved very useful in the Artemis server was exposing these as a stream over a tcp/ip port.

Consumers of the stream (basically events) receive only the keys they have an interest in, they perform their task, and they send the resulting keys back into the server via the tcp/ip socket. The common binary format is efficient, can be filtered for each connection, and for security.

Would a hardware device as well as a plugin be able to participate in this? I think so.

[motu42]

OpenCPN is a great project, but it cant do everything for every-one, and its complexity is an achilles heel. (same for Signalk). To progress further we need to co-operate, working in our areas of strength. We need an environment where any-one can contribute their speciality easily.


Again sorry to be technical here but its needed to illuminate s solution. Lets look at how a combination of OpenCPN and Signalk systems might work:

A signalk server receives data from n2k, 0183, signalk, and the internet.
- messages converted to signalk formats (a core set, plus contributions)
- signalk messages are converted to key:value and streamed out on on a tcp/ip port (localhost)
- outputs messages in n2k, 0183, signalk are created and dispatched
- additional plugins provide additional data or data processing. (a core set, plus contributions)

OpenCPN provides the native GUI chartplotter.
- Its input is signalk key:value, it connects to signalk server via tcp/ip on (localhost)
- it has plugins that provide additional GUI related functionality
- its output is signalk key:value back to the signalk server

Signal K concentrates on transforming data to/from signalk, applying security, and routing the data where it needs to go.

The OpenCPN devs have a single simple key:value (binary) data format and no limitations shoe-horning stuff into 0183. They can concentrate on the best chartplotter etc

They (or any-one) can contribute functionality (eg polars) by building a polar data plugin to create the raw polar data from/to the tcp/ip stream. The data then becomes available in opencpn, and a gui plugin can display it.

[motu42]

Originally Posted by rgleason

Would a hardware device as well as a plugin be able to participate in this? I think so.

Yes, anything that can use tcp/ip.

At this point I think this port should only be available on the local network (eg a safe network). At first I thought that it should only be on localhost, but then I considered the use case where you need to spread the load on the signalk/opencpn server. You could run signalk on one host and opencpn on another, or maybe an AI analysis program that plotted ways to win the race etc.


Its tempting to use it everywhere, but Signal K development has shown there is a lot of hidden complexity in distributing data beyond the boat, best to move slowly.

[motu42]

Another important use case that is solved by using signalk is multiple chart plotters in one vessel.



Currently its possible to run one OpenCPN at the nav table, and one at the helm. But which one controls the autohelm? And if you set a route on one, does it appear on the other? Even the big name commercial vendors struggle to offer more than a screen sharing app.



Using Signalk to transport and route the data makes it possible. The details are a bit techy so I wont go there for now, but using Signal k for the core data transport and routing allows multiple devices to display a single co-ordinated view of the vessel.