SignalK development ?

[verkerkbr]

Progress of SignalK goes rather slow. But now a real working set instruments are available in the admin application, called mxtommy/kip.

Comes at first in a demo mode. But can be installed with the real data of system.

Now there is practical application available.

See images. One combined with OpenCPN

[sbender]

Originally Posted by verkerkbr

Progress of SignalK goes rather slow. But now a real working set instruments are available in the admin application, called mxtommy/kip.

Comes at first in a demo mode. But can be installed with the real data of system.

Now there is practical application available.

See images. One combined with OpenCPN

I would definitely disagree that progress on Signal K is slow. There's a ton stuff going on. And there have been practical applications available for a long time.

- we have finalized and released version 1 of the Signal K specification
- we have multiple new hardware vendors with builtin Signal K support (most notably Victron Energy (WIP) and the Stainless Lobster Fridge Optimizer)
- huge usability updates have been made to the Signal K node server administration interface and installation
- there is now a Signal K Cloud. You can make your boat data available via the cloud and accessed anywhere for free.
- My iOS Signal K application WilhelmSK gets regular updates and now can show Navionics maps along side your Signal K data and your boat data from the cloud.

I'm sure I'm missing some things..

[verkerkbr]

Originally Posted by sbender

I would definitely disagree that progress on Signal K is slow. There's a ton stuff going on. And there have been practical applications available for a long time.

- we have finalized and released version 1 of the Signal K specification
- we have multiple new hardware vendors with builtin Signal K support (most notably Victron Energy (WIP) and the Stainless Lobster Fridge Optimizer)
- huge usability updates have been made to the Signal K node server administration interface and installation
- there is now a Signal K Cloud. You can make your boat data available via the cloud and accessed anywhere for free.
- My iOS Signal K application WilhelmSK gets regular updates and now can show Navionics maps along side your Signal K data and your boat data from the cloud.

I'm sure I'm missing some things..

Sure, SignalK works great and the new admin interface is a very good tool for installing applications. But what is missing are real applications for "Jo Sailorman" using Linux or Windows or doing the navigation on a Raspberry Pi.

He does not have a lobster fridge and probably no IOS for navigation.

MXTommy/kip is the first practical instrument he can use on his boat.

Certainly, SignalK has a great future. But good working instruments are needed.

And you need a simple instruction how to put a SignalK server on a system.

There is no SignalK to be found in the repositories of Linux Mint.

In OpenPlotter it is allready build in.

Regards,

Bram

[nohal]

Originally Posted by sbender

I would definitely disagree that progress on Signal K is slow. There's a ton stuff going on. And there have been practical applications available for a long time.

- we have finalized and released version 1 of the Signal K specification

Congratulations. It looks like it is time I get back to the SignalK integration work that has been on hold for more than a year because it was a constantly moving target I didn't have time to follow and fix.

- we have multiple new hardware vendors with builtin Signal K support (most notably Victron Energy (WIP) and the Stainless Lobster Fridge Optimizer)

That's nice and a huge success, but unfortunately of very little interest to general public as pointed out above. The people will probably want to know what changes for their BU-353, oldish SPX autopilot that somehow works receiving RMB,RMC, APB and XTE over NMEA0183 and that legacy wind instrument. The more progressive ones will ask "How do I connect my NMEA2000? Picture, please!"

- huge usability updates have been made to the Signal K node server administration interface and installation

Also a very nice progress of the last weeks, like the 1.0 version.

- there is now a Signal K Cloud. You can make your boat data available via the cloud and accessed anywhere for free.

Nice, although we currently have no use for that I'm sure some nice applications will come from it.

- My iOS Signal K application WilhelmSK gets regular updates and now can show Navionics maps along side your Signal K data and your boat data from the cloud.

I don't know, the price clearly tag said "This is not what you will use to start playing with SignalK"

I'm sure I'm missing some things..

I don't know if you really feel like missing something, but from a point of view of the guy that will get bombarded by these questions as soon as he announces the SK support in OpenCPN I definitely miss

• Answer to the question "I am running Windows, how do I install SignalK? All I'm able to do is download something and do a double click."
• Answer to the question "I am running macOS, how do I install SignalK? All I'm able to do is download something called DMG, do a double click and pull something to Applications. But only in case there is a nice enough picture in the window that appeared." Or alternatively "You total morons don't have a proper installer. Where is the PKG I download and double click?"
• Answer to the question "I see installation instructions only for Raspberry PI, how do I install it on my Ubuntu/Debian/Fedora/OpenSUSE/<fill in your favourite weird distribution here>?"

Maybe I will, besides this, find some less boring stuff I miss when I actually start looking at it again, but these are the questions I will have to answer the second I publish any SignalK integration in OpenCPN. Questions I don't feel like I should be answering.

Keep up with the excellent work you are doing, it is coming along really nicely

Pavel

[seandepagnier]

Pavel,

OpenCPN should implement its own signalk server in c++. It may be connected to other signalk servers, but not required, much like how nmea data is handled today.

[nohal]

Originally Posted by boat_alexandra

Pavel,

OpenCPN should implement its own signalk server in c++. It may be connected to other signalk servers, but not required, much like how nmea data is handled today.

Sean

Why? And why should this be obligatory?

Just to be clear, by SignalK server I understand something like signalk-server-node, not that trivial SK router that we of course will have internally.

Pavel

[seandepagnier]

Pavel,

You asked about dependencies and installers. signalk node server requires manually installing node as the version in apt can't be used. So it cannot be easily installed on any platform, raspbian included, without significant steps. Only openplotter has performed these steps.

If you take a look at the source code itself, it is not very complicated. It can be implemented in a few thousand lines in any language which is great and why I strongly support we adopt this data standard.

Implementing a signalk server using javascript is a nice hack, but it is not really something we should consider as a requirement for opencpn:

1) It requires a separate process.
2) There are too many dependencies.
3) and it's difficult for us to maintain as it uses a different language.
4) javascript is slow. In the openplotter forums, it is accepted that you need a faster device than the raspberry to make much use of node red despite the fact it is doing very trivial things.
5) websockets are nice, but we need normal sockets
6) There are license issues. The node server is implemented with an apache license which does not guarentee user freedom like the GPL.

It would be much better that we have a GPL signalk server in c++ specifically integrated to opencpn. It may connect to the node server if the user decides, like any other connection.

[nohal]

Originally Posted by boat_alexandra

Pavel,

You asked about dependencies and installers. signalk node server requires manually installing node as the version in apt can't be used. So it cannot be easily installed on any platform, raspbian included, without significant steps. Only openplotter has performed these steps.

If you take a look at the source code itself, it is not very complicated. It can be implemented in a few thousand lines in any language which is great and why I strongly support we adopt this data standard.

Implementing a signalk server using javascript is a nice hack, but it is not really something we should consider as a requirement for opencpn:

1) It requires a separate process.
2) There are too many dependencies.
3) and it's difficult for us to maintain as it uses a different language.
4) javascript is slow. In the openplotter forums, it is accepted that you need a faster device than the raspberry to make much use of node red despite the fact it is doing very trivial things.
5) websockets are nice, but we need normal sockets
6) There are license issues. The node server is implemented with an apache license which does not guarentee user freedom like the GPL.

It would be much better that we have a GPL signalk server in c++ specifically integrated to opencpn. It may connect to the node server if the user decides, like any other connection.

Your call. The few points I agree with still do not convince me we need to reinvent this specific wheel as I simply do not consider them practically important.

We of course will not make signalk-server-node, or whichever other package a technical dependency as we really do not care a slightest bit where the SK data come from and we will of course support SK data over any type of socket, but it still has absolutely nothing to do with reimplementing a full featured server (with administration, all the alien connectors etc. etc. etc. - simply the stuff the SK team invested hundreds, probably thousands, of man hours into and we would have to repeat) IMO.

What I was primarily talking about is the simple fact that nowadays a "normal person" with a computer has virtually no option to get the thing working, unless that computer is a RPi or the guy is a developer or other type of a hipster.

Pavel

[teppokurki]

Originally Posted by boat_alexandra

4) javascript is slow. In the openplotter forums, it is accepted that you need a faster device than the raspberry to make much use of node red despite the fact it is doing very trivial things.

And you are absolutely sure that it is the language, not the app?

Part of the reason for starting SK development in JavaScript was that it is cross platform. No plugin cross platform compilation issues for example.

Originally Posted by boat_alexandra

It would be much better that we have a GPL signalk server in c++ specifically integrated to opencpn. It may connect to the node server if the user decides, like any other connection.

Wouldn't it make more sense to tackle some practical problem, like systematic engine and electricity related data handling, with Signal K? NMEA0183 does not have good coverage there and just stuffing the data in XDRs will provide spot solutions but will not lead to anything systematic. Producing SK is easy from your own code and conversions from N2K exist. Core navigation data is easy enough to convert to 0183 so no immediate wins available there.

Imho making SK a first class citizen as a data format within OpenCPN, a natively handled input and interchange format, would yield most bang for the development effort in the mid to long term, if not right away.

[seandepagnier]

Thank you for your comments.

Originally Posted by teppokurki

And you are absolutely sure that it is the language, not the app?

It is not the language, but the way almost everyone uses it. Assuming we have fast connections, processors and plenty of ram. None of these are realistic assumptions.

Part of the reason for starting SK development in JavaScript was that it is cross platform. No plugin cross platform compilation issues for example.

The issue is, javascript is a web language and signalk is not a web application. There is no internet offshore and I do not run a web browser for anything while sailing.

When I am online, I disable javascript by default. The scripts do not add useful content, instead they make me run out of memory, and websites very slow to load.

Most websites use javascript for datamining and ads (like cruisers forum) this language is not typically used for common good.

I realize javascript can be used for applications that have nothing to do with a browser, and people can also live on boats that are on dry land.

I have been using my varient of signalk in pypilot (now included in openplotter), and everything is written in python which is in my opinion a much better language for this task compared to javascript.

Wouldn't it make more sense to tackle some practical problem, like systematic engine and electricity related data handling, with Signal K? NMEA0183 does not have good coverage

Without an engine I have zero interest in "engine" data, and my solar panel is wired directly to a flooded battery and I always have plenty of power. I don't need data for this, someone else can deal with it.

I have been working on autopilot data.

there and just stuffing the data in XDRs will provide spot solutions but will not lead to anything systematic. Producing SK is easy from your own code and conversions from N2K exist. Core navigation data is easy enough to convert to 0183 so no immediate wins available there.

I am of course in favor of the signalk format for the most part. I have some disagreements mostly as the format could easily be more concise and efficient with absolutely no tradeoffs.

If you need examples I can explain it, but the differences between pypilot signalk and node signalk are slight, it would be nice to unify things at some point. For now there is translation in openplotter.

in pypilot there is a signalk server and several clients working already doing a lot of things that are not easily done, or very awkward from javascript. WIth that said, I am not opposed to javascript, but I support python, and I do not think signalk should be fixed to any language in particular as it is only a data format.

python has better support for the types of things we want to do on boats already, like scientific libraries. The problems we want to solve on boats like machine learning and low level hardware communication are mostly done from python these days. javascript is built around web technology that we don't even use offshore.

Imho making SK a first class citizen as a data format within OpenCPN, a natively handled input and interchange format, would yield most bang for the development effort in the mid to long term, if not right away.

I agree with this completely. It would solve a lot of problems from within opencpn. Currently there are multiple messages for the same data in nmea0183, and every plugin is required to parse raw nmea messages, and have no way to access data not defined in nmea.

We need an improved data format to provide syncronization between multiple connected opencpn sessions for tracks, routes, and waypoints.

Using signalk as a core data format would simplify a lot of plugins and also fix several problems and improve efficiency all at the same time.

OpenCPN is written in c++ and that is unlikely to change anytime soon. For that reason, the signalk handling needs to be reimplemented in c++. There is no reasonable way to make use of anything written in javascript, or even python.

We cannot require users to run a separate signalk server outside of opencpn to make use of this functionallity although they may at their option do so to take advantage of those features and plugin in signalk-node-sever (nmea2k conversion data logging, node red etc.. ) or in pypilot (autopilot or imu) but this has no impact on opencpn.

[teppokurki]

Originally Posted by boat_alexandra

Without an engine I have zero interest in "engine" data, and my solar panel is wired directly to a flooded battery and I always have plenty of power. I don't need data for this, someone else can deal with it.

Oh. My comment was about SK and OpenCPN in general for all readers of this thread and OpenCPN developers, not directed at you.

Originally Posted by boat_alexandra

I do not think signalk should be fixed to any language in particular as it is only a data format.

Agreed! Just to be clear: I care about advancing our common interests of open source marine electronics, not a particular implementation or a programming language.

Originally Posted by boat_alexandra

We need an improved data format to provide syncronization between multiple connected opencpn sessions for tracks, routes, and waypoints.

Now this is interesting! I am working on code to provide a generic API to track data, including all the logged sensor data. The idea is to provide an API with metadata, time and geo (bounding box) querying abilities that would return
- metadata about the track such as start and end timestamps, human oriented name, origin, destination, description etc
- track data (timestamped positions) with sensor data (wind speed, boat speed etc) in selected time resolution as min/max/avg
Current implementation returns GeoJSON data with additional properties for the data.

Originally Posted by boat_alexandra

Using signalk as a core data format would simplify a lot of plugins and also fix several problems and improve efficiency all at the same time.

I can believe that. What would be the best next step in this direction?

Originally Posted by boat_alexandra

For that reason, the signalk handling needs to be reimplemented in c++.

Sure, this is what I meant when talking about SK as a first class OpenCPN citizen.

[250224]

Originally Posted by verkerkbr

SignalK looked very promising at first sight.

For me it is and was a completely wrong approach.

A web-y layer on top of a proprietary protocol? I cannot believe this will really change something.

What would be needed imho would be to reverse engineer NMEA 2000, kind of what did Samba with SMB.

[teppokurki]

Originally Posted by blu3534

A web-y layer on top of a proprietary protocol?

Merriam-Websters definition for the word proprietary talks about something that is used, produced, or marketed under exclusive legal right of the inventor or maker. None of that applies to Signal K.

Originally Posted by blu3534

What would be needed imho would be to reverse engineer NMEA 2000, kind of what did Samba with SMB.

For your information NMEA 2000 has been (mostly) reverse engineered for years.

[merrimac]

As author of CANBoat (reverse engineering NMEA 2000) which provides a JSON feed to the Signal K server, and the Navico radar plugin for O, I have experience with both sides here.

I think some internal aspects of how O does data passing, in particular between O and the plugins, could be done in such a way that eases the burden on the plugins. This would get rid of old and redundant code.

I think there are three things that could be done in OpenCPN:

1. Rewrite the internal data passing and storage to use a simple key/value structure.
2. Add a receiver for K to O, so that it can use a Signal K server.
3. A way to generate simple Signal K feed that can then be sent to one or more Signal K servers.

Feature #1 would simplify the internal interfaces. Some important data such as (GPS) location and speed are available using standard C++ method calls that a plugin can subscribe to. If the plugin needs more data it needs to add a NMEA0183 parser and ask for the NMEA0183 data. With some plugins like the dashboard this is for visualization, with others it is used for calculations. For instance, the radar plugin does this to access heading and magnetic variation.

My preference for a new internal API to pass data is to use a simple key/value metaphor, in order that the plugins (and O itself, when data is passed back up) do not need to do any parsing anymore -- whether it is the { } , of JSON or the $ , * of NMEA0183. It seems a natural fit to use the keys used by the Signal K data definitions.

The code in O itself needs a translation from NMEA0183 -> key/value and key/value -> NMEA0183 as we can't drop NMEA0183 externally. If the translators are data driven using some sort of data file, users with translation issues can, if needed, try to work around such issues by changing the content of the data file.

The API could either be super simple with a publish method that is called whenever a data point is generated, or more advanced with a subscribe mechanism where plugins and internal parts of O itself can declare their interest in particular data points. For instance, the radar plugin could ask O to give it heading and variation data. The radar plugin could use a third API call to publish ARPA targets that it generated.

I'm saying 'key/value' here but I don't necessarily mean that in a strict sense. What I do mean is that the datastructures in C++ should be fixed (as C++ doesn't handle dynamical typing very well) and abstract from the actual content. My mental image has something like a DataPoint class with a key, a value, a source, a timestamp. We should model these on Signal K obviously.

Once the internal API has been changed the ability to receive Signal K is easy to add. It would be nice for those users who have a Signal K server that want to use their data in O. OpenCPN doesn't need to write specific interfaces to NMEA 2000, fridge optimizers, Victron solar chargers etc. and whatever and whoever else is starting to adopt Signal K.

Feature 3 would be used to send out data generated in OpenCPN. That means data such as waypoints, tracks, current waypoint, ARPA targets. Once feature 1 is coded providing Signal K data in delta format is very easy to do.

[250224]

Originally Posted by teppokurki

Merriam-Websters definition for the word proprietary talks about something that is used, produced, or marketed under exclusive legal right of the inventor or maker. None of that applies to Signal K.

Thanks for your reply! - I meant, SignalK on top of NMEA 2000. As practically all devices are or will be NMEA 2000 you need a converter, e.g. iKommunicate, for data access. What good is it when you have a "modern 'web friendly' format which can be shared, processed and displayed on the latest web apps, mobile devices and cloud servers." when ~95 % of the relevant data comes from proprietary NMEA 2000?

For the "modern web friendly" top layer, SignalK is probably fine. But it won't change anything on the foundations, right? Do you forever want to forget about a more rigorous 'open boat approach'?

(For a more fundamental approach I suppose JSON schemas would need to evolve into something else. Don't want to complain about the specification (certainly a lot of hard work went into it and it is done well!), but still, it seems difficult to get an overview from e.g. https://github.com/SignalK/specifica...r/gitbook-docs, and https://github.com/SignalK/specifica...as/vessel.json really is not easy to read. Wouldn't protobuf etc. be able to represent the scheme more clearly?)

Originally Posted by teppokurki

For your information NMEA 2000 has been (mostly) reverse engineered for years.

You probably mean Canboat? Canboat is GPL3 and thus, unfortunately, forget it: no sane (commercial) device manufacture will ever touch such source. No developer that potentially is interested in a (liberal) NMEA 2000 protocol re-implementation attempt can look at Canboat. - I am not aware that Canboat did reverse engineer a meaningfull subset of PGNs. Is there a comprehensive database anywhere?