Node-RED Installer

Selecting Mods to install into Node-RED

The controls architecture is built around Node-RED and Docker, and the Node-RED Installer is the method used to set everything up.

The concept is as follows:

1. A startup flow is loaded into Node-RED and deployed.
2. This flow pulls the latest files from the GitHub repository, and creates a Docker container running Node-RED on port 5099, on which it installs a Node-RED Setup flow, with access to all credentials.
3. The setup flow provides a menu system to select the required Application (Node-RED flows) and Mods (additional Node-RED flows to be added to the application).
4. As items are selected, they are installed, along with any missing node types and credentials, onto the original Node-RED on port 1880.
5. Additional Node-RED containers, on ports 5001+, are started to implement isolated services such as data management, that may need separate access rights or updating.
6. The setup container is closed.

In essence, the Node-RED Installer gives Node-RED the ability to program itself, and create new instances of Node-RED or any other software.

This method has a number of advantages:

• Entirely handled by Node-RED. Other systems are used by Node-RED, but using the same platform to orchestrate everything makes things more user friendly.
• A standard simple Node-RED flow can be used as a starting point to install any system.
  https://github.com/heatweb/plumbing-controller/blob/main/flows/flows_install_installer.json
• Node-RED provides a powerful method to manage containers, with the ability to spin up temporary containers based on logic, running any services needed to boost the systems abilities.
• Credentials for both Node-RED elements and Docker containers can all be managed by the single setup container, that is killed once the system is up.
• Better updating control, with the ability to build (and rebuild) custom Node-RED flows from scratch, from a large selection of flow pages that each provide a specific function.
• Updating is performed live, without closing down existing flows. Variables are kept in memory during updates. The system can then be rebooted (if required) to clear out old variables in memory, but is generally not needed for minor updates or adding functions.
• Easier maintenance, with a set of maintained core flows (and starting points), expanded and customised as required.
• Manufacturers can develop their own flows adapted for different functions, place them into the GitHub repository, and make available to controllers at setup.
• Improved reliability, with services isolated in separate containers. Redundancy can be built into the architecture.
• Complicated setup procedures, including systems scripts, can be maintained in a Node-RED flow (stored on GitHub), deployed as needed in privileged containers, and removed.

Prerequisites:

• Node-RED on port 1880 (with Dashboard nodes installed)
• Docker

Credentials

Credentials are generated during installation of Docker containers., and are stored in /boot/heatweb/credentials folder as individual txt files (name is the cred id, content is the cred).

The following are standard credentials.

• adminPassword
• remoteAdminCommand
• localInfluxToken
• remoteInfluxServer
• remoteInfluxBucket
• remoteInfluxToken
• localMqttPassword
• emailServer
• emailPort
• emailUser
• emailPassword

Composer

It is possible to build up a Node-RED installation from the various modules using the Installer.

The Composer allows one to pre-define the list of modules to install, the order to install in, and targets.

A Composer file is a JSON file that contains an array of instructions.

A Composer file allows installations to be rebuilt using the latest module versions and as such is an update function.

• Deploy complete flow
• Deploy individual flow - if duplicates then duplicate/overwrite/keep/fail
• Apply changes before deploying
• Remove a flow
• Remove a node
• Update source from Git
• Copy file
• Set Boot Config
• Set Boot Credential
• Backup all flows
• Backup all flows, config & credentials
• Run system command

Composer JSON

The following is an example application, installing the flows_beta_plumbing_controller_1880.json flow, followed by three mods, with a change of serial port on the last mod. The flow_dhw_dashboard.json mod is linked to a specific version in the GitHub history, to prevent it from updating.

[
    {
        "dataSource": "https://raw.githubusercontent.com/heatweb/plumbing-controller/main/flows/flows_beta_plumbing_controller_1880.json",
        "targetHost": "localhost",
        "targetPort": 1880,
        "action": "flows"
    },
    {
        "dataSource": "https://raw.githubusercontent.com/heatweb/plumbing-controller/main/flows/mods/flow_dhw_control.json",
        "targetHost": "localhost",
        "targetPort": 1880,
        "action": "flow"
    },
    {
        "dataSource": "https://raw.githubusercontent.com/heatweb/plumbing-controller/4b6cd4a366e082fe15963598baf85b3893d58d19/flows/mods/flow_dhw_dashboard.json",
        "targetHost": "localhost",
        "targetPort": 1880,
        "action": "flow"
    },
    {
        "dataSource": "https://raw.githubusercontent.com/heatweb/plumbing-controller/main/flows/mods/flow_modbus_novocon.json",
        "targetHost": "localhost",
        "targetPort": 1880,
        "action": "flow",
        "replacements": [
            [
                "ttyS0",
                "ttyAMA4"
            ]
        ]
    }
]

Node-RED

[{"id":"d45724a5b9c70467","type":"tab","label":"Composer","disabled":false,"info":"","env":[]},{"id":"5c0a883d29582acd","type":"inject","z":"d45724a5b9c70467","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"[{\"dataSource\":\"https://raw.githubusercontent.com/heatweb/plumbing-controller/main/flows/flows_beta_plumbing_controller_1880.json\",\"targetHost\":\"localhost\",\"targetPort\":1880,\"action\":\"flows\"},{\"dataSource\":\"https://raw.githubusercontent.com/heatweb/plumbing-controller/main/flows/mods/flow_dhw_control.json\",\"targetHost\":\"localhost\",\"targetPort\":1880,\"action\":\"flow\"},{\"dataSource\":\"https://raw.githubusercontent.com/heatweb/plumbing-controller/4b6cd4a366e082fe15963598baf85b3893d58d19/flows/mods/flow_dhw_dashboard.json\",\"targetHost\":\"localhost\",\"targetPort\":1880,\"action\":\"flow\"},{\"dataSource\":\"https://raw.githubusercontent.com/heatweb/plumbing-controller/main/flows/mods/flow_modbus_novocon.json\",\"targetHost\":\"localhost\",\"targetPort\":1880,\"action\":\"flow\",\"replacements\":[[\"ttyS0\",\"ttyAMA4\"]]}]","payloadType":"json","x":190,"y":480,"wires":[["a66678b2ea23686e"]]},{"id":"a66678b2ea23686e","type":"change","z":"d45724a5b9c70467","name":"","rules":[{"t":"set","p":"composer.default","pt":"global","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":500,"y":480,"wires":[[]]},{"id":"c333189a999e8742","type":"inject","z":"d45724a5b9c70467","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"default","payloadType":"str","x":190,"y":600,"wires":[["72309de8a79eeadc"]]},{"id":"72309de8a79eeadc","type":"function","z":"d45724a5b9c70467","name":"Composer","func":"\nvar composition = global.get(\"composer.\" + msg.payload);\n\nif (!composition) { return null; }\n\nflow.set(\"composition\",composition);\nflow.set(\"fetched\", 0);\n\nfor (var item in composition) {\n\n    var msg1={};\n    msg1.url = composition[item].dataSource;\n    msg1.index = item;\n    node.send([msg1,null]);\n\n}\n\nreturn null;","outputs":2,"noerr":0,"initialize":"","finalize":"","libs":[],"x":390,"y":600,"wires":[["986cee236e456e68"],[]]},{"id":"5daabdcf26ba0523","type":"function","z":"d45724a5b9c70467","name":"flows","func":"msg.headers={};\nmsg.headers.Authorization = \"Bearer \" + flow.get(\"auth.access_token\");\nmsg.headers['Content-type'] = \"application/json\";\nmsg.headers['Node-RED-Deployment-Type'] = \"full\";\n\n\n\nvar targetPort = msg.payload.targetPort || 1880;\nvar targetHost = msg.payload.targetHost || \"localhost\";\n\ntargetHost = targetHost.replace(\"localhost\",\"host.docker.internal\")\n\n\nmsg.url = \"http://\" + targetHost + \":\" + targetPort + \"/\" + (msg.payload.action||\"flow\");\n\n//var cfrom = '\"broker\":\"localhost\"';\n//var cto = '\"broker\":\"localhost\", \"credentials\":{\"username\":\"nodereduser\", \"password\":\"' + global.get(\"noderedmqttpass\") + '\"}';\n\nvar credentials = global.get(\"credentials\")||{};\nvar config = global.get(\"config\") || {};\n\n//var ff = msg.payload;\nvar ff = msg.payload.data;\n\nif (msg.payload.replacements) {\n\n   var ffstr = JSON.stringify(ff);\n\n   var reps = msg.payload.replacements;\n   for (var rep in reps) {\n\n      ffstr = ffstr.replaceAll(reps[rep][0], reps[rep][1]);\n   }\n\n   ff = JSON.parse(ffstr);\n}\n\n\nfunction checkTab(nodeitem) {\n   return nodeitem.type == \"tab\";\n}\n\nfunction filterTab(nodeitem) {\n   return nodeitem.type != \"tab\";\n}\n\nfunction filterMqttBroker(nodeitem) {\n   return nodeitem.type != \"mqtt-broker\";\n}\n\nif (msg.payload.action == \"flow\") {   // individual flows can't contain tabs, and do not want to overwrite mqtt broker\n\n   var tabf = ff.filter(checkTab)[0];\n   tabf.nodes = ff.filter(filterTab).filter(filterMqttBroker);\n   ff = tabf;\n\n}\n\n\nfor (var part in ff) {\n\n   if (ff[part].type == \"heatwebConfig\") { \n      \n      ff[part].name = config.name ; \n      ff[part].description = config.description;\n      ff[part].nodeId = config.nodeId;\n      ff[part].networkId = config.networkId; \n\n   }\n   \n\n   if (credentials.localMqttPassword) {\n\n      if (ff[part].type == \"mqtt-broker\" && (ff[part].broker == \"mqtt\" || ff[part].broker == \"localhost\")) { ff[part].credentials = { \"user\": \"admin\", \"password\": credentials.localMqttPassword }; }\n    \n   }\n\n   if (credentials.adminPassword) {\n\n      if (ff[part].type == \"MySQLdatabase\" && ff[part].host == \"mysql\") { ff[part].credentials = { \"user\": \"root\", \"password\": credentials.adminPassword }; }\n\n   }\n\n   if (credentials.localInfluxToken) {\n\n      if (ff[part].type == \"influxdb\" && ff[part].name == \"local\") { ff[part].credentials = { \"token\": credentials.localInfluxToken }; }\n\n   }\n   if (credentials.remoteInfluxToken) {\n\n      if (ff[part].type == \"influxdb\" && ff[part].name == \"heatweb\") { ff[part].credentials = { \"token\": credentials.remoteInfluxToken }; }\n\n   }\n\n   if (credentials.emailPassword && credentials.emailUser) {\n\n      if (ff[part].type == \"e-mail\") { ff[part].credentials = { \"userid\": credentials.emailUser, \"password\": credentials.emailPassword }; }\n\n   }\n\n   \n\n\n}\n\nmsg.payload = JSON.stringify(ff);\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":390,"y":760,"wires":[["f4bdd8a482d1cebf"]]},{"id":"f4bdd8a482d1cebf","type":"http request","z":"d45724a5b9c70467","name":"","method":"POST","ret":"txt","url":"","tls":"","x":530,"y":760,"wires":[["df5cf78bb5e02533","7af6011b014b1971","24d72b1de9de0132"]]},{"id":"24d72b1de9de0132","type":"function","z":"d45724a5b9c70467","name":"save results","func":"var composition = flow.get(\"composition\");\n\nif (!composition) { return null; }\n\n\ncomposition[msg.index].deployed = msg.payload;\n\n\n\n\n// if (msg.filename) {\n\n//     var fn = msg.filename.replace(/\\//g, \"_\").replace(/\\./g, \"_\");\n//     global.set(\"results.\"+fn, JSON.parse(msg.payload));\n\n// }\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":770,"y":760,"wires":[[]]},{"id":"986cee236e456e68","type":"http request","z":"d45724a5b9c70467","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":570,"y":600,"wires":[["14b1e8754eecdaaf"]]},{"id":"14b1e8754eecdaaf","type":"function","z":"d45724a5b9c70467","name":"compile","func":"\nvar composition = flow.get(\"composition\");\n\nif (!composition) { return null; }\n\ntry {\n\n    composition[msg.index].data = JSON.parse(msg.payload);\n\n} \ncatch (err) {\n\n    if (msg.payload[0]) { composition[msg.index].data = msg.payload; }\n\n}\n\nflow.set(\"composition\", composition);\n\nvar fetched = flow.get(\"fetched\");\nfetched++;\nflow.set(\"fetched\", fetched);\n\nif (fetched < composition.length) { return null; }\n\nflow.set(\"posted\", 0);\n\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":760,"y":600,"wires":[["df5cf78bb5e02533","3f74b3674bd337f4"]]},{"id":"df5cf78bb5e02533","type":"function","z":"d45724a5b9c70467","name":"shift","func":"var posted = flow.get(\"posted\") || 0;\nvar composition = flow.get(\"composition\");\n\n//if (posted = composition.length) { return null; }\n\nvar msg1 = {};\nmsg1.payload = composition[posted]; //.data;\nmsg1.index = 1*posted;\n\nposted++;\nflow.set(\"posted\", posted);\n\n\nif (!msg1.payload) { return null; }\n\nreturn msg1;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":470,"y":680,"wires":[["5daabdcf26ba0523","9fcd3db98350446e"]]},{"id":"9fcd3db98350446e","type":"debug","z":"d45724a5b9c70467","name":"debug 30","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":660,"y":680,"wires":[]},{"id":"3f74b3674bd337f4","type":"debug","z":"d45724a5b9c70467","name":"debug 31","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":940,"y":600,"wires":[]},{"id":"7af6011b014b1971","type":"debug","z":"d45724a5b9c70467","name":"debug 32","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":720,"y":820,"wires":[]},{"id":"18fbc27a546d106e","type":"function","z":"d45724a5b9c70467","name":"","func":"//var cmd = \"curl http://localhost:5002/auth/token --data 'client_id=node-red-admin&grant_type=password&scope=*&username=admin&password=Duu1cahe'\";\n\nmsg.payload={};\nmsg.payload['client_id']=\"node-red-admin\";\nmsg.payload['grant_type']=\"password\";\nmsg.payload['scope']=\"*\";\nmsg.payload['username']=\"admin\";\nmsg.payload['password']=\"admin\";\n\n\nmsg.url = \"http://\" + flow.get(\"targethost\") + \":\" + flow.get(\"targetport\") + \"/auth/token\";\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":240,"y":1040,"wires":[["916f81d06dcecf50"]]},{"id":"916f81d06dcecf50","type":"http request","z":"d45724a5b9c70467","name":"","method":"POST","ret":"txt","url":"","tls":"","x":400,"y":1040,"wires":[["ca67a6abcd19c97b","f6720a2a1363cb64"]]},{"id":"953dcb2e53d3b510","type":"change","z":"d45724a5b9c70467","name":"","rules":[{"t":"set","p":"auth","pt":"flow","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":920,"y":1040,"wires":[[]]},{"id":"46a3af1257eda101","type":"json","z":"d45724a5b9c70467","name":"","pretty":false,"x":740,"y":1040,"wires":[["953dcb2e53d3b510"]]},{"id":"ca67a6abcd19c97b","type":"debug","z":"d45724a5b9c70467","name":"debug 33","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":570,"y":980,"wires":[]},{"id":"f6720a2a1363cb64","type":"switch","z":"d45724a5b9c70467","name":"","property":"payload","propertyType":"msg","rules":[{"t":"cont","v":"access_token","vt":"str"},{"t":"else"}],"checkall":"false","repair":false,"outputs":2,"x":580,"y":1040,"wires":[["46a3af1257eda101"],[]]},{"id":"097f147bfa40c445","type":"comment","z":"d45724a5b9c70467","name":"need to pull in the authentication","info":"","x":250,"y":960,"wires":[]},{"id":"01e6f04422e315bb","type":"inject","z":"d45724a5b9c70467","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":210,"y":100,"wires":[["5da31f3d874fcd4c"]]},{"id":"5da31f3d874fcd4c","type":"file in","z":"d45724a5b9c70467","name":"","filename":"/boot/heatweb/config.json","filenameType":"str","format":"utf8","chunk":false,"sendError":false,"encoding":"none","allProps":false,"x":430,"y":100,"wires":[["03689db0a1229d7d"]]},{"id":"60014b30ddf0f588","type":"change","z":"d45724a5b9c70467","name":"","rules":[{"t":"set","p":"config","pt":"global","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":100,"wires":[[]]},{"id":"03689db0a1229d7d","type":"json","z":"d45724a5b9c70467","name":"","property":"payload","action":"obj","pretty":false,"x":630,"y":100,"wires":[["60014b30ddf0f588"]]},{"id":"307de46a7f2b649a","type":"inject","z":"d45724a5b9c70467","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":210,"y":180,"wires":[["c38ffe5bfd0dd27d","1d3bfec916ba0477"]]},{"id":"c38ffe5bfd0dd27d","type":"file in","z":"d45724a5b9c70467","name":"","filename":"/boot/heatweb/credentials.json","filenameType":"str","format":"utf8","chunk":false,"sendError":false,"encoding":"none","allProps":false,"x":490,"y":180,"wires":[["ad93699b4f67b45e"]]},{"id":"fe3ce37e31818309","type":"change","z":"d45724a5b9c70467","name":"","rules":[{"t":"set","p":"credentials","pt":"global","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":900,"y":180,"wires":[[]]},{"id":"ad93699b4f67b45e","type":"json","z":"d45724a5b9c70467","name":"","property":"payload","action":"obj","pretty":false,"x":710,"y":180,"wires":[["fe3ce37e31818309"]]},{"id":"68123565cd072358","type":"fs-file-lister","z":"d45724a5b9c70467","name":"","start":"/boot/heatweb/credentials","pattern":"*.txt","folders":"*","hidden":true,"lstype":"files","path":true,"single":false,"depth":"1","stat":false,"showWarnings":true,"x":460,"y":300,"wires":[["7d19b9387d19b985","51f46724bf3c8081"]]},{"id":"7d19b9387d19b985","type":"debug","z":"d45724a5b9c70467","name":"debug 34","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":700,"y":280,"wires":[]},{"id":"1d3bfec916ba0477","type":"delay","z":"d45724a5b9c70467","name":"","pauseType":"delay","timeout":"1","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":280,"y":300,"wires":[["68123565cd072358"]]},{"id":"f651936739ae9faf","type":"file in","z":"d45724a5b9c70467","name":"","filename":"filename","filenameType":"msg","format":"utf8","chunk":false,"sendError":false,"encoding":"none","allProps":false,"x":820,"y":340,"wires":[["d3a616ac9f624f5f"]]},{"id":"51f46724bf3c8081","type":"change","z":"d45724a5b9c70467","name":"","rules":[{"t":"set","p":"filename","pt":"msg","to":"payload","tot":"msg","dc":true}],"action":"","property":"","from":"","to":"","reg":false,"x":650,"y":340,"wires":[["f651936739ae9faf"]]},{"id":"d3a616ac9f624f5f","type":"function","z":"d45724a5b9c70467","name":"store creds","func":"\nvar fn = msg.filename.substr(msg.filename.lastIndexOf(\"/\")+1);\nfn = fn.substr(0,fn.lastIndexOf(\".\"));\n\nglobal.set(\"credentials.\" + fn, msg.payload.trim());\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":990,"y":340,"wires":[[]]}]