Site collector

What it does

 PLANT NETWORK                      SITE DMZ                         INTERNET
 +----------------+  OPC UA 4840   +----------------------+  443   +-----------------------+
 | DCS / PLC /    | <------------- | kymoscope-collector  | -----> | Kymoscope             |
 | Kepware        |   (the         |  - reads, shapes     | (only  |  - relay: forwards    |
 +----------------+   collector    |  - encrypts          | out-   |    ciphertext         |
 +----------------+   connects     |  - disk buffer       | bound) |  - holds no key, cannot 
 | Logger files   | -- to it) ---> |    (ciphertext)      |        |    read the values    |
 +----------------+                +----------------------+        +-----------------------+
                                                                              |
                                                          your browser decrypts and charts

• It is an OPC UA client. It opens the connection to your server; nothing connects to it.
• Every value is stored at the rate the server offers it: nothing is resampled, averaged or deadbanded by the collector. A tag that reports only on change is stored when it changes.
• A value with a Bad status is stored as a gap with its status, never as a number. Booleans and enumerations become numbers (the enumeration labels are not sent). Text, arrays and structures are not channels and are dropped; the health stream counts them in shaper_dropped.
• Data is encrypted on the collector with your project key before it leaves. The relay holds no key and cannot read it.
• It publishes one stream per source, and optionally one more with its own health.

Requirements

• Host: Windows Server 2019 or later, a Linux server, or Docker. Python 3.10 to 3.13 for a pip install, from python.org or your distribution (not the Microsoft Store).
• To the control system: the OPC UA port of your server or gateway (4840 or the vendor's port), from the collector to the server.
• To Kymoscope: outbound TCP 443 (HTTPS and WebSocket), directly or through an HTTP proxy (see Proxy). No inbound ports.
• Disk: room for the buffer (up to 10 GB per stream by default) plus the 1 GB free-disk floor.
• Clock: synchronised (NTP). Data keeps its source timestamps, and the health stream reports the offset to the OPC UA server.

No other host is contacted. Package downloads (PyPI, the Docker registry) are needed only while installing.

Before installing

A checklist for the plant's IT team, before anything is installed on a Windows host:

• Application control (WDAC, AppLocker, Smart App Control): the Windows service runs in pythonservice.exe, the service host from pywin32, which is not code-signed (python.exe from python.org is). If your policy blocks unsigned binaries, allow that file by path or hash, run the collector in Docker or WSL instead, or wait for the signed installer.
• Network: outbound TCP 443 to the two hosts under Requirements, or an HTTP proxy that allows them (see Proxy). Nothing inbound.
• Python: 3.10, 3.11, 3.12 or 3.13 from python.org, installed for all users if the service will run under a dedicated account. The Microsoft Store Python is not supported: service install refuses it.
• Account: LocalSystem (the default) or a dedicated local account; the Windows service steps below list the rights it needs.

Install

Windows service (pip)

1. Install: pip install "kymoscope-collector[windows]". To read Parquet, TDMS or MF4 files add the files extra: "kymoscope-collector[windows,files]".
2. Create a folder (for example C:\ProgramData\Kymoscope) with collector.yaml and tags.csv (see below).
3. Check the configuration in a console first: kymoscope-collector run --config C:\ProgramData\Kymoscope\collector.yaml, then stop it with Ctrl+C.
4. Install and start the service from an administrator console.

kymoscope-collector service install --config C:\ProgramData\Kymoscope\collector.yaml
kymoscope-collector service start --config C:\ProgramData\Kymoscope\collector.yaml
kymoscope-collector service stop --config C:\ProgramData\Kymoscope\collector.yaml
kymoscope-collector service remove --config C:\ProgramData\Kymoscope\collector.yaml

service takes --config with every action. If the configuration reads a secret with env:NAME, set that variable for the service account, or use file:key.txt so the secret stays in a file only that account can read.

The service runs as the LocalSystem account unless you change it. To run it under a dedicated account (recommended), create a local user such as kymoscope, then, from an administrator console:

sc config KymoscopeCollector obj= .\kymoscope password= <the password>
icacls C:\ProgramData\Kymoscope /grant kymoscope:(OI)(CI)M
icacls C:\ProgramData\Kymoscope\pki\own\key.pem /grant kymoscope:R

The account needs: read access to the Python installation (service install prints its folder) and to the virtual environment, if you use one, so a Python installed for one user only, under C:\Users, will not do; modify rights on state_dir (the buffers, certificates and logs are written there), plus read access to pki/own/key.pem if it already exists (it is private to the account that created it, for example yours after a browse --config, and does not inherit the folder's rights); read access to collector.yaml, tags.csv and any file: secret; modify rights on every watched folder and its done and failed folders (files are moved); and the "Log on as a service" right (Local Security Policy, User Rights Assignment: sc config does not grant it). The service starts automatically (delayed) and restarts itself after a failure, including a stop caused by an error.

Docker

The image is ghcr.io/javijur/kymoscope-collector. Its /data volume holds the configuration, the tag list and the state (certificates, buffers). It runs run --config /data/collector.yaml.

docker run -d --name kymoscope-collector --restart unless-stopped \
  -v /srv/kymoscope:/data \
  -e KYMOSCOPE_API_KEY=kymoscope_... \
  ghcr.io/javijur/kymoscope-collector

Put api_key: env:KYMOSCOPE_API_KEY and state_dir: /data/state in /srv/kymoscope/collector.yaml (/data/collector.yaml in the container) so state survives container upgrades.

The container runs as user id 10001, so the mounted folder must be writable by it. A folder you create with mkdir belongs to root; give it to that user first, or start the container with --user set to the folder's owner. If /data is not writable the container stops at once and says so.

sudo mkdir -p /srv/kymoscope && sudo chown -R 10001:10001 /srv/kymoscope

Linux (systemd)

1. Install: pip install kymoscope-collector (preferably into a virtual environment).
2. Create the service user: sudo useradd --system --no-create-home --shell /usr/sbin/nologin kymoscope.
3. Copy the unit file to /etc/systemd/system/. It ships inside the package, at <site-packages>/kymoscope_collector/packaging/kymoscope-collector.service (in a source checkout: packages/collector/packaging/). Edit the paths to match your host.
4. Put the configuration in /etc/kymoscope/collector.yaml with state_dir: /var/lib/kymoscope; the unit creates that folder for the service user.
5. Run systemctl daemon-reload, then systemctl enable --now kymoscope-collector.

The unit uses ProtectSystem=strict: everything except the state folder is read-only for the service. A files source writes to its watched folder (processed files move to done), so list every such folder in the commented ReadWritePaths= line of the unit and uncomment it, and make sure the kymoscope user can write to those folders.

Updating

The disk buffer, certificates and stream identities live in state_dir and survive an update, so data collected meanwhile is sent afterwards.

Windows service

Stop the service first: pip cannot replace files the running service has open, and fails with WinError 32 (the file is being used by another process). From an administrator console, with the same Python or virtual environment the service was installed from:

kymoscope-collector service stop --config C:\ProgramData\Kymoscope\collector.yaml
pip install -U "kymoscope-collector[windows]"
kymoscope-collector service start --config C:\ProgramData\Kymoscope\collector.yaml

Docker

docker pull ghcr.io/javijur/kymoscope-collector
docker rm -f kymoscope-collector
docker run -d --name kymoscope-collector ...   # the same options as before

Linux (systemd)

sudo systemctl stop kymoscope-collector
sudo /opt/kymoscope/venv/bin/pip install -U kymoscope-collector   # the environment the unit runs
sudo systemctl start kymoscope-collector

collector.yaml reference

A complete example first, then every key. The file is read when the collector starts, so restart after changing it (tags.csv is the exception). A mistake stops the start with one sentence naming the key; so does a key the collector does not know (a typo), which also lists the valid ones. A password combined with security: None is allowed but logged as a warning, because the credentials then travel unencrypted.

# collector.yaml. Relative paths resolve against this file's folder.
api_key: env:KYMOSCOPE_API_KEY
project: Plant A
state_dir: C:/ProgramData/Kymoscope
buffer:
  max_mb: 10240
  min_free_mb: 1024
health:
  stream: Plant A health
  every_s: 5
sources:
  - name: Plant A DCS
    type: opcua
    url: opc.tcp://dcs-gw:4840
    security: Basic256Sha256
    mode: SignAndEncrypt
    user: kymo-reader
    password: env:OPCUA_PASSWORD
    tags: tags.csv
  - name: Cycler drops
    type: files
    path: D:/cycler/export
    pattern: "*.csv"
  - name: Bench rig
    type: python
    module: random_walk.py
    class: RandomWalk
    options: {tags: 5, hz: 10}

Top level

OPC UA source (type: opcua)

File source (type: files)

Python source (type: python)

Proxy

Where the plant reaches the internet only through an HTTP proxy, name it in collector.yaml. The collector tunnels both the HTTPS calls to the Kymoscope service and the relay WebSocket through it (HTTP CONNECT), so the data stays end-to-end encrypted and the proxy sees only the host names.

proxy: env:KYMOSCOPE_PROXY        # e.g. http://kymo:p%40ss@proxy.plant.local:8080
no_proxy: [historian.plant.local, .corp.local]

• Credentials go in the URL. Percent-encode special characters (@ is %40, : is %3A, \ is %5C). Only Basic proxy authentication is supported; for NTLM or Kerberos, let the two hosts through without authentication.
• Keep the password out of the file with env: or file:. The log shows the proxy host and port, never the credentials.
• The setting applies to the whole collector process. OPC UA connections never use it.
• A Windows service does not see the proxy set in a user's Windows or browser settings, so set proxy even if a console run works without it. A proxy that inspects TLS must let the two hosts through untouched.

tags.csv

The tag list says which OPC UA variables a source collects. It is a plain CSV you can edit in a spreadsheet.

node_id,name,group,sampling_ms,unit,description
"nsu=urn:plant:dcs;s=FIC101.PV",FIC101,,,m3/h,Feed flow
"nsu=urn:plant:dcs;s=TIC201.PV",TIC201,,1000,degC,Reactor temperature

Only node_id is required. The file is checked every 10 seconds; when it changes the collector re-subscribes by itself. A change reconnects the source, so every tag of that source has a short gap while it does. A file with an error is ignored and the previous list stays in use (the log says why).

From a browse of the server

kymoscope-collector browse --url opc.tcp://dcs-gw:4840 --config collector.yaml --out tags.csv
# optional: --root "nsu=urn:plant:dcs;s=Unit1" to start below a node, --depth 10 (the default)

It walks the address space and writes every readable numeric, boolean or enumeration variable. Strings, arrays and variables without read access are skipped and counted in the last line it prints. Without --config it connects with no security. With one, it uses the OPC UA source for that URL (the first OPC UA source if none matches), with the same certificates. Then delete the rows you do not want.

From a PI or DCS tag export

Export the tag list to CSV and give it the header above: one row per tag, with the server's node id for that tag in node_id (written nsu=<namespace URI>;s=<identifier>) and the tag name, unit and description in the other columns. The columns may be in any order; columns with other names are ignored.

Certificates

With security on, both sides prove who they are with certificates. The collector creates its own on first use (valid ten years) in state_dir/pki/own and keeps the private key there.

1. Show the thumbprint (the SHA-1 of the certificate, the form an OPC UA server's trust list shows): kymoscope-collector cert --config collector.yaml. It prints the file path, the thumbprint and the days left.
2. On the OPC UA server, trust that certificate. The file to import is state_dir/pki/own/cert.der. Until this is done the server rejects the connection; most servers list the refused certificate so you can compare its thumbprint and trust it from there.
3. Start the collector. With trust_server: first_use it accepts the certificate the server presents the first time and remembers it for that URL.

Trusting the server

• first_use (default): the first certificate seen at a URL is stored in state_dir/pki/trusted. A different certificate at the same URL afterwards is refused. Credentials are sent only after the server's certificate has passed this check.
• pinned: only certificates already in state_dir/pki/trusted are accepted. Copy the server's certificate there as <sha1>.der (its SHA-1 thumbprint as file name) for strict change control.
• Refused certificates are saved in state_dir/pki/rejected/<sha1>.der, and the log names the file. After you have checked it with the server's owner, move it to pki/trusted. The collector retries every 60 seconds, so no restart is needed. When a server legitimately gets a new certificate, the same applies: the new one lands in rejected; check it and move it.

Renewal

Run kymoscope-collector cert --config collector.yaml --renew. The old certificate and key are kept in pki/own/old-YYYYMMDD-HHMMSS, a new one is created and its thumbprint is printed. The server must trust the new certificate before the collector can connect again, so do this in a maintenance window and restart the collector afterwards. The channel <source>.own_cert_days_left in the health stream shows the time left.

Security notes

• Outbound only. The collector listens on no port. Two conduits: control system to collector (the OPC UA port) and collector to Kymoscope (443).
• Strong by default. OPC UA uses SignAndEncrypt with Basic256Sha256. Older SHA-1 policies are not offered. Using no security takes two deliberate settings (security: None and allow_insecure: true) and logs a warning at every connection.
• Read-only. The collector only reads and subscribes; it never writes to the server. Give it a dedicated OPC UA user with read access only.
• Encrypted before it leaves. Values are encrypted on the collector with AES-256-GCM using a key derived from your API key secret. The relay and Kymoscope servers carry ciphertext only.
• The buffer on disk is ciphertext. What waits in state_dir/buffer during an outage is already encrypted, so a copied disk does not reveal plant data. The API key (with its secret part) is the one secret on the host: keep it in an environment variable or a file with restricted permissions, not in a shared collector.yaml.
• Secrets out of the YAML. api_key and password accept env: and file:, so the YAML can be reviewed and versioned without secrets.
• Private key. state_dir/pki/own/key.pem identifies the collector to the OPC UA server. The collector creates it readable only by its owner (mode 0600 on Linux; on Windows only the current user, SYSTEM and Administrators).

Store-and-forward and the disk floor

The collector sends each block live first. Only when a send fails does it write the block, already encrypted, to the stream's disk buffer in state_dir/buffer. If the network or Kymoscope is unavailable, acquisition continues and the buffer fills. When the connection returns, live data goes first and the backlog drains with the bandwidth that live data leaves spare; replayed data is marked historical. A collector restart keeps the buffer.

• Size limit: buffer.max_mb per stream (default 10 GB).
• Disk floor: buffer.min_free_mb (default 1 GB). When the disk has less free space, the collector stops adding to the buffer rather than fill the disk.
• In both cases new blocks are dropped while the condition lasts, and they are counted in <source>.buffer_dropped. Alarm on that channel and on disk_free_gb.

While a stream cannot be registered (no network, a wrong key, or a project that waits for approval), its blocks wait in a memory queue bounded by buffer.memory_mb (default 256); registration is retried every 5 seconds. When the queue is full the oldest blocks are dropped and counted in <source>.queue_dropped.

Health stream

With health.stream set, the collector publishes a row every health.every_s seconds as a normal stream that you chart and alert on like any other. Channels are named <source name>.<key>.

A Python source adds whatever its own health() method returns, under the same naming.

How many tags one source handles

One OPC UA source handles about 2,000 tags reporting at distinct millisecond times per second comfortably. Split a larger tag list into several sources: several type: opcua entries with different tags files, each with its own stream name. The collector keeps every request within the server's own operation limits (nodes per read, per browse, per subscription call).

Custom Python sources

For a bench, a serial device or a vendor SDK, write a small class and point a source at it. The collector adds shaping, encryption, buffering and health.

import asyncio
import random
import time

from kymoscope_collector import Adapter, TagInfo, Value


class RandomWalk(Adapter):
    def __init__(self, tags=5, hz=10):          # receives the source's "options"
        self.n, self.hz = int(tags), float(hz)

    async def discover(self):                   # the channels this source offers
        return [TagInfo(id=f"RW{i}", name=f"RW{i}") for i in range(1, self.n + 1)]

    async def start(self, emit):                # runs until the collector stops
        x = [0.0] * self.n
        while True:
            now = time.time()
            for i in range(self.n):
                x[i] += random.gauss(0, 1)
            emit([Value(f"RW{i + 1}", now, x[i]) for i in range(self.n)])
            await asyncio.sleep(1 / self.hz)

  - name: Bench rig
    type: python
    module: random_walk.py
    class: RandomWalk
    options: {tags: 5, hz: 10}

• discover() returns the channels as TagInfo(id, name, unit, description).
• start(emit) runs until cancelled and calls emit([Value(tag, t, v)]), with t in Unix seconds and tag equal to the TagInfo.id. An optional status is an OPC UA status code (0 is Good).
• health() optionally returns a mapping of numbers that appear in the health stream.
• If the class raises an error, the collector logs it and restarts the source after 5 seconds.

File watcher

A source of type: files watches a folder for files written by loggers, cyclers and DAQ software. A file is read once it has not been modified for min_age_s (default 10 seconds), and every row keeps its own timestamp.

pip install "kymoscope-collector[files]"

• By default every supported extension is picked up; set pattern (for example *.csv) to narrow it.
• Never twice: the collector keeps a ledger of processed files in its state folder, so a file is never sent twice, even if it cannot be moved out of the watched folder (read-only folder, file locked by another program).
• done: processed files move to done_dir (default: done inside the watched folder). A name that already exists there gets a numeric suffix. Files older than keep_done_days (default 30) are deleted from there.
• failed: a file that cannot be read moves to failed inside the watched folder, next to a .error.txt file that says why. Fix the cause and move the file back to the watched folder to retry. files_failed in the health stream counts them.
• The channels are the columns of the newest file present at start, so at least one sample file must be in the folder (or in done) when the collector starts. Columns that first appear later are picked up without a restart.
• Each file is read completely and sent as one batch, in time order, so a very large file needs memory for all of it.

Try it without a plant

The package includes a demo OPC UA server shaped like a plant, so you can test the whole chain.

kymoscope-collector simulate        # demo server on port 4840; options: --port, --tags 1000, --fast 50, --hz 100, --secure
kymoscope-collector browse --url opc.tcp://localhost:4840 --out tags.csv
kymoscope-collector run --config collector.yaml

With --secure the demo only offers signed or encrypted endpoints, like a real server. Without --config, browse connects with no security, so for a --secure demo pass your --config.

Troubleshooting