Live streams

Collector: serial and SCPI

Read balances, gas analysers and meters over RS-232, RS-485 or TCP, and SCPI/VISA instruments, with a parsing rule per line.

Serial and SCPI instruments

Two source types for lab and legacy instruments. serial reads lines of text from anything on an RS-232 or RS-485 port, a USB serial adapter, or a TCP port (serial servers such as Moxa NPort in raw mode): balances, gas analysers, pH and conductivity meters, furnace controllers. scpi asks SCPI instruments by VISA resource name: multimeters, power supplies, scopes and spectrum analysers over LAN, GPIB, USB or serial. Both are read only by default and are polled or streamed at instrument speed (a serial line is rarely more than a few hundred values per second).

serial: example, a balance that streams

collector.yaml

sources:
  - name: Packing scale
    type: serial
    port: COM3                 # /dev/ttyUSB0 on Linux
    baudrate: 9600
    terminator: '\r\n'         # single quotes: the backslashes are kept for the collector
    # the scale prints:  ST,GS,   12.345 kg
    regex: '(?P<state>ST|US),GS, *(?P<weight>[-+]?\d+\.\d+) *kg'
    columns:
      - { name: Weight, group: weight, unit: kg }

serial: example, a gas analyser you ask, over a serial server

collector.yaml

sources:
  - name: Flue gas analyser
    type: serial
    transport: tcp             # a Moxa/Lantronix serial server in raw TCP mode
    host: 10.20.0.60
    tcp_port: 4001
    mode: poll
    command: 'READ?'           # sent with the terminator, every poll_ms
    poll_ms: 2000
    # the analyser answers:  14:05:33;O2=20.9;CO2=412.5;CO=3.1
    delimiter: ';'
    timestamp: { field: 0, format: '%H:%M:%S' }
    columns:
      - { name: O2,  index: 1 }
      - { name: CO2, index: 2, unit: ppm }
      - { name: CO,  index: 3, unit: ppm }

the same reply with a regex

# the fields above are "O2=20.9": use a regex when the fields carry labels
    regex: '(?P<ts>[\d:]+);O2=(?P<o2>[\d.]+);CO2=(?P<co2>[\d.]+);CO=(?P<co>[\d.]+)'
    timestamp: { field: ts, format: '%H:%M:%S' }   # no columns: every other group is a channel

Name	Type	Description
`transport`	`serial \| tcp` Default: `serial`	A local port, or a raw TCP socket to a serial server.
`port`	`string`	serial: COM3, /dev/ttyUSB0, or any pyserial URL such as rfc2217://host:2217. Required for transport serial.
`baudrate, bytesize, parity, stopbits`	`number, 5-8, N/E/O/M/S, 1/1.5/2` Default: `9600, 8, N, 1`	Line settings; they must match the instrument.
`host, tcp_port`	`string, number`	Required for transport tcp.
`timeout_s`	`number` Default: `2.0`	Read timeout. A call that takes three times this is abandoned and the port reopened; while 3 abandoned calls are still stuck (threads_hung) the source waits, up to 10 minutes, before reopening.
`framing`	`line \| fixed` Default: `line`	line: messages end at terminator. fixed: every message is length bytes.
`terminator`	`string` Default: `\n`	Written as in the example: \r\n, \n, \r, \x03. Also appended to the command.
`length`	`number`	Bytes per message for framing fixed.
`encoding`	`string` Default: `ascii`	Text encoding of the lines; use latin-1 for degree signs and similar.
`mode`	`stream \| poll` Default: `stream`	stream: the instrument sends by itself. poll: send command every poll_ms and read one reply.
`command, poll_ms`	`string, number` Default: `-, 1000`	For mode poll. Input received before the command is discarded.
`regex`	`string`	A Python regular expression with named groups (?P<name>...), searched in each line.
`delimiter`	`string`	Split each line into fields at this text; whitespace splits on runs of blanks. Set exactly one of regex and delimiter.
`columns`	`list`	The channels: name, then group (regex; defaults to the name) or index (delimiter; 0 is the first, -1 the last), unit, scale, offset (value = raw x scale + offset). With a regex, columns may be left out: every named group is a channel.
`timestamp`	`{field, format}`	Take the time from a field: format epoch, epoch_ms, or a strptime pattern such as %Y-%m-%d %H:%M:%S. A pattern with no date takes the day, in this host’s time zone, that puts the stamp closest to the arrival time (23:59:59 received just after midnight is yesterday). Without it the sample is stamped on arrival.

A line that does not parse (no regex match, too few fields, a value that is not a number) is dropped, counted in parse_errors and the first few are written to the log. In stream mode, silence for timeout_s is counted in timeouts; in poll mode a missing reply is a failed poll and the next one waits longer, up to a minute. A lost port or connection is reopened with a growing delay.

scpi: example, a bench multimeter

collector.yaml

sources:
  - name: Bench DMM
    type: scpi
    resource: TCPIP0::192.168.1.10::INSTR   # GPIB0::12::INSTR, ASRL3::INSTR, USB0::0x2A8D::0x0101::MY123::INSTR
    timeout_s: 3
    queries:
      - { name: Voltage, query: 'MEAS:VOLT:DC?', unit: V, poll_ms: 500 }
      - { name: Current, query: 'MEAS:CURR:DC?', unit: A, scale: 1000, poll_ms: 500 }   # scale 1000: reads mA
      - name: Supply
        query: 'MEAS:ALL?'                  # answers "5.01,0.203"
        parse: csv
        names: [Supply V, Supply I]

Name	Type	Description
`resource`required	`string`	The VISA resource name: TCPIP0::host::INSTR, TCPIP0::host::5025::SOCKET, GPIB0::12::INSTR, ASRL3::INSTR, USB0::...::INSTR.
`backend`	`string` Default: `@py`	The pure-Python pyvisa-py backend needs no VISA install (GPIB and some USB instruments need the vendor’s driver: use @ivi with NI-VISA or Keysight VISA installed).
`queries`required	`list`	name, query (must end in ?), unit, scale, offset, poll_ms (default: the source poll_ms), parse: float (default) or csv with names: one channel per comma-separated field, an empty name skips a field.
`poll_ms`	`number` Default: `1000`	Default interval; queries with the same interval are asked together, in order.
`timeout_s`	`number` Default: `5.0`	Per query. A query the instrument does not answer is stored as a gap.
`read_termination, write_termination`	`string`	Set when the instrument needs it, for example \n for serial and raw-socket resources.
`idn`	`true \| false` Default: `true`	Read *IDN? at connect. It is logged and shown as the description of the channels in discover.
`init`	`list` Default: `[]`	Commands sent at each connect. Anything except *IDN? writes to the instrument and is refused unless allow_writes is true.
`allow_writes`	`true \| false` Default: `false`	Allow init commands such as CLS or CONF:VOLT:DC 10, and queries that make the instrument test or calibrate itself (TST?, *CAL?, CAL...?, TEST...?, DIAG...?, ...:SELFtest?). The collector never writes anything else. Queries and init commands may never contain a newline or other control character.

A reply of 9.9E37 or more (the SCPI overload value) is stored as a gap. A reply that is not a number is a gap and counted in parse_errors. If no query of a poll is answered, or the instrument fails in any other way, the connection is reopened.

discover

kymoscope-collector discover lists the configured channels. For scpi it also connects once to read *IDN? (a failure only logs a warning). Neither type scans for instruments: you give the port or the resource name.

Health

Health stream channels

serial: connected, reconnects, lines_per_s, values_per_s, parse_errors, timeouts, threads_abandoned, threads_hung (+ polls, poll_errors, overruns, last_poll_s in poll mode)
scpi:   connected, reconnects, queries_per_s, values_per_s, parse_errors, timeouts, threads_abandoned, threads_hung, polls, poll_errors, overruns, last_poll_s

Troubleshooting

Symptom	Likely cause
`connected` stays 0, log says could not open the port	Wrong port name, another program (a vendor tool, a terminal) holds it, or on Linux the user is not in the `dialout` group.
Only `timeouts` grow	Wrong baud rate or parity (garbage or silence), a null-modem cable where a straight one is needed, or a poll command with the wrong terminator.
`parse_errors` grow	The regex does not match what the log shows; the first lines that failed are written to the log. Test the regex on one line.
SCPI: every query times out	Wrong termination for serial or socket resources, or the instrument is in local mode. GPIB needs a VISA driver and `backend: '@ivi'`.

Permissions to ask the site for

serial: access to the port (a COM port number, or the dialout group on Linux) and, for a serial server, its TCP port from the collector host. Nothing is sent to the instrument unless you set a command.
scpi: TCP 111 and the instrument's VXI-11 port range, or 5025 for raw sockets; for USB or GPIB a VISA install. Install with pip install "kymoscope-collector[serial]" or [scpi].
Serial lines and most instrument LAN ports are not encrypted. Keep the collector on the instrument network; the data leaves your site encrypted.