Start free

Guides

Datasets

Import files and folders, add data from arrays or text, split laps, and control what is plotted.

Import files

engine.datasets.importFiles() takes File objects (or folder entries, below) and imports them one after another. Each imported dataset is selected, so it is plotted straight away.

TypeScript
const input = document.querySelector<HTMLInputElement>('#import')!
input.addEventListener('change', async () => {
  const { ids, errors } = await engine.datasets.importFiles([...input.files!])
  if (errors.length) showToast(`${errors[0].file}: ${errors[0].error.message}`)
  input.value = ''
})
  • Delimited text (CSV, TSV, .dat, .txt: delimiter, decimal commas, units and header rows detected) and binary measurement files: MF4/MDF 3 (CAN frames decoded with a .dbc dropped alongside), EDF/BDF(+), XDF, TDMS (with its .tdms_index), HDF5, MAT v5/v7.3 and COMTRADE (.cfg + .dat, or .cff). Files with several sample rates are combined on the fastest one; datasets.splitByRate(id) gives one dataset per group, and DatasetInfo.fileInfo holds the format, start time, groups, notes and events.
  • Pass { signal } to cancel and listen to import events with phase: 'progress' for a progress bar; { chooseChannels } is asked before a file with over 2,000 channels (or an ambiguous HDF5) is read, and datasets.chooseChannels(id, chooser) reads a binary file again with other channels.
  • Files are parsed into memory while all open datasets together stay within 100 million values (rows × columns), and up to 300 MB per file. In memory, laps and math channels use every row. Larger files are ingested into DuckDB-WASM and queried per viewport; their DatasetInfo.kind is 'duckdb', and laps and math channels only see a ~12,000-row sample.
  • A file that fails does not stop the others; it lands in errors and an import event with phase: 'error' is emitted.
  • state.importing counts the files still in the queue: use it for a spinner.
  • The beforeImport engine option runs first; return false to cancel (the call rejects with import_vetoed).

Import folders

engine.folders turns folders into FolderEntry lists ({ path, file }) that importFiles accepts. Filter them with isImportable: a folder usually holds other files too.

TypeScript
async function importFolder() {
  let entries
  if (engine.folders.canPick()) {
    entries = await engine.folders.pick()            // Chrome, Edge; null when cancelled
  } else {
    entries = await pickWithInput()                  // <input type="file" webkitdirectory>
  }
  if (!entries) return
  await engine.datasets.importFiles(entries.filter((e) => engine.folders.isImportable(e.path)))
}

function pickWithInput() {
  return new Promise((resolve) => {
    const input = Object.assign(document.createElement('input'), { type: 'file', webkitdirectory: true })
    input.onchange = () => resolve(engine.folders.fromFileList(input.files!))
    input.click()
  })
}

// Drag and drop of files and folders (folders are walked recursively):
dropZone.addEventListener('drop', async (e) => {
  e.preventDefault()
  const entries = await engine.folders.fromDataTransfer(e.dataTransfer!)
  await engine.datasets.importFiles(entries.filter((x) => engine.folders.isImportable(x.path)))
})

To let users choose files inside a folder, build a tree with engine.folders.tree(entries). Each DirNode has dirs, files and allPaths (every importable path below it), which is what a tri-state checkbox tree needs.

Add data from arrays or text

TypeScript
// Numeric columns you already have (any ArrayLike<number>). Returns the id synchronously.
const id = engine.datasets.addColumns({
  name: 'Run 42',
  columns: { t: timeArray, speed: speedArray, rpm: rpmArray },
  units: { t: 's', speed: 'km/h', rpm: '1/min' },
})

// Delimited text, parsed like an imported file.
const id2 = await engine.datasets.addText(await (await fetch('/runs/42.csv')).text(), 'run-42.csv')

// New data for the same dataset, as often as every frame: pass only the columns
// that changed. Selection, colour, offset, laps and charts stay as they are.
function step() {
  simulate(speedArray)
  engine.datasets.replaceColumns(id, { speed: speedArray })
  requestAnimationFrame(step)
}
  • All columns must have the same length, or addColumns throws column_length_mismatch; no columns throws no_columns.
  • NaN and non-finite values become gaps.
  • The arrays are copied once; pass Float64Arrays for the fastest load.
  • Above 1 million rows the columns are kept as typed arrays only and decimated per viewport, like a large file: DatasetInfo.kind is 'columnar', and laps and math channels need a smaller dataset.
  • replaceColumns copies only the columns you pass (new names add columns; a different length needs every column). Charts draw each replacement on the next frame, or the same frame when it lands inside an animation frame. A dataset from addColumns can still be re-parsed with configure: its CSV text is generated then, after which it is a text dataset (and replaceColumns no longer applies).

Selection

Charts draw the selection: the ids in state.selection. Datasets, laps and live connections are all selectable by id. Changing the selection resets every worksheet’s zoom.

TypeScript
engine.datasets.select([id])      // plot only this one
engine.datasets.toggle(id)        // on/off; for a dataset split into cycles, toggles all its cycles
engine.datasets.selectAll()
engine.datasets.deselectAll()

const info = engine.getState().datasets.find((d) => d.id === id)
info.selected                     // true, false, or 'partial' when only some cycles are selected

Cycles

A dataset can be split into cycles wherever a channel starts over: a lap on a race track, a run on a test bench, a charge cycle of a battery. Each cycle becomes selectable on its own, with a value computed from a metric column. suggestBeacon finds such a channel, or returns null when nothing cycles.

TypeScript
import { suggestBeacon, formatLapValue } from '@kymoscope/sdk'

const { data } = engine.datasets.raw(id)
const beacon = suggestBeacon(data.rows, data.columnNames)   // e.g. { channel: 'CycleTime', metric: 'CycleTime', stat: 'span' }

if (beacon) await engine.datasets.configure(id, { beacon })   // stat: 'span' | 'mean' | 'max' | 'min'

for (const cycle of engine.getState().datasets.find((d) => d.id === id)!.laps!) {
  console.log(cycle.name, formatLapValue(cycle.value, cycle.unit, cycle.isTime), cycle.selected)
}

await engine.datasets.configure(id, { beacon: null })   // back to one dataset

Cycles need an in-memory dataset; configuring them on an out-of-core dataset throws invalid_argument.

Colours, offsets and parsing

TypeScript
engine.datasets.setColor(id, '#38bdf8')    // a dataset or a lap id; null restores the palette colour
engine.datasets.setOffset(lapId, -1.25)    // shift along X when several datasets are overlaid
engine.datasets.resetOffsets()

// Re-parse with another configuration (in-memory datasets only).
const { parseConfig } = engine.datasets.raw(id)
await engine.datasets.configure(id, { parseConfig: { ...parseConfig, headerRow: 2 } })

engine.datasets.remove(id)
engine.datasets.clear()                    // every dataset; worksheets and charts stay

Users can also shift a dataset by dragging on a multi-band chart: see setShiftMode.