Research

Roadmap

• User interface

• Rule system

• Data retention using Airflow DAGs

Related libraries

• Libs: commlib-py, funcy, mulay, pbus, pika, pytz, quixstreams, toolz

• Libs: timbermafia

• Coiled. – https://pypi.org/project/coiled/

• Tributary – Streaming reactive and dataflow graphs in Python

  • https://github.com/streamlet-dev/tributary/blob/main/CATALOG.md

• https://github.com/dask/dask-ml

• Perspective

  • https://perspective.finos.org/

  • https://github.com/finos/perspective

  • https://perspective.finos.org/block/?example=streaming

  • https://texodus.github.io/mtg-perspective/?seasons-in-the-abyss-67

• Waltz

  • https://waltz.finos.org/

  • http://www.waltztechnology.com/flow-diagram/1

• JavaFBP

  • https://github.com/jpaulm/javafbp

  • https://github.com/jpaulm/fbp-etl

• https://github.com/danielnuriyev/better-etl

• https://github.com/danielnuriyev/dagster-example

  • https://docs.dagster.io/concepts

  • https://docs.dagster.io/tutorial

User interface

• Grafana SVG overlay widget

• https://github.com/open-source-labs/Svelvet

• Realtime visualization https://pypi.org/project/influxdb-client/

Misc

Cogent/Skkynet

• https://cogentdatahub.com/

• https://skkynet.com/

• https://skkynet.com/etk/

• https://cogentdatahub.com/docs/#dhm-book.html

• https://cogentdatahub.com/docs/#cdh-opcbridging.html

• https://cogentdatahub.com/docs/index.html#cdh-brcfgtransformations.html

• https://cogentdatahub.com/products/datahub-iot-gateway/

• https://cogentdatahub.com/features/integration/bridging/

• https://cogentdatahub.com/features/integration/tunnel-mirror/

• https://cogentdatahub.com/docs/index.html#cdh-odbcscripting.html

• https://cogentdatahub.com/features/customize/scripting/

Open MCT

• https://nasa.github.io/openmct/documentation/

• https://nasa.github.io/openmct/static/files/Open_MCT_Users_Guide.pdf

• https://github.com/nasa/openmct/blob/master/API.md#integrating-telemetry-sources

• https://github.com/nasa/openmct/blob/master/API.md#telemetry-datums

• https://github.com/nasa/openmct/discussions/6392

• https://github.com/nasa/openmct/discussions/4616

• https://github.com/tobiasbrown/openmct-dsn

• https://github.com/nasa/openmct/discussions/4318

• https://github.com/nasa/openmct/issues/1058

Prometheus

• https://prometheus.io/docs/instrumenting/pushing/

• https://prometheus.io/docs/practices/pushing/

• https://github.com/prometheus/client_python#exporting-to-a-pushgateway

GStreamer-like pipeline definition language

• https://gstreamer.freedesktop.org/documentation/tools/gst-launch.html

Runtime

• https://pypi.org/project/python-on-whales/

• https://github.com/betaboon/testcontainers-on-whales

IO-Link

• https://en.wikipedia.org/wiki/IO-Link

• https://io-link.com/

• https://www.hydac.com.au/news/post/hydac-kinesys-motion-control-and-automation.html

• https://www.hydac.com.au/news/post/build-a-smart-system-with-hydac-industrie-4-0-smart-sensors.html

• https://www.hydac.com.au/news/post/io-link-communication-interface.html

• https://www.hydac.com.au/io-link-digital-communication-interface.html

Google IoT Core MQTT

• https://github.com/googleapis/python-iot/tree/main/samples/api-client/mqtt_example

2024

Singer

• Look into unlocking accessing Singer Taps, if that makes sense?

  • https://github.com/meltano/meltano

  • https://github.com/meltano/meltano-codespace-ready

  • https://github.com/MeltanoLabs/tap-universal-file

  • https://github.com/MeltanoLabs/tap-postgres

  • https://docs.meltano.com/concepts/plugins/

  • https://docs.meltano.com/reference/command-line-interface#elt

• Pipelinewise

  • https://transferwise.github.io/pipelinewise/

  • https://github.com/transferwise/pipelinewise

  • https://github.com/ets/tap-spreadsheets-anywhere

  • https://github.com/MeltanoLabs/meltano-map-transform

  • https://github.com/meltano/squared

  • https://transferwise.github.io/pipelinewise/installation_guide/running_pipelines.html

  • https://docs.meltano.com/concepts/plugins#transforms

• General

  • https://en.wikipedia.org/wiki/Extract,_transform,_load

  • https://en.wikipedia.org/wiki/Pipeline_(computing)

  • https://en.wikipedia.org/wiki/Data_conversion

  • https://en.wikipedia.org/wiki/Data_synchronization

Zero ETL

• https://www.datacamp.com/blog/what-is-zero-etl

• https://aws.amazon.com/what-is/zero-etl/

• https://aws.amazon.com/rds/aurora/zero-etl/

• https://venturebeat.com/data-infrastructure/cloud-leader-aws-shifts-its-database-focus-to-datazone-and-zero-etl/