########
Research
########


*******
Roadmap
*******
- User interface
- Rule system
- Data retention using Airflow DAGs


*****************
Related libraries
*****************
- Libs: commlib-py, funcy, mulay, pbus, pika, pytz, quixstreams, toolz
- Libs: timbermafia
- Coiled. -- https://pypi.org/project/coiled/
- Tributary -- Streaming reactive and dataflow graphs in Python

  - https://github.com/streamlet-dev/tributary/blob/main/CATALOG.md
- https://github.com/dask/dask-ml
- Perspective

  - https://perspective.finos.org/
  - https://github.com/finos/perspective
  - https://perspective.finos.org/block/?example=streaming
  - https://texodus.github.io/mtg-perspective/?seasons-in-the-abyss-67

- Waltz

  - https://waltz.finos.org/
  - http://www.waltztechnology.com/flow-diagram/1

- JavaFBP

  - https://github.com/jpaulm/javafbp
  - https://github.com/jpaulm/fbp-etl

- https://github.com/danielnuriyev/better-etl
- https://github.com/danielnuriyev/dagster-example

  - https://docs.dagster.io/concepts
  - https://docs.dagster.io/tutorial


**************
User interface
**************
- Grafana SVG overlay widget
- https://github.com/open-source-labs/Svelvet
- Realtime visualization
  https://pypi.org/project/influxdb-client/


****
Misc
****

Cogent/Skkynet
==============
- https://cogentdatahub.com/
- https://skkynet.com/
- https://skkynet.com/etk/
- https://cogentdatahub.com/docs/#dhm-book.html
- https://cogentdatahub.com/docs/#cdh-opcbridging.html
- https://cogentdatahub.com/docs/index.html#cdh-brcfgtransformations.html
- https://cogentdatahub.com/products/datahub-iot-gateway/
- https://cogentdatahub.com/features/integration/bridging/
- https://cogentdatahub.com/features/integration/tunnel-mirror/
- https://cogentdatahub.com/docs/index.html#cdh-odbcscripting.html
- https://cogentdatahub.com/features/customize/scripting/


Open MCT
========
- https://nasa.github.io/openmct/documentation/
- https://nasa.github.io/openmct/static/files/Open_MCT_Users_Guide.pdf
- https://github.com/nasa/openmct/blob/master/API.md#integrating-telemetry-sources
- https://github.com/nasa/openmct/blob/master/API.md#telemetry-datums
- https://github.com/nasa/openmct/discussions/6392
- https://github.com/nasa/openmct/discussions/4616
- https://github.com/tobiasbrown/openmct-dsn
- https://github.com/nasa/openmct/discussions/4318
- https://github.com/nasa/openmct/issues/1058


Prometheus
==========
- https://prometheus.io/docs/instrumenting/pushing/
- https://prometheus.io/docs/practices/pushing/
- https://github.com/prometheus/client_python#exporting-to-a-pushgateway


GStreamer-like pipeline definition language
===========================================
- https://gstreamer.freedesktop.org/documentation/tools/gst-launch.html


Runtime
=======
- https://pypi.org/project/python-on-whales/
- https://github.com/betaboon/testcontainers-on-whales


IO-Link
=======
- https://en.wikipedia.org/wiki/IO-Link
- https://io-link.com/
- https://www.hydac.com.au/news/post/hydac-kinesys-motion-control-and-automation.html
- https://www.hydac.com.au/news/post/build-a-smart-system-with-hydac-industrie-4-0-smart-sensors.html
- https://www.hydac.com.au/news/post/io-link-communication-interface.html
- https://www.hydac.com.au/io-link-digital-communication-interface.html


Google IoT Core MQTT
====================
- https://github.com/googleapis/python-iot/tree/main/samples/api-client/mqtt_example


****
2024
****

Singer
======
- Look into unlocking accessing Singer Taps, if that makes sense?

  - https://github.com/meltano/meltano
  - https://github.com/meltano/meltano-codespace-ready
  - https://github.com/MeltanoLabs/tap-universal-file
  - https://github.com/MeltanoLabs/tap-postgres
  - https://docs.meltano.com/concepts/plugins/
  - https://docs.meltano.com/reference/command-line-interface#elt

- Pipelinewise

  - https://transferwise.github.io/pipelinewise/
  - https://github.com/transferwise/pipelinewise
  - https://github.com/ets/tap-spreadsheets-anywhere
  - https://github.com/MeltanoLabs/meltano-map-transform
  - https://github.com/meltano/squared
  - https://transferwise.github.io/pipelinewise/installation_guide/running_pipelines.html
  - https://docs.meltano.com/concepts/plugins#transforms

- General

  - https://en.wikipedia.org/wiki/Extract,_transform,_load
  - https://en.wikipedia.org/wiki/Pipeline_(computing)
  - https://en.wikipedia.org/wiki/Data_conversion
  - https://en.wikipedia.org/wiki/Data_synchronization

Zero ETL
========
- https://www.datacamp.com/blog/what-is-zero-etl
- https://aws.amazon.com/what-is/zero-etl/
- https://aws.amazon.com/rds/aurora/zero-etl/
- https://venturebeat.com/data-infrastructure/cloud-leader-aws-shifts-its-database-focus-to-datazone-and-zero-etl/