recentpopularlog in

liqweed : kafka   35

Liftbridge - Lightweight, fault-tolerant message streams
Provides lightweight, fault-tolerant message streams by implementing a durable stream augmentation for the NATS messaging system. It extends NATS with a Kafka-like publish-subscribe log API that is highly available and horizontally scalable. Use Liftbridge as a simpler and lighter alternative to systems like Kafka and Pulsar or use it to add streaming semantics to an existing NATS deployment.
Kafka  messaging  opensource  distributed  streaming 
3 days ago by liqweed
Debezium - Stream changes from your databases
An open source distributed platform for change data capture. Start it up, point it at your databases, and your apps can start responding to all of the inserts, updates, and deletes that other apps commit to your databases. Debezium is durable and fast, so your apps can respond quickly and never miss an event, even when things go wrong.
streaming  database  persistence  tools  opensource  event-driven  Kafka  microservices 
march 2018 by liqweed
Cilium - API-aware Networking and Security
Cilium brings API-aware network security filtering to Linux container frameworks like Docker and Kubernetes. Using a new Linux kernel technology called BPF, Cilium provides a simple and efficient way to define and enforce both network-layer and application-layer security policies based on container/pod identity.
Kubernetes  networking  security  Linux  Kafka  opensource  Docker  devops  automation  mesh 
february 2018 by liqweed
kafka-connect-protobuf-converter - Protobuf converter plugin for Kafka Connect
Converter plugin for Kafka Connect. A converter controls the format of the data that will be written to Kafka for source connectors or read from Kafka for sink connectors.
Kafka  protobuf  serialization  opensource 
january 2018 by liqweed
pq - CLI Protobuf toolkit
Protobuf to json deserializer, written in Rust.

pq is a tool which deserializes protobuf messages given a set of pre-compiled .fdset files. It can understand varint/leb128-delimited streams, and it can connect to Kafka.

pq will pretty-print when outputting to a tty, but you should pipe it to jq for more fully-featured json handling.
CLI  tools  querying  serialization  parser  Rust  Kafka 
november 2017 by liqweed
kmq - Kafka-based message queue
Using kmq you can acknowledge processing of individual messages in Kafka, and have unacknowledged messages re-delivered after a timeout.
messaging  Kafka  opensource  distributed 
july 2017 by liqweed
Mocked Streams - Scala Library for Unit-Testing Processing Topologies in Apache Kafka / Kafka Streams
A library for Scala which allows you to unit-test processing topologies of Kafka Streams applications without Zookeeper and Kafka Brokers. Further, you can use your favourite Scala testing framework e.g. ScalaTest and Specs2. 
Kafka  testing  test-stub  opensource  Scala 
october 2016 by liqweed
Schema Registry - Schema registry for Kafka
A serving layer for your metadata. It provides a RESTful interface for storing and retrieving Avro schemas. It stores a versioned history of all schemas, provides multiple compatibility settings and allows evolution of schemas according to the configured compatibility setting. It provides serializers that plug into Kafka clients that handle schema storage and retrieval for Kafka messages that are sent in the Avro format.
Java  Kafka  serialization  opensource  meta-programming 
july 2016 by liqweed
vagrant-kafka - Vagrant setup config partitioned Apache Kafka with clustered Zookeeper
Vagrant config to setup a partitioned Apache Kafka installation with clustered Apache Zookeeper.
Kafka  virtualization  desktop  opensource  tools 
june 2016 by liqweed
Kafka Monitor - Execution & Monitoring tool for Kafka
A framework to implement and execute long-running kafka system tests in a real cluster. It complements Kafka’s existing system tests by capturing potential bugs or regressions that are only likely to occur after prolonged period of time or with low probability. Moreover, it allows you to monitor Kafka cluster using end-to-end pipelines to obtain a number of derived vital stats such as end-to-end latency, service availability and message loss rate. You can easily deploy Kafka Monitor to test and monitor your Kafka cluster without requiring any change to your application. By LinkedIn.
Kafka  monitoring  opensource  tools  GUI 
may 2016 by liqweed
Kafka Streams Plumber - For your dirtiest Kafka streaming jobs
Plumber is for the dirty work you do not want to do: silly transformations of your data structures because of slight mismatches. E.g. adding/removing fields, changing enums, et cetera. The transformation is described in Lua, you know, the language that scripts World of Warcraft, Redis and your wireless router at home for instance.
Kafka  streaming  opensource  big-data  conversion  Redis 
may 2016 by liqweed
Kafka Connect MongoDB
The connector is used to load data both from Kafka to Mongodb and from Mongodb to Kafka.
MongoDB  streaming  opensource  plugin  Kafka 
february 2016 by liqweed
Kafka Connect Cassandra
A Connector and Sink to write events from Kafka to Cassandra. The connector converts the value from the Kafka Connect SinkRecords to Json and uses Cassandra's JSON insert functionality to insert the rows.
Cassandra  streaming  opensource  big-data  plugin  Kafka 
february 2016 by liqweed
logback-kafka-appender - Publish application logs to Apache Kafka
This appender provides a way for applications to publish their application logs to Apache Kafka. This is ideal for applications within immutable containers without a writable filesystem.
messaging  logging  opensource  Kafka 
february 2016 by liqweed
Java Kafka examples
Snippets and small examples demonstrating kafka features and configs. By Gwen Shapira.
messaging  integration  examples  Java  opensource  big-data  streaming  Kafka 
january 2016 by liqweed
Oryx - Real-time Large Scale Machine Learning based on Spark & Kafka
A realization of the lambda architecture built on Apache Spark and Apache Kafka, but with specialization for real-time large scale machine learning. It is a framework for building applications, but also includes packaged, end-to-end applications for collaborative filtering, classification, regression and clustering.
machine-learning  streaming  big-data  opensource  Kafka 
september 2015 by liqweed
Samza - Distributed Stream Processing Framework
A distributed stream processing framework. It uses Apache Kafka for messaging, and Apache Hadoop YARN to provide fault tolerance, processor isolation, security, and resource management. By LinkedIn.
streaming  framework  opensource  Hadoop  distributed  big-data  Kafka 
august 2015 by liqweed
Burrow - Kafka Consumer Lag Checking
A monitoring companion for Apache Kafka that provides consumer lag checking as a service without the need for specifying thresholds. It monitors committed offsets for all consumers and calculates the status of those consumers on demand. An HTTP endpoint is provided to request status on demand, as well as provide other Kafka cluster information. There are also configurable notifiers that can send status out via email or HTTP calls to another service. By LinkedIn.
messaging  tools  monitoring  opensource  golang  Kafka 
june 2015 by liqweed
Samza - Distributed Stream Processing Framework
A distributed stream processing framework. It uses Apache Kafka for messaging, and Apache Hadoop YARN to provide fault tolerance, processor isolation, security, and resource management.
streaming  hadoop  opensource  big-data  messaging  Kafka 
april 2015 by liqweed
Kafka REST Proxy
A RESTful interface to a Kafka cluster. It makes it easy to produce and consume messages, view the state of the cluster, and perform administrative actions without using the native Kafka protocol or clients. Examples of use cases include reporting data to Kafka from any frontend app built in any language, ingesting messages into a stream processing framework that doesn’t yet support Kafka, and scripting administrative actions.
messaging  REST  HTTP  opensource  tools  proxy  Kafka 
march 2015 by liqweed
kafkacat - Generic CLI non-JVM Apache Kafka Producer & Consumer
Generic command line non-JVM Apache Kafka producer and consumer. Like netcat for Kafka.
CLI  messaging  tools  opensource  Kafka 
march 2015 by liqweed
plog - UDP/TCP Logging Service with Fire-and-Forget Support & Custom Pipelines
UDP/TCP logging service with fire-and-forget support & custom pipelines. Fire and forget unboxed or fragmented messages over UDP or TCP, have them forwarded to Kafka 0.8 or the standard output. That should cover syslog. By AirBNB.
logging  distributed  opensource  Groovy  Java  messaging  integration  Kafka 
may 2014 by liqweed
Akka-Kafka
Actor based kafka consumer built on top of the high level kafka consumer.

Manages backpressure so the consumer doesn't overwhelm other parts of the system. The consumer allows asynchronous/concurrent processing of a configurable bounded number of in-flight messages.
Akka  Scala  integration  messaging  opensource  Kafka 
april 2014 by liqweed
Tormenta - Scala Extensions to Storm
Scala extensions for the Storm distributed computation system. Tormenta adds a type-safe wrapper over Storm's Kafka and Kestrel spouts. This type safety allows the user to push mapping and filtering transformations down to the level of the spout itself. By Twitter.
streaming  Scala  opensource  messaging  storm  Kafka 
october 2013 by liqweed
Apache Kafka - A High-Throughput Distributed Messaging System
A publish-subscribe messaging rethought as a distributed commit log.
big-data  performance  opensource  messaging  JVM  Java  Kafka 
october 2013 by liqweed
Samza - Distributed Stream Processing Framework
A distributed stream processing framework. It uses Apache Kafka for messaging, and Apache Hadoop YARN to provide fault tolerance, processor isolation, security, and resource management.
big-data  hadoop  distributed  opensource  messaging  Kafka 
august 2013 by liqweed
Camus - MapReduce Data from Kafka to HDFS
LinkedIn's Kafka -> HDFS pipeline. It is a mapreduce job that does distributed data loads out of Kafka.
hadoop  opensource  messaging  distributed  mapreduce  big-data  Kafka 
january 2013 by liqweed
IronCount
A framework to manage consumers of Kafka topics across multiple nodes.
messaging  distributed  opensource  big-data  mapreduce  Java  Kafka 
march 2012 by liqweed
Kafka
A distributed publish-subscribe messaging system. By LinkedIn.
Scala  opensource  messaging  distributed  Java  Kafka 
december 2010 by liqweed

Copy this bookmark:





to read