Logging solution

[daurnimator]

I have a bit of a pet hate for kibana and to a less degree, elastic-search. Lets set up loki (to be used from grafana).

TODO list:

• collect logs (fluent-bit vs promtail?)
• rotate logs (? is this automatic?)
• make logs searchable
• collect node logs (e.g. dmesg)
• look into @fatalbanana 's ideas below (boltdb?)
• set up ingress for local logcli usage

[fatalbanana]

Loki is cool- however the microservices-style deployment can get into a broken state if a k8s worker vanishes for example (ie. bad things will happen if ingesters do not remove themselves from the hash ring (in Consul or etcd)).

Using the monolith-style deploy avoids this problem; however it can't be scaled out & you may more likely lose logs in failure scenarios.

Support for writing boltdb indexes to the object store was recently added: it's probably worth trying that if you'd prefer not using locally attached storage (probably a good idea?)- since there is nothing like BigTable nearby.

If you (eventually?) need to go for the distributed setup I'd suggest trying memberlist for hash ring- though I'm not convinced it will solve something besides removing a dependency.

Not sure why you'd be tempted to reach for fluent-bit but in general I'd recommend using promtail.

[daurnimator]

basic (monolithic) loki setup done via #32

[daurnimator]

collect node logs (e.g. dmesg)

done via 5d8b683