From the Bored Hacker Blog www.boredhackerblog.info/2021/12/noteslinks-about-log-collection-storage.html
…
*Gathering & shipping logs:*
For Windows Event Logs:
– fluentbit – docs.fluentbit.io/manual/pipeline/inputs/windows-event-log
– fluentd – docs.fluentd.org/input/windows_eventlog
– nxlog – nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#im_msvistalog
– winlogbeat – www.elastic.co/downloads/beats/winlogbeat-oss
– promtail – grafana.com/docs/loki/latest/clients/promtail/scraping/#windows-event-log
…
While doing more research, I came across clickhouse (which is also supported by some of the tools above) (clickhouse.com/) Clickhouse can store json data and you can do sql queries on that data.
I also came across cloki, which is using clickhouse but emulating loki ( github.com/lmangani/cloki)
The backend is a clickhouse database and you push logs into loki emulator, just like you’d push logs into loki. cloki also supports the same query language as loki and will work with grafana loki connector.
…
*Sample logs:*
To play with any of the tools above without making changes in production env, you can use sample logs or data sources.
github.com/logpai/loghub – github repo that links to several sample logs
www.secrepo.com/ – logs related to security. there are some network traffic logs in there
www.sec.gov/dera/data/edgar-log-file-data-set.html – EDGAR log files
log-sharing.dreamhosters.com/ – various log files
www.logs.to/ – log generator (various types)
github.com/mingrammer/flog – log generator
certstream.calidog.io/ – certificate transparency logs
www.hivemq.com/demos/websocket-client/ / broker.mqttdashboard.com – If you want to grab MQTT demo data. I’m pretty sure people are using this for free for their projects too…
…
*Gathering & shipping logs:*
For Windows Event Logs:
– fluentbit – docs.fluentbit.io/manual/pipeline/inputs/windows-event-log
– fluentd – docs.fluentd.org/input/windows_eventlog
– nxlog – nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#im_msvistalog
– winlogbeat – www.elastic.co/downloads/beats/winlogbeat-oss
– promtail – grafana.com/docs/loki/latest/clients/promtail/scraping/#windows-event-log
…
While doing more research, I came across clickhouse (which is also supported by some of the tools above) (clickhouse.com/) Clickhouse can store json data and you can do sql queries on that data.
I also came across cloki, which is using clickhouse but emulating loki ( github.com/lmangani/cloki)
The backend is a clickhouse database and you push logs into loki emulator, just like you’d push logs into loki. cloki also supports the same query language as loki and will work with grafana loki connector.
…
*Sample logs:*
To play with any of the tools above without making changes in production env, you can use sample logs or data sources.
github.com/logpai/loghub – github repo that links to several sample logs
www.secrepo.com/ – logs related to security. there are some network traffic logs in there
www.sec.gov/dera/data/edgar-log-file-data-set.html – EDGAR log files
log-sharing.dreamhosters.com/ – various log files
www.logs.to/ – log generator (various types)
github.com/mingrammer/flog – log generator
certstream.calidog.io/ – certificate transparency logs
www.hivemq.com/demos/websocket-client/ / broker.mqttdashboard.com – If you want to grab MQTT demo data. I’m pretty sure people are using this for free for their projects too…