![]() As of July 2021, HDFS stores the full logs (to be verified if they do not eat too much space on HDFS). ![]() HDFS is solely used as a storage backed to store the logs for 13 months for CSIR purposes. ES must have the internal user ceph_ro configured with permissions to read ceph* indexes.Grafana uses basic auth to ES with user ceph_ro: (The password is stored in Teigi: ceph/gabe/es-ceph_ro-password).ceph_s3_fr_access-csir: Stripped down version of Nethub access logs for CSIR, retained for 13 monthsĮS is also a data source for Monit grafana dashboards:.ceph_s3_fr_daemon: Traefik service logs for Nethub.ceph_s3_fr_access: Access logs of Nethub ().ceph_s3_access-csir: Stripped down version of Gabe access logs for CSIR, retained for 13 months.ceph_s3_daemon: Traefik service logs for Gabe.ceph_s3_access: Access logs for Gabe (s3.cern.ch).This is the only whitelisted pattern, and hence the only one allowed. 13 months (stripped-down version, some fields are filtered out - see below) for CSIR purposes.other 20 days (30 total) on ceph storage.10 days on fast SSD storage, local to the ES cluster.There's not much to configure from our side, just a few useful links and the endpoint config repository: We finally have our dedicated Elasticsearch instance managed by the Elasticsearch Service. and pushes the results (full logs, and CSIR stripped version) to Elasticsearch copies a subset of fields relevant for CSIR to a different index.adds geographical information of the client IP (geoIP).parses the original message as json document.removes the additional fields introduced by the Monit infrastructure (metadata unused by us).The Dockerfile, configuration pipeline, etc., are stored in s3logs-to-es. This Logstash process runs in a Docker container on the Monit Marathon cluster (see Applications -> storage -> s3logs-to-es).įor debugging purposes, stdout and stderr of the container are available on :5050/ - They do not work from Marathon. Logstash is the tool that reads the aggregated log stream from Kafka, does most of the transformation and writes to Elasticsearch. index_prefix defines the index for the logs (is used by Logstashon Monit Marathon and on Elasticsearch).type defines if the logs are access or daemon (used to build path on HDFS).producer is s3 (used to build path on HDFS) - must be whitelisted on the Monit infra.Before sending to the Monit infrastructure, the message is prepared to define the payload data and metadata (see a): Logs from the access (daemon) file are tagged as traefik.access.* ( rvice.*), labelled as s3_access ( s3_daemon). It is installed via puppet ( exmaple for Gabe) by using the shared class fluentbit.pp responsible for installation and configuration of the fluentbit service.įluentbit on the RadosGWs+Træfik frontends is configured to tail two input files, namely the access ( /var/log/traefik/access.log) and the daemon ( /var/log/traefik/service.log) logs of Træfik. Since late April 2022, we use fluentbit on RadosGWs+Træfik frontends as it is much more gentle on memory than Logstash (which we were using previously).įluentbit tails the log files produced by Træfik (both HTTP access logs and Træfik daemon logs), add a few fields and context through metadata, and pushes the records to the Monit Logs infrastructure at URI :10013/s3 using TLS encryption. It pushes the logs to Monit Logs infrastructure for later processing by Logstash for filtering and enrichment running on Monit Marathon.Įventually, logs are then pushed to HDFS ( /project/monitoring/archive/s3/logs) and to Elasticsearch for storage and visualization. Access logs from Træfik reverse-proxy are collected via a side-car process called fluentbit.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |