This is a repost of my original post on dev.to. I will make some very minor changes and include some new insights I leanred.
I would like to share my experience configuring a very simple logging service with PHP and Clickhouse. I need to stress that this is by no mean an optimal setup, I simply want to share my experience and I would love to hear feedbacks to improve.
Requirements
A very simple logging service to track all kinds of events such as nginx error or php error internally and externally. I find ELK too complicated to properly setup and guard, and too resource intensive (and thus costly) for my needs.
Performant enough to handle tens of millions of logs per day on an average server (I pick the Vultr instance below which has 2 CPU and 4 GB of RAM). I'm handling over tens of million logs per day with this setup and still have resources to spare.
Tech stacks
PHP
Clickhouse
Vector
Why PHP?
Because I'm familiar with it and because why not? I don't want to spend days and weeks trying to learn a new thing just for something really quick that I can do for a 1-2 hour with PHP.
Why Clickhouse?
Clickhouse is a very performant, easy-to-setup database designed and optimized for writing data. I have great experience using it to track views and clicks on my website. That said, any database should do just fine as long as it can handle your log volume efficiently.
Why Vector?
I looked at many log collectors such as syslog, syslog-ng, filebeat etc... I'm not a system guy and their documents are terrible, terrible. Filebeat is not so bad but I was not entirely impressed with it (I used it in the past). Then I found Vector and I was blown away by the website (so clean), the documents, the options, and the support.
Preparation
First, we need to collect the logs. Vector is super simple for collecting logs, you can find the website here with all the documents:
https://vector.dev/
I'm going to share some of my current configs, please do note that I'm a Vector virgin, so please do review before use and please let me know if I can improve anything:
https://gist.github.com/yellow1912/782d53cb97b9de86d4fdb2848cc2e520
You will notice that I'm opting to send the logs to a remote server via http/https protocol. You can choose any other protocol you want. I also pass along an authorization token so my log service can check to ensure the authenticity of the log data.
Next time, I will cover Clickhouse database setup.
