Abstract
Queues
Plenty of go projects in their lifetime come to the point when they need to involve asynchronicity when processing some workload. Usually a good way to implement the asynchronous behaviour is to involve some message broker (queue mechanism). There are various technologies the Go developers may use (Redis, RabbitMQ, Kafka, SNS and many others). However, when you just need to use the basics queue mechanism to help you better distribute the jobs for Gophers (Go services replicas) these tools are usually too big gun to be used.
Our story in Dataddo.com
In dataddo.com we struggled with this a bit. We tried using some technologies mentioned above, but we faced the reality with too complex configuration problems, or the abandoned libraries or packages that where limiting us too much. Also we did not want to make our architecture complicated too much and wanted to use the technologies we were already using. So we decided to implement our custom queueing mechanism written in Go using the Postgres database. In our concrete case of switching from RabbitMQ to Postgres we made our developers happier as the used technology stack did shrink and we got nice observability about the "to be processed", "being processing" and "already processed" jobs in queue as a gift.
