# Using Redis with JoobQ

JoobQ is a high-performance job queue library written in Crystal, and Redis is a popular in-memory data structure store often used as a backend for queue management. This guide will walk you through configuring and optimizing Redis for use with JoobQ, ensuring scalability, reliability, and maintainability.

***

### 1. **Why Redis for JoobQ?**

Redis provides several features that make it ideal for job queue management:

* **Fast Operations:** Sub-millisecond latency for read/write operations.
* **Data Persistence:** Configurable persistence options with RDB and AOF.
* **Pub/Sub Capabilities:** Useful for real-time notifications or job processing pipelines.
* **Rich Data Types:** Lists and sorted sets are perfect for implementing queue and priority queues.
* **Wide Language Support:** Excellent support for Crystal and libraries like `redis.cr`.

***

### 2. **Getting Started**

#### 2.1 **Installing Redis**

1. **Using Package Managers:**
   * On Debian-based systems:

     ```bash
     sudo apt update
     sudo apt install redis
     ```
   * On macOS (using Homebrew):

     ```bash
     brew install redis
     ```
2. **Starting Redis:**

   ```bash
   redis-server
   ```
3. **Verify Installation:**

   ```bash
   redis-cli ping
   ```

   Output should be: `PONG`

#### 2.2 **Adding Redis to Your Crystal Project**

Add the `redis.cr` shard to your `shard.yml` file:

```yaml
dependencies:
  redis:
    github: stefanwille/crystal-redis
```

Install the shard:

```bash
shards install
```

***

### 3. **Configuring Redis for JoobQ**

#### 3.1 **Basic Connection**

Use the `Redis` shard to connect to Redis in your JoobQ application:

```crystal
require "redis"

redis = Redis.new
```

For custom configurations, you can specify options such as host, port, and database:

```crystal
redis = Redis.new(host: "127.0.0.1", port: 6379, db: 1)
```

#### 3.2 **JoobQ-Specific Configurations**

Configure JoobQ to use Redis as its backend. This is typically done in your JoobQ initialization:

```crystal
require "joobq"

JoobQ.configure do |config|
  config.store = JoobQ::RedisStore.new(
                   @host : String = ENV.fetch("REDIS_HOST", "localhost"),
                   @port : Int32 = ENV.fetch("REDIS_PORT", "6379").to_i,
                   @password : String? = ENV["REDIS_PASS"]?,
                   @pool_size : Int32 = ENV.fetch("REDIS_POOL_SIZE", "100").to_i,
                   @pool_timeout : Float64 = 0.5
  )
end
```

{% hint style="info" %}
**JoobQ uses Redis::PooledClient -** Using `Redis::PooledClient` over a simple Redis connection significantly improves JoobQ's performance, reliability, and scalability. It allows the system to handle high concurrency workloads, reduce latency, and isolate faults, ensuring efficient job processing under various load conditions. For production systems or applications expected to scale, leveraging `Redis::PooledClient` is a best practice.
{% endhint %}

#### Environment Variables

```crystal
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_POOL_SIZE=50
REDIS_PASS=somepass
REDIS_TIMEOUT=0.2
```

#### 3.3 **Securing Redis**

* **Password Authentication:** Edit your `redis.conf` file:

  ```conf
  requirepass my_secure_password
  ```

  Update the connection:

  ```crystal
  redis = Redis.new(password: "my_secure_password")
  ```
* **TLS Encryption:** If you use Redis with TLS (e.g., AWS Elasticache), configure the connection accordingly:

  ```crystal
  redis = Redis.new(host: "my-redis-host", ssl: true)
  ```

***

### 4. **Optimizing Redis for JoobQ**

#### 4.1 **Memory Management**

* **Eviction Policy:** Use `volatile-lru` or `allkeys-lru` to manage memory efficiently:

  ```conf
  maxmemory-policy volatile-lru
  ```
* **Memory Limits:** Set a maximum memory limit to prevent overuse:

  ```conf
  maxmemory 256mb
  ```

#### 4.2 **Persistence**

* For jobs requiring durability, enable AOF persistence:

  ```conf
  appendonly yes
  appendfsync everysec
  ```

#### 4.3 **Connection Pooling**

JoobQ may benefit from a Redis connection pool for high-throughput environments. Use the `redis-pool` shard for pooling:

```crystal
config.store = JoobQ::RedisStore.new(
                   ...
                   ...
                   ...
                   @pool_size : Int32 = ENV.fetch("REDIS_POOL_SIZE", "100").to_i,
                   @pool_timeout : Float64 = 0.5
  )
```

***

### 5. **Best Practices**

#### 5.1 **Monitoring Redis**

* Use `redis-cli` to monitor real-time performance:

  ```bash
  redis-cli monitor
  ```
* Enable the Redis slow log for debugging slow commands:

  ```conf
  slowlog-log-slower-than 10000
  slowlog-max-len 128
  ```

5.2 **Scaling**

* **Sharding:** Divide queues across multiple Redis instances to handle more load.
* **Redis Cluster:** For large-scale deployments, consider a Redis Cluster for horizontal scalability.

***

### 6. **Monitoring and Debugging**

#### 6.1 **JoobQ Metrics**

Signup for JoobQUI advance metrics and queue management from a centralize tool

#### 6.2 **Redis Insights**

Tools like [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) provide a GUI for monitoring Redis metrics and debugging issues.

***

### 7. **Troubleshooting**

#### 7.1 **Redis Connection Issues**

* **Error: Connection Refused**
  * Ensure Redis is running: `redis-server`
  * Check if the correct port (default: 6379) is open.
* **Error: Authentication Required**
  * Verify the password in your Redis connection configuration.

#### 7.2 **High Latency**

* Monitor `latency doctor` in `redis-cli`:

  ```bash
  redis-cli latency doctor
  ```
* Reduce latency by optimizing memory and avoiding blocking commands.

***

By following this guide, you can effectively integrate Redis with JoobQ to build scalable and reliable job queue systems. Optimize your Redis configurations to suit your workload, monitor performance, and adhere to best practices for seamless operations.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.joobq.io/guides/using-redis-with-joobq.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.