Skip to content

Commit 73bca81

Browse files
iBotPeachesiBotPeaches
committed
chore: progression on cluster guide
1 parent 5127232 commit 73bca81

File tree

1 file changed

+293
-3
lines changed
  • blog/2024/2024-08-28-redis-cluster-tls-laravel

1 file changed

+293
-3
lines changed

‎blog/2024/2024-08-28-redis-cluster-tls-laravel/index.mdx‎

Lines changed: 293 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,10 @@ authors: [justinw, connort]
55
tags: [laravel, redis, redis-cluster, tls]
66
---
77

8-
## Introduction
9-
108
As a project we built evolved with websockets becoming a far more important feature we wanted to move off of a regular implementation of Redis onto a Redis Cluster to scale for larger usage while giving us the safety of failover and redundancy.
119

10+
{/* truncate */}
11+
1212
In order to test this we took our `docker-compose.yml` file and set up a few nodes to replicate a cluster that we might have configured in AWS.
1313

1414
```
@@ -39,6 +39,296 @@ While Valkey is not Redis we wanted to take this chance to explore compatibility
3939

4040
:::
4141

42+
After these containers booted up we found a [little test php script](https://gist.github.com/michael-grunder/ec1cd54b321c454d63864091ff288401) that can confirm your PhpRedis is working great.
4243

43-
{/* truncate */}
44+
```text
45+
7ed6ecafd4a4:/code# php cluster-quick-check.php --host sourcetoad_valkey_cluster_1 --port 6379
46+
Checking general cluster INFO: OK
47+
Checking [0:5460] (172.18.0.28:6379): OK
48+
Checking [5461:10922] (172.18.0.29:6379): OK
49+
Checking [10923:16383] (172.18.0.30:6379): OK
50+
Attempting to set key 'phpredis-cluster-key:0'
51+
Success setting 'phpredis-cluster-key:0'
52+
Attempting to set key 'phpredis-cluster-key:1'
53+
Success setting 'phpredis-cluster-key:1'
54+
Attempting to set key 'phpredis-cluster-key:9'
55+
Redirected to '172.18.0.28:6379'
56+
Redirected to '172.18.0.30:6379'
57+
Success setting 'phpredis-cluster-key:9'
58+
Cluster seems OK
59+
7ed6ecafd4a4:/code#
60+
```
61+
62+
Now we had the confidence of a working cluster and re-configured our Laravel installation to point to that cluster. With a single change to our `.env` we refreshed and were met with some crashes.
63+
64+
* `MOVED 15031 172.18.0.30:6379`
65+
66+
Of course that would be expected. We haven't changed anything yet, so off to the Laravel Docs we went to the [Redis Clusters section](https://laravel.com/docs/12.x/redis#clusters). The docs guided you on introducing a `clusters.default` array into your existing `config/database.php` file.
67+
68+
At the time of a base Laravel 12 install. The file would look like this:
69+
70+
```php
71+
/*
72+
|--------------------------------------------------------------------------
73+
| Redis Databases
74+
|--------------------------------------------------------------------------
75+
|
76+
| Redis is an open source, fast, and advanced key-value store that also
77+
| provides a richer body of commands than a typical key-value system
78+
| such as Memcached. You may define your connection settings here.
79+
|
80+
*/
81+
82+
'redis' => [
83+
'client' => env('REDIS_CLIENT', 'phpredis'),
84+
'options' => [
85+
'cluster' => env('REDIS_CLUSTER', 'redis'),
86+
'prefix' => env('REDIS_PREFIX', Str::slug(env('APP_NAME', 'laravel'), '_').'_database_'),
87+
'persistent' => env('REDIS_PERSISTENT', false),
88+
],
89+
'default' => [
90+
'url' => env('REDIS_URL'),
91+
'host' => env('REDIS_HOST', '127.0.0.1'),
92+
'username' => env('REDIS_USERNAME'),
93+
'password' => env('REDIS_PASSWORD'),
94+
'port' => env('REDIS_PORT', '6379'),
95+
'database' => env('REDIS_DB', '0'),
96+
],
97+
'cache' => [
98+
'url' => env('REDIS_URL'),
99+
'host' => env('REDIS_HOST', '127.0.0.1'),
100+
'username' => env('REDIS_USERNAME'),
101+
'password' => env('REDIS_PASSWORD'),
102+
'port' => env('REDIS_PORT', '6379'),
103+
'database' => env('REDIS_CACHE_DB', '1'),
104+
],
105+
],
106+
```
107+
108+
So we worked to add in a new section as described like this:
109+
110+
```php
111+
'clusters' => [
112+
'default' => [
113+
[
114+
'url' => env('REDIS_URL'),
115+
'host' => env('REDIS_HOST', '127.0.0.1'),
116+
'username' => env('REDIS_USERNAME'),
117+
'password' => env('REDIS_PASSWORD'),
118+
'port' => env('REDIS_PORT', '6379'),
119+
'database' => env('REDIS_DB', '0'),
120+
],
121+
]
122+
]
123+
```
124+
125+
Very quickly we learned this level of configuration was not for us due to the way Laravel operates. If we look at the pseudocode of how Laravel loads a Redis connection we'd see this:
126+
127+
```php
128+
$name = $name ?: 'default';
129+
$options = $this->config['options'] ?? [];
130+
131+
if (isset($this->config[$name])) {
132+
return $this->resolveConnection($this->config[$name]);
133+
}
134+
135+
if (isset($this->config['clusters'][$name])) {
136+
return $this->resolveCluster($name);
137+
}
138+
139+
// https://github.com/laravel/framework/blob/12.x/src/Illuminate/Redis/RedisManager.php#L104
140+
```
141+
142+
Since we had a `redis.default` block as well as a `redis.clusters.default` block our non-cluster connection was always loaded. The code in this resolve method has not changed in 7 years, but we are thinking that perhaps the loading of the cluster should go ahead of the normal connection. That would mean once you add the optional `clusters` block with connection name - it would win loading if both a cluster and non-cluster connection had the same name.
143+
144+
However, that is also not preferred as that change would mean the instant a code change introduced a `redis.clusters.default` block - it would win. This might explain why this code hasn't changed in almost a decade.
145+
146+
So we rewrote our configuration block slightly to this:
147+
148+
```php
149+
'clusters' => [
150+
'aws' => [
151+
[
152+
'url' => env('REDIS_CLUSTER_URL'),
153+
'host' => env('REDIS_CLUSTER_HOST', '127.0.0.1'),
154+
'username' => env('REDIS_CLUSTER_USERNAME'),
155+
'password' => env('REDIS_CLUSTER_PASSWORD'),
156+
'port' => env('REDIS_CLUSTER_PORT', '6379'),
157+
'database' => env('REDIS_CLUSTER_DB', '0'),
158+
],
159+
],
160+
],
161+
```
162+
163+
This gave us 2 major advantages:
164+
165+
1. We could deploy this change without configuring the cluster with no change.
166+
2. We could leverage a different env for cluster and non-cluster in case we had to revert quickly.
167+
168+
The downside to this is our connection was no longer named `default`. So in order to switch to our cluster connection, we invoked the connection name of `aws` which was shorthand for our ElastiCache Valkey instance in AWS.
169+
170+
Our `.env` to swap to clusters then was roughly:
171+
172+
```text
173+
QUEUE_CONNECTION=redis
174+
175+
SESSION_DRIVER=redis
176+
SESSION_CONNECTION=aws
177+
178+
CACHE_STORE=redis
179+
CACHE_PREFIX=alpha_
180+
181+
REDIS_CLUSTER_HOST=clustercfg.project-redis-cluster.xxxxxx.use1.cache.amazonaws.com
182+
REDIS_CLUSTER_PORT=6379
183+
REDIS_PERSISTENCE=true
184+
REDIS_PREFIX=alpha_
185+
REDIS_CACHE_CONNECTION=aws
186+
REDIS_CACHE_LOCK_CONNECTION=aws
187+
REDIS_QUEUE_CONNECTION=aws
188+
```
189+
190+
So lets break this down:
191+
192+
* Anything `_DRIVER`, `_STORE` or `_CONNECTION` is simply pointing that feature of Laravel to our new Redis Cluster `aws` connection.
193+
* Anything `REDIS_CLUSTER_` is for configuration of our Redis Cluster.
194+
* Anything `_PREFIX` is because Redis Clusters does NOT support multiple databases. So we prefix items to prevent collisions on a shared server.
195+
* `REDIS_PERSISTENCE` keeps Laravel using the same connection instead of opening a connection on each Redis usage.
196+
197+
With all of that we booted up our system and clicked around. We had a few crashes that became apparent when utilizing the cache or queues. These errors were:
198+
199+
* `Cannot use 'DEL' with redis-cluster`
200+
* `Cannot use 'EVAL' with redis-cluster`
201+
202+
So before taking to Google we first checked the [Queue documentation](https://laravel.com/docs/12.x/queues#redis) on Laravel and found a call-out.
203+
204+
:::warning
205+
206+
If your Redis queue connection uses a Redis Cluster, your queue names must contain a key hash tag. This is required in order to ensure all the Redis keys for a given queue are placed into the same hash slot:
207+
208+
```php
209+
'redis' => [
210+
'queue' => env('REDIS_QUEUE', '{default}'),
211+
],
212+
```
213+
214+
:::
215+
216+
So we understood the problem, but with our complex queue system and multiple lower environments on the same cluster we had to get creative to implement this properly. We introduced a new custom `QueueServiceProvider`
217+
218+
```php
219+
<?php
220+
declare(strict_types=1);
221+
222+
namespace App\Support\RedisCluster;
223+
224+
class QueueServiceProvider extends \Illuminate\Queue\QueueServiceProvider
225+
{
226+
protected function registerRedisConnector($manager): void
227+
{
228+
$manager->addConnector('redis', function () {
229+
// @phpstan-ignore-next-line
230+
return new RedisClusterConnector($this->app['redis']);
231+
});
232+
}
233+
}
234+
```
235+
236+
This loaded our custom `RedisClusterConnector`, which was basically a shell to our key override the queue class.
237+
238+
```php
239+
<?php
240+
declare(strict_types=1);
241+
242+
namespace App\Support\RedisCluster;
243+
244+
use Illuminate\Queue\Connectors\RedisConnector;
245+
246+
class RedisClusterConnector extends RedisConnector
247+
{
248+
public function connect(array $config): RedisClusterQueue
249+
{
250+
return new RedisClusterQueue(
251+
redis: $this->redis,
252+
default: $config['queue'],
253+
connection: $config['connection'] ?? $this->connection,
254+
retryAfter: $config['retry_after'] ?? 60,
255+
blockFor: $config['block_for'] ?? null,
256+
dispatchAfterCommit: $config['after_commit'] ?? null,
257+
migrationBatchSize: $config['migration_batch_size'] ?? -1
258+
);
259+
}
260+
}
261+
```
262+
263+
Now our custom `RedisClusterQueue` could be loaded.
264+
265+
```php
266+
<?php
267+
declare(strict_types=1);
268+
269+
namespace App\Support\RedisCluster;
270+
271+
use Illuminate\Queue\RedisQueue;
272+
use Illuminate\Support\Facades\App;
273+
274+
class RedisClusterQueue extends RedisQueue
275+
{
276+
public function getQueue($queue): string
277+
{
278+
$isCluster = config('queue.connections.redis.connection') === 'aws';
279+
280+
if ($isCluster) {
281+
$queueName = ($queue ?: $this->default);
282+
return sprintf('{queues:%s}', $queueName);
283+
}
284+
285+
return parent::getQueue($queue);
286+
}
287+
}
288+
```
289+
290+
All we had to do was remove the stock loader and load our own.
291+
292+
```php
293+
// Illuminate\Queue\QueueServiceProvider::class,
294+
App\Support\RedisCluster\QueueServiceProvider::class,
295+
```
296+
297+
This code would automatically build a key hash tag based on the queue name. Sure enough with successful queue tests we found the `keys *` command breaking down our naming pattern.
298+
299+
```text
300+
127.0.0.1:6379> keys *
301+
1) "local_{queues:default}:notify"
302+
2) "local_{queues:default}"
303+
3) "local_{queues:openai}"
304+
4) "local_{queues:openai}:notify"
305+
5) "local_{queues:openai-moderation}"
306+
6) "local_{queues:openai-moderation}:notify"
307+
127.0.0.1:6379>
308+
```
309+
310+
This was working great
311+
312+
---
313+
314+
#### Common Errors
315+
316+
##### `Cannot use 'EVAL' with redis-cluster`
317+
* You are missing a key hash tag `{example}` in your Redis key.
318+
319+
##### `MOVED 15031 172.18.0.30:6379`
320+
321+
* You are sending cluster commands, but your connection (PhpRedis) is not in cluster mode.
322+
* Ensure your `.env` / `config/database.php` is configured properly.
323+
324+
##### `Can't communicate with any node in the cluster`
325+
326+
* Your cluster server is unreachable (or requiring SSL) and you aren't providing it.
327+
328+
##### `Couldn't map cluster keyspace using any provided seed`
329+
330+
* Your cluster server is unreachable, generally because its expecting TLS and you aren't sending it.
331+
332+
##### Laravel Horizon won't work with a Cluster
44333

334+
* As of April 28, 2025 - Horizon [does not officially support](https://github.com/laravel/horizon/issues/274#issuecomment-457218217) Redis Clusters.

0 commit comments

Comments
 (0)