35 Commits

Author SHA1 Message Date
Dan Milne
59d3385d8a Remove code which generates a deprecation warning 2023-01-26 13:04:49 +11:00
Dan Milne
3d5c102c08 Merge upstream/master 2023-01-26 13:03:05 +11:00
Nick Elser
7bb28cc007 update some badges 2021-01-21 10:38:29 -08:00
Nick Elser
65cae9aa58 update changelog & version 2021-01-21 10:32:06 -08:00
Nick Elser
6b6eb4e590 Merge pull request #15 from doits/monotonic_time
use monotonic time for retry timeout check
2021-01-21 10:27:46 -08:00
Nick Elser
485f2bff37 Merge pull request #12 from mlarraz/redis_pool
Support connection pooled Redis
2021-01-21 10:25:56 -08:00
Matt Larraz
3a37a74982 Support connection pooled Redis 2021-01-20 16:47:30 -08:00
Nick Elser
b02c256c25 Merge pull request #16 from mlarraz/ruby_versions
Switch to Github Actions and update supported Ruby versions
2021-01-20 16:45:40 -08:00
Matt Larraz
d57f6a15ac Switch to Github Actions and update supported Ruby versions 2021-01-20 16:08:37 -08:00
Dan Milne
d293ef6fcf Allow Ruby 3 2021-01-11 11:29:17 +11:00
Markus Doits
6e74322ff1 use monotonic time for retry timeout check 2021-01-07 12:52:21 +01:00
Nick Elser
b9d3f1b7a1 Bump version and update changelog. 2019-09-04 14:36:40 -07:00
Nick Elser
270c05b80e Merge pull request #10 from levkk/levkk/support-pooled-clients
Add support for pooled memcached clients by using #with
2019-09-04 14:33:26 -07:00
Lev Kokotov
60e167e146 relax bundler dependency version 2019-08-22 16:16:57 -07:00
Lev Kokotov
ad08c8b5ea bump Bundler to 2 2019-08-22 16:14:01 -07:00
Lev Kokotov
9b8ef6c244 Add support for pooled memcached clients by using #with 2019-08-22 16:04:14 -07:00
Nick Elser
b8a1d7d9ac Merge pull request #8 from nickelser/update_changelog
Update changelog
2018-10-05 14:06:42 -07:00
Nick Elser
c58a247156 Run on more modern rubies, as well. 2018-10-05 16:40:25 -04:00
Nick Elser
8c37c24ee6 Merge branch 'master' into update_changelog 2018-10-05 16:31:49 -04:00
Nick Elser
29da8cf090 Add changelog entry, remove spurious gemspec entry. 2018-10-05 16:31:33 -04:00
Nick Elser
8ed488f071 Merge pull request #5 from keylimetoolbox/double-initial-set
Fix #initial_set which is causing a double attempt and delay on lock acquisition and incorrect drop on short acquisition_timeout
2018-10-05 13:28:16 -07:00
Nick Elser
152b6acf9c Merge pull request #7 from nickelser/update_rubocop
Update Rubocop, and bump the version.
2018-10-05 13:25:44 -07:00
Nick Elser
5e10afe534 Update Rubocop, and bump the version. 2018-10-05 16:14:49 -04:00
Nick Elser
0423eb9e12 Merge pull request #6 from GandalftheGUI/ian_remillard/remove_keys_after_last_lock_released
Add 'Time To Live' to mitigate potential memory leak
2018-10-05 13:06:29 -07:00
Ian Remillard
ca46f5f369 add default for expire on set 2018-10-01 10:48:50 -07:00
Ian Remillard
1022a6f9d3 move to expire only when all locks are released 2018-10-01 10:35:30 -07:00
Ian Remillard
6be3a5bdda edits to docs 2018-09-28 16:43:44 -07:00
Ian Remillard
aa4da5d739 update docs for new options 2018-09-28 13:19:01 -07:00
Ian Remillard
fdb0b7f9d5 adds lock ttl and lock_release_removes_key 2018-09-28 12:43:52 -07:00
Jeremy Wadsack
a13edcf7d1 Fix #initial_set which is causing a double attempt and delay on lock acquisition
The call to `#initial_set` in `#retry` and `#acquire_lock` is followed by `next` which leads to a second pass through the `#retry_with_timeout` loop and a sleep call for up to `:acquisition_delay`. This delay isn't necessary if the value can be set without a race condition.

Removing the `next` call causes the client to continue to retry because the transaction has been changed outside the transaction boundary:

In Redis, calling `SET` within a `WATCH`/`UNWATCH` block but not inside a `MULTI`/`EXEC` block will [cause the EXEC to fail the transaction](https://github.com/antirez/redis-doc/issues/734), so the first `#set` call fails and it requires a second pass. To resolve this I changed `#initial_set` to call `#set` within a `MULTI` block so that it would be inside the transaction.

In Memcache the call to `SET` without the `CAS` during `#initial_set` is going to cause the `SET` with `CAS` to fail (return `EXISTS`), and resulting in a second pass. To resolve this I changed `#initial_set` to use `SET` with `CAS` and return the CAS value to be used in the subsequent `#set` call that stores the lock token.
2018-08-30 13:06:16 -07:00
Nick Elser
af1c476f08 Bump version. 2016-10-06 10:22:36 -07:00
Nick Elser
58fae54022 Minor style fixes. 2016-10-06 10:22:29 -07:00
Nick Elser
2088fd90b3 Merge pull request #1 from Shuttlerock/master
Allow to use custom token for lock
2016-10-06 10:20:35 -07:00
Vokhmin Alexey V
05661e143c Allow to use custom token for lock 2016-10-05 13:47:10 +03:00
Nick Elser
a23282dcc6 don't go around allocating empty strings willy-nilly 2015-05-07 00:16:28 -07:00
12 changed files with 152 additions and 58 deletions

38
.github/workflows/CI.yml vendored Normal file
View File

@@ -0,0 +1,38 @@
name: CI
on:
push:
branches:
- master
pull_request:
jobs:
build:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
ruby:
- '2.5'
- '2.6'
- '2.7'
- '3.0'
- ruby-head
continue-on-error: ${{ matrix.ruby == 'ruby-head' }}
services:
memcached:
image: memcached
ports:
- 11211:11211
redis:
image: redis
ports:
- 6379:6379
steps:
- uses: actions/checkout@v2
- uses: ruby/setup-ruby@v1
with:
ruby-version: ${{ matrix.ruby }}
bundler-cache: true
- run: |
bundle exec rake

View File

@@ -74,7 +74,7 @@ Style/SpaceInsideBrackets:
Style/AndOr:
Enabled: false
Style/TrailingComma:
Style/TrailingCommaInLiteral:
Enabled: true
Style/SpaceBeforeComma:
@@ -98,7 +98,7 @@ Style/SpaceAfterColon:
Style/SpaceAfterComma:
Enabled: true
Style/SpaceAfterControlKeyword:
Style/SpaceAroundKeyword:
Enabled: true
Style/SpaceAfterNot:
@@ -163,7 +163,7 @@ Style/StringLiterals:
EnforcedStyle: double_quotes
Metrics/CyclomaticComplexity:
Max: 8
Max: 10
Metrics/LineLength:
Max: 128
@@ -214,3 +214,6 @@ Metrics/ParameterLists:
Metrics/PerceivedComplexity:
Enabled: false
Style/Documentation:
Enabled: false

View File

@@ -1,6 +0,0 @@
language: ruby
rvm:
- 2.2.0
services:
- memcached
- redis-server

View File

@@ -1,3 +1,27 @@
## 0.4.0
- Monotonic clock for locks, avoiding issues with DST (thanks @doits)
- Pooled connection support (thanks @mlarraz)
- Switch to Github actions for tests (thanks @mlarraz)
- Update supported Ruby versions (thanks @mlarraz & @pat)
## 0.3.4
- Support for connection pooling when using memcached locks, via `with` blocks using Dalli (thanks to Lev).
## 0.3.3
- Default TTL for keys to allow for short-lived locking keys (thanks to Ian Remillard) without leaking memory.
- Vastly improve initial lock acquisition, especially on Redis (thanks to Jeremy Wadscak).
## 0.3.2
- Custom lock tokens (thanks to avokhmin).
## 0.3.1
- Slight memory leak fix.
## 0.3.0
- Dramatically simplify the interface by forcing clients to specify the key & resources at lock initialization instead of every method call.

View File

@@ -1,4 +1,4 @@
# Suo [![Build Status](https://travis-ci.org/nickelser/suo.svg?branch=master)](https://travis-ci.org/nickelser/suo) [![Code Climate](https://codeclimate.com/github/nickelser/suo/badges/gpa.svg)](https://codeclimate.com/github/nickelser/suo) [![Test Coverage](https://codeclimate.com/github/nickelser/suo/badges/coverage.svg)](https://codeclimate.com/github/nickelser/suo) [![Gem Version](https://badge.fury.io/rb/suo.svg)](http://badge.fury.io/rb/suo)
# Suo [![Build Status](https://github.com/nickelser/suo/workflows/CI/badge.svg)](https://github.com/nickelser/suo/actions?query=workflow%3ACI) [![Code Climate](https://codeclimate.com/github/nickelser/suo/badges/gpa.svg)](https://codeclimate.com/github/nickelser/suo) [![Gem Version](https://badge.fury.io/rb/suo.svg)](http://badge.fury.io/rb/suo)
:lock: Distributed semaphores using Memcached or Redis in Ruby.
@@ -46,7 +46,7 @@ suo = Suo::Client::Memcached.new("protected_key", client: some_dalli_client, acq
# manually locking/unlocking
# the return value from lock without a block is a unique token valid only for the current lock
# which must be unlocked manually
token = suo
token = suo.lock
foo.baz!
suo.unlock(token)
@@ -72,12 +72,21 @@ suo.lock do |token|
end
```
### Time To Live
```ruby
Suo::Client::Redis.new("bar_resource", ttl: 60) #ttl in seconds
```
A key representing a set of lockable resources is removed once the last resource lock is released and the `ttl` time runs out. When another lock is acquired and the key has been removed the key has to be recreated.
## TODO
- more race condition tests
## History
View the [changelog](https://github.com/nickelser/suo/blob/master/CHANGELOG.md)
View the [changelog](https://github.com/nickelser/suo/blob/master/CHANGELOG.md).
## Contributing

View File

@@ -2,7 +2,6 @@ require "securerandom"
require "monitor"
require "dalli"
require "dalli/cas/client"
require "redis"

View File

@@ -5,25 +5,30 @@ module Suo
acquisition_timeout: 0.1,
acquisition_delay: 0.01,
stale_lock_expiration: 3600,
resources: 1
resources: 1,
ttl: 60,
}.freeze
BLANK_STR = "".freeze
attr_accessor :client, :key, :resources, :options
include MonitorMixin
def initialize(key, options = {})
fail "Client required" unless options[:client]
@options = DEFAULT_OPTIONS.merge(options)
@retry_count = (@options[:acquisition_timeout] / @options[:acquisition_delay].to_f).ceil
@client = @options[:client]
@resources = @options[:resources].to_i
@key = key
super() # initialize Monitor mixin for thread safety
end
def lock
token = acquire_lock
def lock(custom_token = nil)
token = acquire_lock(custom_token)
if block_given? && token
begin
@@ -51,16 +56,13 @@ module Suo
retry_with_timeout do
val, cas = get
if val.nil?
initial_set
next
end
cas = initial_set if val.nil?
cleared_locks = deserialize_and_clear_locks(val)
refresh_lock(cleared_locks, token)
break if set(serialize_locks(cleared_locks), cas)
break if set(serialize_locks(cleared_locks), cas, expire: cleared_locks.empty?)
end
end
@@ -77,7 +79,7 @@ module Suo
acquisition_lock = remove_lock(cleared_locks, token)
break unless acquisition_lock
break if set(serialize_locks(cleared_locks), cas)
break if set(serialize_locks(cleared_locks), cas, expire: cleared_locks.empty?)
end
rescue LockClientError => _ # rubocop:disable Lint/HandleExceptions
# ignore - assume success due to optimistic locking
@@ -91,16 +93,13 @@ module Suo
attr_accessor :retry_count
def acquire_lock
token = SecureRandom.base64(16)
def acquire_lock(token = nil)
token ||= SecureRandom.base64(16)
retry_with_timeout do
val, cas = get
if val.nil?
initial_set
next
end
cas = initial_set if val.nil?
cleared_locks = deserialize_and_clear_locks(val)
@@ -124,7 +123,7 @@ module Suo
fail NotImplementedError
end
def initial_set(val = "") # rubocop:disable Lint/UnusedMethodArgument
def initial_set(val = BLANK_STR) # rubocop:disable Lint/UnusedMethodArgument
fail NotImplementedError
end
@@ -133,10 +132,10 @@ module Suo
end
def retry_with_timeout
start = Time.now.to_f
start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
retry_count.times do
elapsed = Time.now.to_f - start
elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start
break if elapsed >= options[:acquisition_timeout]
synchronize do
@@ -158,7 +157,7 @@ module Suo
end
def deserialize_locks(val)
unpacked = (val.nil? || val == "") ? [] : MessagePack.unpack(val)
unpacked = (val.nil? || val == BLANK_STR) ? [] : MessagePack.unpack(val)
unpacked.map do |time, token|
[Time.at(time), token]

View File

@@ -7,21 +7,29 @@ module Suo
end
def clear
@client.delete(@key)
@client.with { |client| client.delete(@key) }
end
private
def get
@client.get_cas(@key)
@client.with { |client| client.get_cas(@key) }
end
def set(newval, cas)
@client.set_cas(@key, newval, cas)
def set(newval, cas, expire: false)
if expire
@client.with { |client| client.set_cas(@key, newval, cas, @options[:ttl]) }
else
@client.with { |client| client.set_cas(@key, newval, cas) }
end
end
def initial_set(val = "")
@client.set(@key, val)
def initial_set(val = BLANK_STR)
@client.with do |client|
client.set(@key, val)
_val, cas = client.get_cas(@key)
cas
end
end
end
end

View File

@@ -1,39 +1,54 @@
module Suo
module Client
class Redis < Base
OK_STR = "OK".freeze
def initialize(key, options = {})
options[:client] ||= ::Redis.new(options[:connection] || {})
super
end
def clear
@client.del(@key)
with { |r| r.del(@key) }
end
private
def with(&block)
if @client.respond_to?(:with)
@client.with(&block)
else
yield @client
end
end
def get
[@client.get(@key), nil]
[with { |r| r.get(@key) }, nil]
end
def set(newval, _)
ret = @client.multi do |multi|
multi.set(@key, newval)
def set(newval, _, expire: false)
ret = with do |r|
r.multi do |rr|
if expire
rr.setex(@key, @options[:ttl], newval)
else
rr.set(@key, newval)
end
end
end
ret && ret[0] == "OK"
ret && ret[0] == OK_STR
end
def synchronize
@client.watch(@key) do
yield
end
with { |r| r.watch(@key) { yield } }
ensure
@client.unwatch
with { |r| r.unwatch }
end
def initial_set(val = "")
@client.set(@key, val)
def initial_set(val = BLANK_STR)
set(val, nil)
nil
end
end
end

View File

@@ -1,3 +1,3 @@
module Suo
VERSION = "0.3.0"
VERSION = "0.4.0".freeze
end

View File

@@ -16,19 +16,18 @@ Gem::Specification.new do |spec|
spec.files = `git ls-files -z`.split("\x0")
spec.bindir = "bin"
spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
spec.test_files = spec.files.grep(%r{^(test|spec|features)/})
spec.require_paths = ["lib"]
spec.required_ruby_version = "~> 2.0"
spec.required_ruby_version = ">= 2.5"
spec.add_dependency "dalli"
spec.add_dependency "redis"
spec.add_dependency "msgpack"
spec.add_development_dependency "bundler", "~> 1.5"
spec.add_development_dependency "rake", "~> 10.0"
spec.add_development_dependency "rubocop", "~> 0.30.0"
spec.add_development_dependency "bundler"
spec.add_development_dependency "rake", "~> 13.0"
spec.add_development_dependency "rubocop", "~> 0.49.0"
spec.add_development_dependency "minitest", "~> 5.5.0"
spec.add_development_dependency "codeclimate-test-reporter", "~> 0.4.7"
end

View File

@@ -31,6 +31,12 @@ module ClientTests
assert_equal false, locked
end
def test_lock_with_custom_token
token = 'foo-bar'
lock = @client.lock token
assert_equal lock, token
end
def test_empty_lock_on_invalid_data
@client.send(:initial_set, "bad value")
assert_equal false, @client.locked?