Deadlock when pushing mutliple gems at once

[jenshenny]

There is an issue being reported of a gem being pushed with multiple platforms and where some platforms weren't successfully pushed. The culprit is a deadlock from bulk_reorder_versions query.

Steps to Reproduce

This can be reproduced locally with this script that pushes 20 version of gems at once.

Example output

There was a problem saving your gem: 
There was a problem saving your gem: 
Successfully registered gem: test-gem123 (8.0.0)
Successfully registered gem: test-gem123 (11.0.0)
Successfully registered gem: test-gem123 (5.0.0)
Successfully registered gem: test-gem123 (14.0.0)
Successfully registered gem: test-gem123 (9.0.0)
There was a problem saving your gem. Please try again.
There was a problem saving your gem. Please try again.
Successfully registered gem: test-gem123 (10.0.0)
Successfully registered gem: test-gem123 (2.0.0)
Successfully registered gem: test-gem123 (20.0.0)
Successfully registered gem: test-gem123 (16.0.0)
Successfully registered gem: test-gem123 (7.0.0)
There was a problem saving your gem. Please try again.
Successfully registered gem: test-gem123 (6.0.0)
Successfully registered gem: test-gem123 (17.0.0)
Successfully registered gem: test-gem123 (15.0.0)
Server error. Please try again.
Successfully registered gem: test-gem123 (19.0.0)

In log/development.log

Exception: �[1mActiveRecord::Deadlocked: PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 79978 waits for ShareLock on transaction 765545; blocked by process 86731.
Process 86731 waits for ShareLock on transaction 765547; blocked by process 79978.
HINT:  See server log for query details.
CONTEXT:  while updating tuple (2,18) in relation "versions"

script

tmp/scripts/push_gem

#!/usr/bin/env ruby
# frozen_string_literal: true

require_relative "../../config/environment"

name = "test-gem123"

puts "Creating directory #{name}"
gem_dir = "#{__dir__}\/#{name}"
Dir.mkdir(gem_dir)

gems = []

(1..20).each do |i|
puts "Creating gemspec for #{name}"
  gemspec = <<~GEMSPEC
    Gem::Specification.new do |spec|
      spec.name = "#{name}"
      spec.version = "#{i}.0.0"
      spec.summary = "A rubygem for #{name}"
      spec.authors = "#{name}"
      spec.license = "MIT"
      spec.homepage = "https://example.com"
      spec.files = []
    end
  GEMSPEC
  gemspec_path = "#{gem_dir}/#{name}.gemspec"
  File.write(gemspec_path, gemspec)

  puts "Building gem #{name}, version #{i}.0.0"
  system("gem build #{gemspec_path} -C #{gem_dir}")
  gems << "#{name}-#{i}.0.0.gem"
end

pids = []
gems.each do |gem|
  pid = fork do
    system("RUBYGEMS_HOST=http://localhost:3000 gem push #{gem_dir}/#{gem}")
  end
  pids << pid
end

pids.each { |pid| Process.wait(pid) }

puts "Deleting gem files"
system("rm -rf #{gem_dir}")

Expected Behavior

Gems are pushed successfully

Current Behavior

Some gem versions were not pushed successfully

Possible Solution

Could probably retry on deadlock

[marcoroth]

Thanks for the repro @jenshenny! 🙏🏼

I was able to work around it in my case by changing the build-gems.yml file in this pull request: marcoroth/herb#888. That way, it's trying to release the gems sequentially instead of concurrently.