MySQL - Too Many Connections Error

Getting Too Many Connections errors from MySQL. At the point where couldn’t determine why so many additional connections had appears from nowhere we had to immediately raise this limit. Fortunately you can do it while MySQL is still running.

mysql> show variables like 'max_connections';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 300   |
+-----------------+-------+
1 row in set (0.00 sec)

mysql> SET GLOBAL max_connections = 500;
Query OK, 0 rows affected (0.00 sec)

mysql> show variables like 'max_connections';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 500   |
+-----------------+-------+
1 row in set (0.00 sec)

You can also check how many connections there are at a given time.

mysql> show status where `variable_name` = 'Threads_connected';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| Threads_connected | 275   |
+-------------------+-------+
1 row in set (0.00 sec)

Be sure to memorialise your config change in my.cnf because restarting the server will loose this change.

[mysqld]
max_connections                 = 500

Capistrano Deployment to server with non-standard SSH port

I recently changed the default SSH port of one of my client’s servers. It appears that the first time I tried to deploy via Capistrano it hanged and I had to kill the task.

export GIT_ASKPASS="/bin/echo" GIT_SSH="/tmp/somerepo/git-ssh.sh" ; /usr/bin/env git ls-remote git@github.com:someuser/somerepo.git

I added the GIT_SSH_COMMAND export to give some debug.

export GIT_SSH_COMMAND="ssh -vvv" export GIT_ASKPASS="/bin/echo" GIT_SSH="/tmp/somerepo/git-ssh.sh" ; /usr/bin/env git ls-remote git@github.com:someuser/somerepo.git

It appears to be connecting on the custom port for that new server.

export GIT_SSH_COMMAND="ssh -vvv" export GIT_ASKPASS="/bin/echo" GIT_SSH="/tmp/somerepo/git-ssh.sh" ; /usr/bin/env git ls-remote git@github.com:someuser/somerepo.git
OpenSSH_6.9p1, OpenSSL 1.0.2d 9 Jul 2015
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to github.com [192.30.253.112] port 1234.

So for some inexplicable reason it’s trying to forward using the same port as the initial connection is made from. I can’t find an option to turn this off in the Capistrano 3 config so I am going to force the port to be 22. Editing ~/.ssh/config on the server with the following fixes it. Be sure if this is a new file to chmod permissions to 644.

Host github.com
Port 22

Sublime 3 is hanging

Sublime 3 keeps opening files and hanging and generally telling me it’s reloading open files, kill Sublime, re-open and those files are still being loaded up on start up.

Need to just nuke that session….

rm ~/Library/Application\ Support/Sublime\ Text\ 3/Local/Session.sublime_session

Boom!

Encoding and Extraneous Characters

Came across and interesting problem. We decode a base64 string that is received as a parameter and decode it to work out where to build a new path. The problem here is that the extra characters \xDB\x9D\x00 shouldn’t be there and cause YAML to blow up

2.2.2 :017 > Base64.decode64('LS0tCi0gOnBvbHltb3JwaGljX3BhdGgKLSAhcnVieS9BY3RpdmVSZWNvcmQ6%250AUHJvamVjdAogIGF0dHJpYnV0ZXM6CiAgICBpZDogODAyNQo=%250A')
 => "---\n- :polymorphic_path\n- !ruby/ActiveRecord:\xDB\x9D\x00Project\n  attributes:\n    id: 8025\n"
2.2.2 :023 > YAML.load("---\n- :polymorphic_path\n- !ruby/ActiveRecord:\xDB\x9D\x00Project\n  attributes:\n    id: 8025\n")
Psych::SyntaxError: (<unknown>): control characters are not allowed at line 1 column 1

Somehow control characters are appearing in our Base64 string and we need to strip them. It just doesn’t seem as simple as

2.2.2 :025 > "---\n- :polymorphic_path\n- !ruby/ActiveRecord:\xDB\x9D\x00Project\n  attributes:\n    id: 8025\n".gsub("\xDB\x98\x00",'')
 => "---\n- :polymorphic_path\n- !ruby/ActiveRecord:۝\u0000Project\n  attributes:\n    id: 8025\n" 

I found that if you call ord on a single string character you get the ASCII code

2.2.2 :042 > Base64.decode64('LS0tCi0gOnBvbHltb3JwaGljX3BhdGgKLSAhcnVieS9BY3RpdmVSZWNvcmQ6%250AUHJvamVjdAogIGF0dHJpYnV0ZXM6CiAgICBpZDogODAyNQo=%250A').chars[45..47].map(&:ord)
 => [219, 157, 0] 

So if I strip those ASCII characters from my string I should be good? Right?

class String
  def without_extended_characters
    chars.select {|s| (1..127).include?(s.ord) }.join
  end
end

and then

2.2.2 :043 > Base64.decode64('LS0tCi0gOnBvbHltb3JwaGljX3BhdGgKLSAhcnVieS9BY3RpdmVSZWNvcmQ6%250AUHJvamVjdAogIGF0dHJpYnV0ZXM6CiAgICBpZDogODAyNQo=%250A').without_extended_characters
 => "---\n- :polymorphic_path\n- !ruby/ActiveRecord:Project\n  attributes:\n    id: 8025\n" 

That’s much better and now YAML can decode this successfully.

It is at this stage I notice…. %250A in the Base64 string and at the end of the string. This represents a carriage return, that can happen in parameters easily enough, so I am decoding a carriage return and then stripping it from the string. I should just strip the carriage returns in the first place.

2.2.2 :045 > Base64.decode64('LS0tCi0gOnBvbHltb3JwaGljX3BhdGgKLSAhcnVieS9BY3RpdmVSZWNvcmQ6%250AUHJvamVjdAogIGF0dHJpYnV0ZXM6CiAgICBpZDogODAyNQo=%250A'.gsub('%250A',''))
 => "---\n- :polymorphic_path\n- !ruby/ActiveRecord:Project\n  attributes:\n    id: 8025\n" 

Oh well, I learnt about String#ord all the same.

Clam AV and that pesky server

----------- SCAN SUMMARY -----------
Known viruses: 6288879
Engine version: 0.98.7
Scanned directories: 79272
Scanned files: 257567
Infected files: 14
Total errors: 18985
Data scanned: 4639.85 MB
Data read: 5473.88 MB (ratio 0.85:1)
Time: 692.947 sec (11 m 32 s)

/tmp/clamav
/tmp/clamav/maldetect-1.6.tar.gz.001
/tmp/clamav/rfxn.yara.002.001
/tmp/clamav/rfxn.yara.004.001
/tmp/clamav/rfxn.yara.001
/tmp/clamav/rfxn.yara.001.001
/tmp/clamav/rfxn.yara.003.001
/tmp/clamav/x86_64.h.001

Manually restarting Puma

rails@epic-invite-staging:~/apps/epic-invite/staging/current$ bundle exec pumactl -P tmp/pids/puma.pid restart
Command restart sent success

Adding a new role to PostgreSQL

rails@snarf:/var/www/mini-epic/current$ sudo -u postgres createuser --interactive
Enter name of role to add: rails
Shall the new role be a superuser? (y/n) y
rails@snarf:/var/www/mini-epic/current$ RAILS_ENV=staging rake db:create
rails@snarf:/var/www/mini-epic/current$ psql epic_invite_staging < /tmp/mini-epic.pgsql 

Instance method defined on class?

class Dave
  def monitor; end
end

Dave.method_defined?(:monitor)
 => true 
Dave.method_defined?(:iareweasel)
 => false

Wow, never seen a C backtrace in Rails before

-- C level backtrace information -------------------------------------------
0   libruby.2.2.0.dylib                 0x0000000103e6e885 rb_vm_bugreport + 149
1   libruby.2.2.0.dylib                 0x0000000103d0dbb4 rb_bug + 468
2   libruby.2.2.0.dylib                 0x0000000103d0deac rb_bug_errno + 92
3   libruby.2.2.0.dylib                 0x0000000103e7cb2c thread_timer + 588
4   libsystem_pthread.dylib             0x00007fffbb4ac9af _pthread_body + 180
5   libsystem_pthread.dylib             0x00007fffbb4ac8fb _pthread_body + 0

-- Other runtime information -----------------------------------------------

* Loaded script: bin/rails

* Loaded features:

    0 enumerator.so
    1 rational.so
    2 complex.so

Enumerable#any?

I always thought Enumerable#any? was equivalent to Enumerable#count > 1

But no…

[nil, nil, nil].any?
#=> false