“...I've been working since 2008 with Ruby / Ruby on Rails, love a bit of Elixir / Phoenix and learning Rust. I also poke through other people's code and make PRs for OpenSource Ruby projects that sometimes make it. Currently working for InPay...”

Rob Lacey (contact@robl.me)
Senior Software Engineer, Brighton, UK

Uploading files direct to S3 with AWS gem

Sometimes you just have to upload huge files to S3, today it was a 10Gb mp4. We established a long time ago that uploading via a Rails app via Paperclip was never going to work for files over a certain size. Indeed it is a waste of resources to upload a temporary file to our application server to act as a proxy to S3. So we’ve been uploading to S3 directly for some time with some changes to Paperclip and PLUpload. Today, PLUpload fell over on a 10Gb file and that’s fine but we need a solution when this fails. Personally I find the AWS gem documentation quite difficult to navigate and never having quite as many examples as I would like. So….

s3 = Aws::S3::Resource.new(
  region: 'us-east-1',
  access_key_id: 'ACCESS_KEY_ID',
  secret_access_key: 'SECRET_ACCESS_KEY'
)
# => #<Aws::S3::Resource:0x007fe7a6a81d18 @client=#<Aws::S3::Client> 

obj = s3.bucket('assets.blah.com').object('video/001/011/221/original-files.mp4')

# => #<Aws::S3::Object:0x007fe7a5a2afc8 @bucket_name="assets.blah.com", @key="video/001/011/221/original-files.mp4", @data=nil, @client=#<Aws::S3::Client> 

obj.upload_file("/Users/roblacey/Desktop/original-files.mp4")

LD-JSON Game

Testing out LD-JSON Markup for Game

<script type='application/ld+json'>{
  "@context": "http://schema.org",
  "@type": "Game",
  "mainEntityOfPage": {
    "@type": "WebPage",
    "@id": "https://robl.me/ld-json-game"
  },
  "name": "The Quest For Shangri-La",
  "image": [
    "https://cf.geekdo-images.com/u2_soEahqZ44vw2Bvy_Qw6ZTiP8=/0x0/pic200178.jpg"
  ],
  "description": "The Battle Begins!

Who can say what rewards or tortures await us when the curtain of death is parted? Those who listen to the message of the Dark Carnival through the songs and lives of the Insane Clown Posse know of one such destination for the pure of heart: Shangri-La, with its fountains of FAYGO!!!!, music blasting over green fields, and fine hotties everywhere, in an endless existence free of care, drama, and death.

Shangri-La, paradise though it may be, is not beyond the reach of evil. A sinister shadow has fallen over its once radiant light - evidence of an evil beyond imagining!

Get ready to battle your way through Detroit, the Nethervoid, and the Dark Carnival as you travel the board on an epic Quest to cross The Bridge to Shangri-La. Only there can you solve the mystery by revealing one of the ten different ending cards!

Will you survive...?
Deadly Traps!
Backstabbing Players!
And worst of all, the dark horrors awaiting you in Shangri-La!
Or will you succumb to the shadows, like the rest?

Draw cards and...
Battle Savage Fiends!
Uncover Magical Artifacts!
Find Helpful Homies!
Amass Weapons and Cash!

You play a character...
on a Quest to reach Shangri-La, at the center of the board. Will you be a Dark Carnival character like Violent J or Jamie Madrox, driven to save Shangri-La from the evil that plagues it? Or will you play a Nethervoid character like Cemetary Girl or Ol' Evil Eye, bent on taking over all of Shangri-La for yourself?

Winner: Underground Game of the Year!
Juggalos, come out to play-ee-ay!

For 2-6 Players, Ages 13 and up
Playing Time: 2 Hours
Available at HatchetGear.com or a store near you!"
}</script>

Darwinning! card game. Evolve to rule the food chain!

Kat and I worked on this first walkthrough video for I Play Red a nice first project and plenty of lessons learned. The game itself is really nice, and I think we’re going to back it even though we have a prototype copy. I am looking forward to what she decides to do for her second video.

The Walkthrough for Darwinning! is up now up, and the new artwork looks amazing. Personally I prefer the new art, Kat still likes the quirkiness of the older art.

Eliminate annoying errors...grrr

DEPRECATION WARNING: Sprockets method `register_engine` is deprecated.
Please register a mime type using `register_mime_type` then
use `register_compressor` or `register_transformer`.
https://github.com/rails/sprockets/blob/master/guides/extending_sprockets.md#supporting-all-versions-of-sprockets-in-processors
 (called from block (2 levels) in <class:Railtie> at /Users/rl/.rvm/gems/ruby-2.3.0@themadkatter/gems/sass-rails-5.0.4/lib/sass/rails/railtie.rb:57)

Solution: https://github.com/rails/sass-rails/issues/381

Edit Gemfile.lock

...
gem 'thor', '3.6.3'
...

Then bundle update

Robs-MBP:iplayred rl$ bundle update sprockets
Fetching source index from http://gems.github.com/
Fetching source index from https://rubygems.org/
Resolving dependencies......
...
Fetching sprockets 3.6.3 (was 3.7.1)
Installing sprockets 3.6.3 (was 3.7.1)
...
Bundler attempted to update sprockets but its version regressed from 3.7.1 to 3.6.3
Bundle updated!
Expected string default value for '--rc'; got false (boolean)
Rails 4.2.7.1

Apparently this is because of a change in the thor gem https://stackoverflow.com/questions/40881938/new-ruby-on-rails-setup-expected-string-default-value-for-rc-got-false

Robs-MBP:iplayred rl$ cat Gemfile.lock | grep thor
    railties (4.2.7.1)
      actionpack (= 4.2.7.1)
      activesupport (= 4.2.7.1)
      rake (>= 0.8.7)
      thor (>= 0.18.1, < 2.0)

Looks like I need to downgrade to 0.19.1.

Edit Gemfile.lock

...
gem 'thor', '0.19.1'
...

Then bundle update

Robs-MBP:iplayred rl$ bundle update thor
Fetching source index from http://gems.github.com/
Fetching source index from https://rubygems.org/
Resolving dependencies......
...
Using thor 0.19.1 (was 0.19.4)
...
Bundler attempted to update thor but its version regressed from 0.19.4 to 0.19.1
Bundle updated!
Robs-MBP:iplayred rl$ rails version
Looks like your app's ./bin/rails is a stub that was generated by Bundler.

In Rails 4, your app's bin/ directory contains executables that are versioned
like any other source code, rather than stubs that are generated on demand.

Here's how to upgrade:

  bundle config --delete bin    # Turn off Bundler's stub generator
  rake rails:update:bin         # Use the new Rails 4 executables
  git add bin                   # Add bin/ to source control

You may need to remove bin/ from your .gitignore as well.

When you install a gem whose executable you want to use in your app,
generate it and add it to source control:

  bundle binstubs some-gem-name
  git add bin/new-executable

Rails 4.2.7.1

Bin stubs

Robs-MBP:iplayred rl$ rails c
Looks like your app's ./bin/rails is a stub that was generated by Bundler.

In Rails 4, your app's bin/ directory contains executables that are versioned
like any other source code, rather than stubs that are generated on demand.

Here's how to upgrade:

  bundle config --delete bin    # Turn off Bundler's stub generator
  rake rails:update:bin         # Use the new Rails 4 executables
  git add bin                   # Add bin/ to source control

You may need to remove bin/ from your .gitignore as well.

When you install a gem whose executable you want to use in your app,
generate it and add it to source control:

  bundle binstubs some-gem-name
  git add bin/new-executable

I Play Red

Rather proud to have relaunched I Play Red with a far cleaner interface and presentation. It was important to take the visually stunning elements of Boardgame and Table top games and make them more prominent, and better organising the other content that while still relevant is secondary to the blog posts and game walkthroughs. The design makes rather nice use of parallax to divide the sections and moves to a more streamlined and readable layout.

The work was a response to an increase in content and all in preparation for the Spiel 2017 event in Essen and so was time sensitive.

Why every promoter should be using Songkick

Back in 2009, whilst freelancing, I applied to join both the teams of PledgeMusic and Songkick . I was lucky enough to join PledgeMusic and I am still there today. SongKick are still a cool company though and their product is a very simple one. They harvest gig listing data from thousands of sources and build a list of venues, bands and gigs (historical and upcoming).

This data is very simple yet very powerful, you can use their website or their app to select your city (Brighton) or cities you’d travel to (hell anywhere cool in Europe) and import your favourite artists from Facebook, Spotify or your music from your phone. That’s it, but this means you can now easily search for what bands that you like are coming up and track those events, announce you’re going or if they have tickets buy them from Songkick or via one of their partner vendors (the usual suspects).

Having travelled to Europe plenty in the last 10 years this services makes it really easy to add a gig to our sightseeing itinerary. Last year it was D-A-D in Gothenberg, Conny Bloom in Oreboro and then into Stockholm for a pootle around the city before home. And before that we travelled to Brussels to see Messer Chups after getting similar notifications.

This service is not only powerful for Consumers but Bands and Promoters should take the time to ensure their events are listed on the site, add them if not or just ensure the bands are listed properly and date and venue are accurate. And this guarantees that little bit more exposure and your target customers don’t end up missing Messer Chups when they come to Brighton. This may have happened :(

Having decided to attend Mammothfest 2017 this year. I took a look at the listing and it appeared that over 20 bands (both established and new on the scene) were not listed. Since any user can edit a listing I added in the missing ones and one day later the app is notifying me about new gigs for ‘Hole In The Sky’ and ‘Mutually Assured Destruction". Excellent, so not only is the gig listing more accurate but fans are automatically notified that new bands are added to the bill. Potentially making that push in sales that any artists, promoter, gig needs. Don’t miss out on that little bit more exposure because just a Facebook Event just doesn’t cut it anymore

So why not pop on over and let me know if you’re also going to Damnation 2017

MySQL - Too Many Connections Error

Getting Too Many Connections errors from MySQL. At the point where couldn’t determine why so many additional connections had appears from nowhere we had to immediately raise this limit. Fortunately you can do it while MySQL is still running.

mysql> show variables like 'max_connections';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 300   |
+-----------------+-------+
1 row in set (0.00 sec)

mysql> SET GLOBAL max_connections = 500;
Query OK, 0 rows affected (0.00 sec)

mysql> show variables like 'max_connections';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 500   |
+-----------------+-------+
1 row in set (0.00 sec)

You can also check how many connections there are at a given time.

mysql> show status where `variable_name` = 'Threads_connected';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| Threads_connected | 275   |
+-------------------+-------+
1 row in set (0.00 sec)

Be sure to memorialise your config change in my.cnf because restarting the server will loose this change.

[mysqld]
max_connections                 = 500

Capistrano Deployment to server with non-standard SSH port

I recently changed the default SSH port of one of my client’s servers. It appears that the first time I tried to deploy via Capistrano it hanged and I had to kill the task.

export GIT_ASKPASS="/bin/echo" GIT_SSH="/tmp/somerepo/git-ssh.sh" ; /usr/bin/env git ls-remote git@github.com:someuser/somerepo.git

I added the GIT_SSH_COMMAND export to give some debug.

export GIT_SSH_COMMAND="ssh -vvv" export GIT_ASKPASS="/bin/echo" GIT_SSH="/tmp/somerepo/git-ssh.sh" ; /usr/bin/env git ls-remote git@github.com:someuser/somerepo.git

It appears to be connecting on the custom port for that new server.

export GIT_SSH_COMMAND="ssh -vvv" export GIT_ASKPASS="/bin/echo" GIT_SSH="/tmp/somerepo/git-ssh.sh" ; /usr/bin/env git ls-remote git@github.com:someuser/somerepo.git
OpenSSH_6.9p1, OpenSSL 1.0.2d 9 Jul 2015
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to github.com [192.30.253.112] port 1234.

So for some inexplicable reason it’s trying to forward using the same port as the initial connection is made from. I can’t find an option to turn this off in the Capistrano 3 config so I am going to force the port to be 22. Editing ~/.ssh/config on the server with the following fixes it. Be sure if this is a new file to chmod permissions to 644.

Host github.com
Port 22

Sublime 3 is hanging

Sublime 3 keeps opening files and hanging and generally telling me it’s reloading open files, kill Sublime, re-open and those files are still being loaded up on start up.

Need to just nuke that session….

rm ~/Library/Application\ Support/Sublime\ Text\ 3/Local/Session.sublime_session

Boom!

Encoding and Extraneous Characters

Came across and interesting problem. We decode a base64 string that is received as a parameter and decode it to work out where to build a new path. The problem here is that the extra characters \xDB\x9D\x00 shouldn’t be there and cause YAML to blow up

2.2.2 :017 > Base64.decode64('LS0tCi0gOnBvbHltb3JwaGljX3BhdGgKLSAhcnVieS9BY3RpdmVSZWNvcmQ6%250AUHJvamVjdAogIGF0dHJpYnV0ZXM6CiAgICBpZDogODAyNQo=%250A')
 => "---\n- :polymorphic_path\n- !ruby/ActiveRecord:\xDB\x9D\x00Project\n  attributes:\n    id: 8025\n"
2.2.2 :023 > YAML.load("---\n- :polymorphic_path\n- !ruby/ActiveRecord:\xDB\x9D\x00Project\n  attributes:\n    id: 8025\n")
Psych::SyntaxError: (<unknown>): control characters are not allowed at line 1 column 1

Somehow control characters are appearing in our Base64 string and we need to strip them. It just doesn’t seem as simple as

2.2.2 :025 > "---\n- :polymorphic_path\n- !ruby/ActiveRecord:\xDB\x9D\x00Project\n  attributes:\n    id: 8025\n".gsub("\xDB\x98\x00",'')
 => "---\n- :polymorphic_path\n- !ruby/ActiveRecord:۝\u0000Project\n  attributes:\n    id: 8025\n"

I found that if you call ord on a single string character you get the ASCII code

2.2.2 :042 > Base64.decode64('LS0tCi0gOnBvbHltb3JwaGljX3BhdGgKLSAhcnVieS9BY3RpdmVSZWNvcmQ6%250AUHJvamVjdAogIGF0dHJpYnV0ZXM6CiAgICBpZDogODAyNQo=%250A').chars[45..47].map(&:ord)
 => [219, 157, 0]

So if I strip those ASCII characters from my string I should be good? Right?

class String
  def without_extended_characters
    chars.select {|s| (1..127).include?(s.ord) }.join
  end
end

and then

2.2.2 :043 > Base64.decode64('LS0tCi0gOnBvbHltb3JwaGljX3BhdGgKLSAhcnVieS9BY3RpdmVSZWNvcmQ6%250AUHJvamVjdAogIGF0dHJpYnV0ZXM6CiAgICBpZDogODAyNQo=%250A').without_extended_characters
 => "---\n- :polymorphic_path\n- !ruby/ActiveRecord:Project\n  attributes:\n    id: 8025\n"

That’s much better and now YAML can decode this successfully.

It is at this stage I notice…. %250A in the Base64 string and at the end of the string. This represents a carriage return, that can happen in parameters easily enough, so I am decoding a carriage return and then stripping it from the string. I should just strip the carriage returns in the first place.

2.2.2 :045 > Base64.decode64('LS0tCi0gOnBvbHltb3JwaGljX3BhdGgKLSAhcnVieS9BY3RpdmVSZWNvcmQ6%250AUHJvamVjdAogIGF0dHJpYnV0ZXM6CiAgICBpZDogODAyNQo=%250A'.gsub('%250A',''))
 => "---\n- :polymorphic_path\n- !ruby/ActiveRecord:Project\n  attributes:\n    id: 8025\n"

Oh well, I learnt about String#ord all the same.