“...I've been working since 2008 with Ruby / Ruby on Rails, love a bit of Elixir / Phoenix and learning Rust. I also poke through other people's code and make PRs for OpenSource Ruby projects that sometimes make it. Currently working for InPay...”

Rob Lacey
Senior Software Engineer, UK

Bugs that happen in Continuous Integration but not local

Your CI environment probably doesn’t mirror your development setup 100% of the time. So when stuff spectactularly fails testing elsewhere and not locally you might be tempted to throw your laptop in the bin.

81) BLAH
      Got 0 failures and 2 other errors:

      81.1) Failure/Error: BLAH
            
            IOError:
              closed stream
            # ./vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.46.0/lib/seahorse/client/net_http/patches.rb:29:in `block in new_transport_request'
            # ./vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.46.0/lib/seahorse/client/net_http/patches.rb:28:in `catch'
            # ./vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.46.0/lib/seahorse/client/net_http/patches.rb:28:in `new_transport_request'
            # ./vendor/bundle/ruby/2.3.0/gems/fakeweb-1.3.0/lib/fake_web/ext/net_http.rb:50:in `request_with_fakeweb'
            # ./vendor/bundle/ruby/2.3.0/gems/webmock-3.0.1/lib/webmock/http_lib_adapters/net_http.rb:97:in `block in request'
            # ./vendor/bundle/ruby/2.3.0/gems/webmock-3.0.1/lib/webmock/http_lib_adapters/net_http.rb:110:in `block in request'
            # ./vendor/bundle/ruby/2.3.0/gems/webmock-3.0.1/lib/webmock/http_lib_adapters/net_http.rb:109:in `request'
            # ./vendor/bundle/ruby/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/http/default.rb:121:in `response_for'
            # ./vendor/bundle/ruby/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/http/default.rb:76:in `request'
            # ./vendor/bundle/ruby/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/http/common.rb:62:in `call'
            # ./vendor/bundle/ruby/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/bridge.rb:166:in `execute'
            # ./vendor/bundle/ruby/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/oss/bridge.rb:584:in `execute'
            # ./vendor/bundle/ruby/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/oss/bridge.rb:50:in `get'
            # ./vendor/bundle/ruby/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/common/navigation.rb:30:in `to'
            # ./vendor/bundle/ruby/2.3.0/gems/capybara-3.10.1/lib/capybara/selenium/driver.rb:46:in `visit'
            # ./vendor/bundle/ruby/2.3.0/gems/capybara-3.10.1/lib/capybara/session.rb:265:in `visit'
            # ./vendor/bundle/ruby/2.3.0/gems/capybara-3.10.1/lib/capybara/dsl.rb:51:in `block (2 levels) in <module:DSL>'
            # ./vendor/bundle/ruby/2.3.0/gems/rspec-rails-3.8.0/lib/rspec/rails/example/feature_example_group.rb:29:in `visit'

BackboneJS, Rails 5

Basic Backbone scaffold, but….where the hell has this indentation error come from?

Showing /Users/cex/repos/robl/app/views/layouts/application.html.haml where line #7 raised:

SyntaxError: [stdin]:4:9: unexpected indentation
Rails.root: /Users/cex/repos/robl

Application Trace | Framework Trace | Full Trace
(execjs):7539:8
(execjs):7545:14
(execjs):1:102
app/views/layouts/application.html.haml:7:in `_app_views_layouts_application_html_haml__3268572193542758911_70297544684220'
config/initializers/cookies.rb:10:in `call'

Precompiling Javascript and CSS in magic, but then then Javascript bugs like this appear after running a scaffold generator which spits out 20 files then this is frustrating. A stray object which should have had some nested attributes, was just left hanging.

blah:
// should have been
blah: {}

If only it could have pointed me to the right file, rather than just a line/char in some file. Next up. How does the default Backbone scaffold cope with MongoDB → JSON objects that don’t have a standard ‘id’.

Elixir: what is a cat?

I like cats, but what are they made of?

Robs-iMac:hello roblacey$ iex
Erlang/OTP 21 [erts-10.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]

Interactive Elixir (1.7.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> 'cat'
'cat'
iex(2)> Enum.to_list 'cat'
'cat'
iex(3)> [head | tail] = 'cat'
'cat'
iex(4)> head
99
iex(5)> tail
'at'

Ok, so Single-quoted strings are really lists of characters. Double-quoted strings are not.

iex(6)> Enum.to_list "cat"   
** (Protocol.UndefinedError) protocol Enumerable not implemented for "cat"
    (elixir) lib/enum.ex:1: Enumerable.impl_for!/1
    (elixir) lib/enum.ex:141: Enumerable.reduce/3
    (elixir) lib/enum.ex:2979: Enum.reverse/1
    (elixir) lib/enum.ex:2613: Enum.to_list/1

Rails 5.1 testing

Today I got almost 300 spec errors when flipping our core app to use Rails 5.1. Why? Well, it’s just syntax in controller, request specs but the syntax makes things much clearer and is probably a welcome change if not a time consuming fix.

get :show, { :id => 1 }, { 'HTTP_REFERER' => '/something' }
get :show, params: { :id => 1 }, headers: { 'HTTP_REFERER' => '/something' }

Now XHR requests are being wonky.

xhr :show, params: { :id => 1 }
     NoMethodError:
       super: no superclass method `xhr' for #<RSpec::ExampleGroups

Ok that’s make sense.

get :show, :xhr => true, params: { :id => 1 }

Playing with Phoenix Server

Upgrade Phoenix server.

Robs-iMac:repos roblacey$ mix archive.install hex phx_new 1.4.0
Resolving Hex dependencies...
Dependency resolution completed:
New:
  phx_new 1.4.0
* Getting phx_new (Hex package)
All dependencies up to date
Compiling 10 files (.ex)
Generated phx_new app
Generated archive "phx_new-1.4.0.ez" with MIX_ENV=prod
Are you sure you want to install "phx_new-1.4.0.ez"? [Yn] Y
* creating /Users/roblacey/.mix/archives/phx_new-1.4.0

Generate the application.

Robs-iMac:repos roblacey$ mix phoenix.new hello
* creating hello/config/config.exs
* creating hello/config/dev.exs
* creating hello/config/prod.exs
* creating hello/config/prod.secret.exs
* creating hello/config/test.exs
* creating hello/lib/hello.ex
* creating hello/lib/hello/endpoint.ex
* creating hello/test/views/error_view_test.exs
* creating hello/test/support/conn_case.ex
* creating hello/test/support/channel_case.ex
* creating hello/test/test_helper.exs
* creating hello/web/channels/user_socket.ex
* creating hello/web/router.ex
* creating hello/web/views/error_view.ex
* creating hello/web/web.ex
* creating hello/mix.exs
* creating hello/README.md
* creating hello/web/gettext.ex
* creating hello/priv/gettext/errors.pot
* creating hello/priv/gettext/en/LC_MESSAGES/errors.po
* creating hello/web/views/error_helpers.ex
* creating hello/lib/hello/repo.ex
* creating hello/test/support/model_case.ex
* creating hello/priv/repo/seeds.exs
* creating hello/.gitignore
* creating hello/brunch-config.js
* creating hello/package.json
* creating hello/web/static/css/app.css
* creating hello/web/static/css/phoenix.css
* creating hello/web/static/js/app.js
* creating hello/web/static/js/socket.js
* creating hello/web/static/assets/robots.txt
* creating hello/web/static/assets/images/phoenix.png
* creating hello/web/static/assets/favicon.ico
* creating hello/test/controllers/page_controller_test.exs
* creating hello/test/views/layout_view_test.exs
* creating hello/test/views/page_view_test.exs
* creating hello/web/controllers/page_controller.ex
* creating hello/web/templates/layout/app.html.eex
* creating hello/web/templates/page/index.html.eex
* creating hello/web/views/layout_view.ex
* creating hello/web/views/page_view.ex

Fetch and install dependencies? [Yn] Y
* running mix deps.get
* running npm install && node node_modules/brunch/bin/brunch build

We are all set! Run your Phoenix application:

    $ cd hello
    $ mix phoenix.server

You can also run your app inside IEx (Interactive Elixir) as:

    $ iex -S mix phoenix.server

Before moving on, configure your database in config/dev.exs and run:

    $ mix ecto.create

Looks like I had already created a database.

Robs-iMac:hello roblacey$ mix ecto.create
warning: found quoted keyword "test" but the quotes are not required. Note that keywords are always atoms, even when quoted, and quotes should only be used to introduce keywords with foreign characters in them
  mix.exs:52

Compiling 12 files (.ex)
Generated hello app
The database for Hello.Repo has already been created

So apparently Phoenix is super fast, scary fast. Well let’s see with ab 10 concurrent requests (longest request is 144ms)

Robs-iMac:hello roblacey$ ab -n 10 -c 10 http://phoenix.dv/
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking phoenix.dv (be patient).....done


Server Software:        Cowboy
Server Hostname:        phoenix.dv
Server Port:            80

Document Path:          /
Document Length:        1931 bytes

Concurrency Level:      10
Time taken for tests:   0.059 seconds
Complete requests:      10
Failed requests:        0
Total transferred:      22600 bytes
HTML transferred:       19310 bytes
Requests per second:    169.19 [#/sec] (mean)
Time per request:       59.105 [ms] (mean)
Time per request:       5.910 [ms] (mean, across all concurrent requests)
Transfer rate:          373.41 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       0
Processing:     8   16  10.7     13      46
Waiting:        7   16  10.6     13      45
Total:          8   16  10.6     13      46

Percentage of the requests served within a certain time (ms)
  50%     13
  66%     13
  75%     13
  80%     14
  90%     46
  95%     46
  98%     46
  99%     46
 100%     46 (longest request)

1000 concurrent requests, 144ms

Robs-iMac:hello roblacey$ ab -n 1000 -c 1000 http://localhost:4000/
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        Cowboy
Server Hostname:        localhost
Server Port:            4000

Document Path:          /
Document Length:        1931 bytes

Concurrency Level:      1000
Time taken for tests:   0.152 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      2260000 bytes
HTML transferred:       1931000 bytes
Requests per second:    6569.65 [#/sec] (mean)
Time per request:       152.215 [ms] (mean)
Time per request:       0.152 [ms] (mean, across all concurrent requests)
Transfer rate:          14499.43 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   36  11.8     35      55
Processing:    25   64  16.7     69      90
Waiting:        6   64  16.8     69      90
Total:         46  100  24.2    103     144

Percentage of the requests served within a certain time (ms)
  50%    103
  66%    115
  75%    121
  80%    124
  90%    131
  95%    136
  98%    140
  99%    142
 100%    144 (longest request)

10,000 requests, Ok so I get a socket error. Too many open files (24) That’s ok, I’m not at that stage yet anyway.

Robs-iMac:hello roblacey$ ab -n 10000 -c 10000 http://localhost:4000/
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
socket: Too many open files (24)

So that’s pretty fast, but that’s just a static page. This would be better tested with a page that does a little more, or an API request that reads from the database.

Lastly just adding a new route, controller and view and template for a /hello

diff --git a/web/controllers/hello_controller.ex b/web/controllers/hello_controller.ex
new file mode 100644
index 0000000..f3e97b4
--- /dev/null
+++ b/web/controllers/hello_controller.ex
@@ -0,0 +1,7 @@
+defmodule Hello.HelloController do
+  use Hello.Web, :controller
+
+  def index(conn, params) do
+    render(conn, "world.html")
+  end
+end
diff --git a/web/router.ex b/web/router.ex
index e7a505b..6df752f 100644
--- a/web/router.ex
+++ b/web/router.ex
@@ -15,7 +15,7 @@ defmodule Hello.Router do
 
   scope "/", Hello do
     pipe_through :browser # Use the default browser stack
-
+    get "/hello", HelloController, :index
     get "/", PageController, :index
   end
 
diff --git a/web/templates/hello/world.html.eex b/web/templates/hello/world.html.eex
new file mode 100644
index 0000000..e69de29
diff --git a/web/views/hello_view.ex b/web/views/hello_view.ex
new file mode 100644
index 0000000..00f1881
--- /dev/null
+++ b/web/views/hello_view.ex
@@ -0,0 +1,3 @@
+defmodule Hello.HelloView do
+  use Hello.Web, :view
+end

Intriguing.

Bunny Consumer

No, not something that eats bunnies. I wanted to know if I could easily run a RabbitMQ consumer that just sat there listening. It seems incorrect to have a loop sat there just doing nothing all the time but I guess it the connection is open then it should just sit there, if not it can stop and have forty winks.

Gemfile

source 'https://rubygems.org'

gem 'bunny'
gem 'pry'
gem 'pry-coolline'

gimmick.rb

require 'rubygems'
# ENV['BUNDLE_GEMFILE'] ||= File.expand_path('Gemfile', __dir__)
require 'bundler/setup'
Bundler.require(:default)

conn = Bunny.new
conn.start
# open a channel
ch = conn.create_channel

# declare a queue
q = ch.queue('test1')

q.subscribe do |_, _, payload|
  Thread.new do
    puts "payload #{payload}"
    sleep(5)
    puts "finished sleeping"
  end
end

while conn.status == :open do
  # nothing is happening here
end

As expected, when an event occurs the subscribe block is called and the payload is send to STDOUT. Wait five seconds and finish. Had I not added Thread.new consumer would have waited 5 seconds and order is not quaranteed.

Robs-iMac:consumer roblacey$ ruby gimmick.rb 
payload 2
payload 1
payload 4
payload 3
payload 5
finished sleeping
finished sleeping
finished sleeping
finished sleeping
finished sleeping

Interesting to note that when using manual_ack. Events are not automatically re-read from the queue until a new connection is made if the job is not ack’ed.

q.subscribe(manual_ack: true) do |delivery_info, _, payload|
  Thread.new do
    puts "payload #{payload} - Redelivered #{delivery_info.redelivered?}\n"
    sleep(5)
    puts "finished sleeping #{payload}\n"
    puts payload.class
    ch.acknowledge(delivery_info.delivery_tag, false) unless payload == '1' && !delivery_info.redelivered?
  end
end

while conn.status == :open do
  # nothing is happening here
end

Guard / RSpec error messages

Some warnings and errors appears today, or at least I only just noticed them.

  1. Firstly I forget to load guard without bundle exec and this makes me a bad person.
  2. Apparently I might have more than one version of rspec-expectations and rb-notify installed and I should probably clean them up.

So this might be the case, oh actually. I see the problem. I have a project that has two branches, master and upgrade/rails51. I’m using RVM but each branch has a different .ruby-gemset because of course Rails 5.1 is an upgrade and will have multiple dependencies that are all upgraded too. I must remember to switch RVM environments when flipping branches.

Robs-iMac:core roblacey$ guard
Warning: you have a Gemfile, but you're not using bundler or RUBYGEMS_GEMDEPS
09:36:26 - INFO - Guard here! It looks like your project has a Gemfile, yet you are running
> [#] `guard` outside of Bundler. If this is your intent, feel free to ignore this
> [#] message. Otherwise, consider using `bundle exec guard` to ensure your
> [#] dependencies are loaded correctly.
> [#] (You can run `guard` with --no-bundler-warning to get rid of this message.)
WARN: Unresolved specs during Gem::Specification.reset:
      rb-inotify (~> 0.9, >= 0.9.7)
      rspec-expectations (~> 3.8.0)
WARN: Clearing out unresolved specs.

Converting PDFs to Greyscale

Needed to remove the colour from PDFs that were too big to print our our crappy printer.

find . -iname '*.pdf' -exec sh -c 'gs  -sOutputFile=grey/{} -sDEVICE=pdfwrite  -sColorConversionStrategy=Gray  -dProcessColorModel=/DeviceGray  -dCompatibilityLevel=1.4  -dNOPAUSE  -dBATCH {}' \;

Greyscale will print much better, although the colour versions look amazing.

dyld: Library not loaded: /usr/local/opt/icu4c/lib/libicui18n.62.dylib

I keep getting this, first in Ruby, so had to re-install Ruby 2.3.3. Now node.

Robs-iMac:something roblacey$ node -v
dyld: Library not loaded: /usr/local/opt/icu4c/lib/libicui18n.62.dylib
  Referenced from: /usr/local/bin/node
  Reason: image not found
Abort trap: 6

I must have installed something that played havoc with this lib.

Robs-iMac:something roblacey$ brew reinstall node
==> Reinstalling node 
==> Downloading https://homebrew.bintray.com/bottles/node-11.6.0.mojave.bottle.tar.gz
######################################################################## 100.0%
==> Pouring node-11.6.0.mojave.bottle.tar.gz
==> Caveats
Bash completion has been installed to:
  /usr/local/etc/bash_completion.d
==> Summary
   /usr/local/Cellar/node/11.6.0: 3,938 files, 46.6MB
Robs-iMac:something roblacey$ node -v
v11.6.0

Elixir Day #Something (because I started again) - Collection Types

So what are the Collection Types?

# List
[ 1, 2, 3 ]
# Tuple
{ 1, 2, 3 }
# Keyword List
[ one: 1, two: 2, three: 3 ]
# Map
{ one: 1, two: 2, three: 3 }

My first question was why Keyword List and Map. They have their differences.

https://stackoverflow.com/questions/28180366/what-is-the-benefit-of-keyword-lists

Speed of read/write access and Keyword Lists ordered, Maps not.

GPK of the Day Mean GENE