The HTTP Archive format, or HAR, is a JSON-formatted archive file format for logging of a web browser’s interaction with a site. The common extension for these files is .har
Awesome. Google Chrome allows you to export a session of network traffic as an HAR, as do a number of others. Nice. If you’re scraping a website and want to capture the requests that help construct that page e.g. a single page JS application that makes 10 calls to a JSONAPI. Automate this and you’ve got quite an arsenal, you can part scrape a page’s content, grab urls of all images loaded, pinch cookies, etc all from the HAR.
Tell me more, how do I construct my own????
The specification for this format is produced by the Web Performance Working Group1 of the World Wide Web Consortium (W3C). This document was never published by the W3C Web Performance Working Group and has been abandoned
Not a few days ago I was patting myself on the back for my Magical Devise Journey forcing only particular Devise strategies in some controllers. So after all that. I realised that is no good if we are logged in and that authentication is stored I am still logged in if I try to access API endpoints from the browser. What gives?
Well I was all obsessed over the strategies passed in as args I didn’t spot something.
def _perform_authentication(*args)
scope, opts = _retrieve_scope_and_opts(args)
user = nil
# Look for an existing user in the session for this scope.
# If there was no user in the session, see if we can get one from the request.
return user, opts if user = user(opts.merge(:scope => scope))
_run_strategies_for(scope, args)
if winning_strategy && winning_strategy.successful?
opts[:store] = opts.fetch(:store, winning_strategy.store?)
set_user(winning_strategy.user, opts.merge!(:event => :authentication))
end
[@users[scope], opts]
end
Hang on, if you find the user for that scope return them or try our auth strategies..
return user, opts if user = user(opts.merge(:scope => scope))
_run_strategies_for(scope, args)
Dang!
def user(argument = {})
opts = argument.is_a?(Hash) ? argument : { :scope => argument }
scope = (opts[:scope] ||= @config.default_scope)
if @users.has_key?(scope)
@users[scope]
else
unless user = session_serializer.fetch(scope)
run_callbacks = opts.fetch(:run_callbacks, true)
manager._run_callbacks(:after_failed_fetch, user, self, :scope => scope) if run_callbacks
end
@users[scope] = user ? set_user(user, opts.merge(:event => :fetch)) : nil
end
end
So where am I finding it if it’s not by strategy….oh right. I am logged in via a session cookie.
def fetch(scope)
key = session[key_for(scope)]
return nil unless key
method_name = "#{scope}_deserialize"
user = respond_to?(method_name) ? send(method_name, key) : deserialize(key)
delete(scope) unless user
user
end
So I need to nuke that session in my API controllers.
I seem to remember the preferred way to nuke sessions in controllers was something like
class ApiController < ApplicationController
protect_from_forgery with: :null_session
end
That’s the skinny. It nullifies the session and cookies in some magic Rails-y way.
module ActionController #:nodoc:
module RequestForgeryProtection
module ProtectionMethods
class NullSession
def initialize(controller)
@controller = controller
end
# This is the method that defines the application behavior when a request is found to be unverified.
def handle_unverified_request
request = @controller.request
request.session = NullSessionHash.new(request.env)
request.env['action_dispatch.request.flash_hash'] = nil
request.env['rack.session.options'] = { skip: true }
request.env['action_dispatch.cookies'] = NullCookieJar.build(request)
end
end
end
end
end
But…..
module ActionController #:nodoc:
module RequestForgeryProtection
def verify_authenticity_token
mark_for_same_origin_verification!
if !verified_request?
if logger && log_warning_on_csrf_failure
logger.warn "Can't verify CSRF token authenticity"
end
handle_unverified_request
end
end
def verified_request?
!protect_against_forgery? || request.get? || request.head? ||
valid_authenticity_token?(session, form_authenticity_param) ||
valid_authenticity_token?(session, request.headers['X-CSRF-Token'])
end
end
end
Dang! Dang! Looks like it doesn’t even bother to run the forgery strategy if it’s a GET. So it will nuke sessions but not unless the request should be verified. No worries we’ll just re-use the method of nuking the session and cookies.
class ApiResourceController < ApplicationController
prepend_before_filter do
ActionController::RequestForgeryProtection::ProtectionMethod::NullSession.new(self).handle_unverified_request
end
end
JBuilder may not be the most efficient way to present an API but for simple cases it works pretty well. Our JSON-API standardized API could return the current authenticated user like so.
show.json.jbuilder
json.data do
json.id current_user.id
json.type 'users'
json.attributes do
json.(current_user, :uuid, :email, :full_name)
end
end
All of the JBuilder examples talk in terms of Arrays of objects being built from a collection. Most of the time they probably are.
json.array! @comments do |comment|
next if comment.marked_as_spam_by?(current_user)
json.body comment.body
json.author do
json.first_name comment.author.first_name
json.last_name comment.author.last_name
end
end
Or
json.array! @people, :id, :name
But not always. If we want to add an array that is made up or arbitrary builder blocks, you find yourself thinking in terms of doing.
json.included do
json.merge! do
json.id current_user.tenant_id
json.type 'tenants'
json.attributes do
json.(current_user.tenant, :name)
end
end
end
json.included [
json.data do
json.id current_user.tenant_id
json.type 'tenants'
json.attributes do
json.(current_user.tenant, :name)
end
end
end
]
Neither of which work, turns out the way to do it is using the undocumented child! method.
json.data do
json.id current_user.id
json.type 'users'
json.attributes do
json.(current_user, :uuid, :email, :full_name)
end
json.included do
json.child! do
json.id current_user.tenant_id
json.type 'tenants'
json.attributes do
json.(current_user.tenant, :name)
end
end
end
end
Today I was solving how to skip particular Devise strategies on specific controllers. So here’s the thing, we use Devise and recently added a Doorkeeper setup for JWTAPI Authentication. This is all very nice, but the more strategies you add the more overhead there is when authenticating. 6 strategies when you know you’re only going to need one is too much. But perhaps the biggest problem is if you’re logged in with a session cookie, API responses work out of the box for the current authenticated user and we want/need our public facing API to be stateless.
This begs the question how does all this magic work anyway. Well, Devise is built on Warden, a Rack middleware. You define strategies in your User model and this in turn indirectly configures a Warden::Proxy object which is stored in the Request#env of your incoming request. I assume this is just flexible and can be accessed inside/outside of controllers in middleware, etc. When your controller needs the current user it queries the Warden::Proxy and all of the prefined strategies are consulted in turn to find it.
module Devise
module Controllers
module Helpers
module ClassMethods
def self.define_helpers(mapping) #:nodoc:
mapping = mapping.name
class_eval <<-METHODS, __FILE__, __LINE__ + 1
def authenticate_#{mapping}!(opts = {})
opts[:scope] = :#{mapping}
warden.authenticate!(opts) if !devise_controller? || opts.delete(:force)
end
def #{mapping}_signed_in?
!!current_#{mapping}
end
def current_#{mapping}
@current_#{mapping} ||= warden.authenticate(scope: :#{mapping})
end
def #{mapping}_session
current_#{mapping} && warden.session(:#{mapping})
end
METHODS
ActiveSupport.on_load(:action_controller) do
if respond_to?(:helper_method)
helper_method "current_#{mapping}", "#{mapping}_signed_in?", "#{mapping}_session"
end
end
end
# The main accessor for the warden proxy instance
def warden
request.env['warden'] or raise MissingWarden
end
end
end
end
end
You give Warden a scope and it will find your User or not.
def current_user
warden.authenticate(scope: :user)
end
module Warden
class Proxy
# Example:
# env['warden'].authenticate(:password, :basic, :scope => :sudo)
#
# :api: public
def authenticate(*args)
user, _opts = _perform_authentication(*args)
user
end
end
end
We can overwrite the authenticate_user!, current_user methods in our API base controller.
def authenticate_user!(opts = {})
opts[:scope] = :user
warden.authenticate!(:jwt, opts) if !devise_controller? || opts.delete(:force)
end
def current_user
@current_user ||= warden.authenticate(:jwt, scope: :user)
end
Or to be more flexible
class ApplicationController < ActionController::Base
class_attribute :warden_strategies
def authenticate_user!(opts = {})
opts[:scope] = :user
warden.authenticate!(*self.class.warden_strategies, opts) if !devise_controller? || opts.delete(:force)
end
def current_user
@current_user ||= warden.authenticate(*self.class.warden_strategies, scope: :user)
end
end
class ApiController < ApplicationController
self.warden_strategies = :jwt
end
If you fancy smashing your brains against a wall. I’m trying to understand this Ruby application. It’s interesting to say the least. https://github.com/tobymao/18xx
1. it’s 18xx, which is classic train game with economic strategy, stocks and share and the like. Pipe smoking (not crack) Socks and Sandals kind of gamer. Kat has 10 of these games, doesn’t wear sandals
2. written in Roda, kinda like Sinatra but different. Much more flexible. But harder learning curve. Nice routing based request flow.
3. Uses Opal, no javascript here. Ruby classes that translate to JS.
4. This feels like it was written by smart people in the most bizarre and almost deliberately convoluted way such that no one ever would want to contribute to their OpenSource project ever. I want to play
We have a broken spec for testing IP Spoofing, turns out it is raising an error.
describe 'IP Spoofing' do
it 'does not raise an IP Spoofing error' do
expect do
get '/', headers: {
'HTTP_CLIENT_IP' => '10.95.157.143',
'HTTP_X_FORWARDED_FOR' => '62.172.169.17, 141.101.99.166, 192.168.255.26'
}
expect(response.status).to eq(400)
end.not_to raise_error
end
end
… we have middleware that captures the Exception and returns a 400 Bad Request status code instead of exploding whilst upgrading to Rails 5.2. Nice 400 errors are better than Exception(s)
module Rack
class Spoofing < CustomMiddleware
def call(env)
@app.call(env)
rescue ActionDispatch::RemoteIp::IpSpoofAttackError
[400, {}, []]
end
end
Rails.application.config.middleware.insert_before Rack::Head, Spoofing
end
We insert our Spoofing middleware just ahead of Rack::Head. We can see this in rake middleware
Robs-MacBook-Pro:some rl$ rake middleware
use Webpacker::DevServerProxy
use Rack::Ping
use Rack::Sendfile
use ActionDispatch::Static
use ActionDispatch::Executor
use ActiveSupport::Cache::Strategy::LocalCache::Middleware
use Rack::Runtime
use Rack::MethodOverride
use ActionDispatch::RequestId
use RequestStore::Middleware
use ActionDispatch::RemoteIp
use Rails::Rack::Logger
use ActionDispatch::ShowExceptions
use ActionDispatch::DebugExceptions
use ActionDispatch::Reloader
use ActionDispatch::Callbacks
use ActionDispatch::Cookies
use ActionDispatch::Session::CookieStore
use ActionDispatch::Flash
use ActionDispatch::ContentSecurityPolicy::Middleware
use Rack::Spoofing
use Rack::Head
use Rack::ConditionalGet
use Rack::ETag
use Rack::TempfileReaper
use Warden::Manager
use Remotipart::Middleware
use Rack::Jayson
use PDFKit::Middleware
use Warden::JWTAuth::Middleware
run Some::Application.routes
Just wondering where that Spoofing exception is raised
Robs-MacBook-Pro:some rl$ bundle show --paths | xargs grep -r IpSpoofAttackError
/Users/rl/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/actionpack-5.2.6/lib/action_dispatch/middleware/remote_ip.rb: class IpSpoofAttackError < StandardError; end
/Users/rl/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/actionpack-5.2.6/lib/action_dispatch/middleware/remote_ip.rb: raise IpSpoofAttackError, "IP spoofing attack?! " \
There it is, it’s raised by RemoteIP, which we can see from our middleware stack is much further down the middleware stack than Rack::Head and so in Rails 5 at least runs earlier in the chain. No wonder we can’t rescue it now. In fact it makes sense to bomb out early rather than waste effort crunching cookies, etc if we don’t need to.
If we look at the Rails 5.1 middleware stack I was almost expecting our Spoofing middleware to appear before RemoteIp. It clearly doesn’t.
use Webpacker::DevServerProxy
use Rack::Ping
use Rack::Sendfile
use ActionDispatch::Static
use Rack::Lock
use BadMultipartFormDataSanitizer
use #<ActiveSupport::Cache::Strategy::LocalCache::Middleware:0x00007f8190611610>
use Rack::Runtime
use Rack::MethodOverride
use ActionDispatch::RequestId
use RequestStore::Middleware
use Rails::Rack::Logger
use ActionDispatch::ShowExceptions
use ActionDispatch::DebugExceptions
use ActionDispatch::RemoteIp
use ActionDispatch::Reloader
use ActionDispatch::Callbacks
use ActionDispatch::Cookies
use ActionDispatch::Session::CookieStore
use ActionDispatch::Flash
use Rack::Spoofing
use ActionDispatch::ParamsParser
use Remotipart::Middleware
use Rack::Head
use Rack::ConditionalGet
use Rack::ETag
use Warden::Manager
use PDFKit::Middleware
use Warden::JWTAuth::Middleware
use Rack::Jayson
run Some::Application.routes
I’m actually wondering if the 400 response even comes from our Spoofing middleware in Rails 5.
Ah, no it doesn’t. Another application error is creating it. Looks like its reaching the application and not raising the spoofing error anyway :S
The RemoteIp middleware initializer looks like this in Rails 4. Turns out check_ip_spoofing is false, something turned it off.
def initialize(app, check_ip_spoofing = true, custom_proxies = nil)
binding.pry
@app = app
@check_ip = check_ip_spoofing
@proxies = if custom_proxies.blank?
TRUSTED_PROXIES
elsif custom_proxies.respond_to?(:any?)
custom_proxies
else
Array(custom_proxies) + TRUSTED_PROXIES
end
end
Looks like there’s an environment setting to turn it on in the first place. Rails 5.x probably has a different default..
module Rails
class Application
class DefaultMiddlewareStack
attr_reader :config, :paths, :app
def initialize(app, config, paths)
@app = app
@config = config
@paths = paths
end
def build_stack
# ...
middleware.use ::ActionDispatch::RemoteIp, config.action_dispatch.ip_spoofing_check, config.action_dispatch.trusted_proxies
# ...
end
end
end
Well, I’m not turning it on in Rails 4 now. We want rid of it anyway. I am however going to move the middleware to sit just in front of RemoteIp so that it can actually catch the damn thing and make sure our 400 is actually coming from the thing we’re testing.