Understanding autoload_paths and namespaces in Ruby on Rails

0 Comments

Image Credit: Ilze Lucero - unsplash.com/@ilzelucero

As I work on larger and more complex Rails projects I use namespaces and directory structures to better organise the code. However, I found myself having to add the odd require_relative in my unit tests to work around NameError: uninitialized constant errors where Rails couldn’t automatically find the classes I was referencing in my now neatly organised projects.

The official documentation is comprehensive, but to me this subject reads at an intermediate to advanced level - I’m pretty sure I had missed something basic so it was time for a little investigation.

Skip the investigation and go straight to the summary

Using Ruby on Rails we start with a new app, a single Product model, and a PriceCalculator whose location we’ll play with: (I’m using Rails 5.2.3 here)
rails new MyApp --api && cd MyApp
rails g model product name:string sku:string:index 'price:decimal{10,2}'

Running via Spring preloader in process 20474
      invoke  active_record
      create    db/migrate/20190624_create_products.rb
      create    app/models/product.rb
      invoke    test_unit
      create      test/models/product_test.rb
      create      test/fixtures/products.yml

rails db:migrate
Then we need some test data..

# test/fixtures/products.yml
# Read about fixtures at http://api.rubyonrails.org/classes/ActiveRecord/FixtureSet.html
one:
  name: Cheap Widget
  sku: CW123
  price: 9.99

two:
  name: Expensive Widget
  sku: EW456
  price: 99.99

..and of course a test..

# test/product_calculators/product_discount_calculator_test.rb
require 'test_helper'

class PriceCalculatorTest < ActiveSupport::TestCase
  test 'cheap widget is reduced by 15%' do
    widget = products(:one)
    calculated_price = PriceCalculator.get_price(widget)

    assert_equal((widget.price * 0.85), calculated_price, 'cheap widget should have a 15% price reduction')
  end
end

..which when run with the following command..
rails test test/product_calculators/product_discount_calculator_test.rb
..resulting in a NameError: uninitialized constant

E

Error:
PriceCalculatorTest#test_cheap_widget_is_reduced_by_15%:
NameError: uninitialized constant PriceCalculatorTest::PriceCalculator
    test/product_calculators/product_discount_calculator_test.rb:6:in `block in <class:PriceCalculatorTest>'

..as expected - our PriceCalculator doesn’t exist yet and Rails is expecting the definition to be somewhere in autoload_paths and having not found a definition, fails with an error expecting the it to be within the calling code, in this case, within our test.

Very helpful but not very organised.

Let’s create a simple implementation and see if Rails can find it automatically:

# app/calculators/price_calculator.rb
# Simple calculator that takes a Product and returns a price with a 15% discount
class PriceCalculator
  def self.get_price(product)
    product.price * 0.85
  end
end

Now, when we re-run our rails test we get the same error - our new class isn’t found. The reason is noted at the end of section 5 in the aforementioned documention:

autoload_paths is computed and cached during the initialization process. The application needs to be restarted to reflect any changes in the directory structure.

We can check this is the reason by viewing the cached paths with the following command:
bin/rails r 'puts ActiveSupport::Dependencies.autoload_paths'

Running via Spring preloader in process 21407
/MyApp/app/channels
/MyApp/app/controllers
/MyApp/app/controllers/concerns
/MyApp/app/jobs
/MyApp/app/mailers
/MyApp/app/models
/MyApp/app/models/concerns
../ruby/gems/2.6.0/gems/activestorage-5.2.3/app/assets
../ruby/gems/2.6.0/gems/activestorage-5.2.3/app/controllers
../ruby/gems/2.6.0/gems/activestorage-5.2.3/app/controllers/concerns
../ruby/gems/2.6.0/gems/activestorage-5.2.3/app/javascript
../ruby/gems/2.6.0/gems/activestorage-5.2.3/app/jobs
../ruby/gems/2.6.0/gems/activestorage-5.2.3/app/models
/MyApp/test/mailers/previews

Yep, no mention of our new app/calculators directory. In development, Spring caches the directories so restarting our application after creating directories is akin to saying “restart Spring”..
spring stop
bin/rails r 'puts ActiveSupport::Dependencies.autoload_paths'

Running via Spring preloader in process 21462
/MyApp/app/calculators     <-- ** new directory **
/MyApp/app/channels
/MyApp/app/controllers
/MyApp/app/controllers/concerns
/MyApp/app/jobs
/MyApp/app/mailers
/MyApp/app/models
/MyApp/app/models/concerns
../ruby/gems/2.6.0/gems/activestorage-5.2.3/app/assets
../ruby/gems/2.6.0/gems/activestorage-5.2.3/app/controllers
../ruby/gems/2.6.0/gems/activestorage-5.2.3/app/controllers/concerns
../ruby/gems/2.6.0/gems/activestorage-5.2.3/app/javascript
../ruby/gems/2.6.0/gems/activestorage-5.2.3/app/jobs
../ruby/gems/2.6.0/gems/activestorage-5.2.3/app/models
/MyApp/test/mailers/previews

Our new directory is found, so if we re-run our test..
rails test test/product_calculators/product_discount_calculator_test.rb

# Running:

.

Finished in 0.028790s, 34.7343 runs/s, 34.7343 assertions/s.

Excellent. No require or require_relative required, and no change to autoload_paths or our application’s configuration either.

Adding namespaces

Organising our classes in sub-directories of app/ is fine for smaller applications buy say we now wanted the following directory structure: app/calculators/pricing/price_calculator.rb
mkdir app/calculators/pricing && mv app/calculators/*.rb app/calculators/pricing/
spring stop && rails test test/product_calculators/product_discount_calculator_test.rb

# Running:

E

Error:
PriceCalculatorTest#test_cheap_widget_is_reduced_by_15%:
NameError: uninitialized constant PriceCalculatorTest::PriceCalculator
    test/product_calculators/product_discount_calculator_test.rb:6:in `block in <class:PriceCalculatorTest>'

If we inspect the list of autoload_paths again, we see our top-level app/calculators path so Rails needs a bit more of a hint to find our calculator..
We need to add a Pricing:: namespace to consumers of our moved calculator class..

# test/product_calculators/product_discount_calculator_test.rb
require 'test_helper'

class PriceCalculatorTest < ActiveSupport::TestCase
  test 'cheap widget is reduced by 15%' do
    widget = products(:one)
    calculated_price = Pricing::PriceCalculator.get_price(widget)

    assert_equal((widget.price * 0.85), calculated_price, 'cheap widget should have a 15% price reduction')
  end
end

rails test test/product_calculators/product_discount_calculator_test.rb

# Running:

E

Error:
PriceCalculatorTest#test_cheap_widget_is_reduced_by_15%:
LoadError: Unable to autoload constant Pricing::PriceCalculator, expected /MyApp/app/calculators/pricing/price_calculator.rb to define it
    test/product_calculators/product_discount_calculator_test.rb:6:in `block in <class:PriceCalculatorTest>'

Rails is looking in the correct place, but now we need to put the class in a matching namespace..

# app/calculators/pricing/price_calculator.rb
# Simple calculator that takes a Product and returns a price with a 15% discount
module Pricing
  class PriceCalculator
    def self.get_price(product)
      product.price * 0.85
    end
  end
end

If we run our test again now it passes:
rails test test/product_calculators/product_discount_calculator_test.rb

# Running:

.

Finished in 0.023214s, 43.0775 runs/s, 43.0775 assertions/s.
1 runs, 1 assertions, 0 failures, 0 errors, 0 skips

Fine, but why don’t we need to start the namespace with Calculators::? - well, this too is a Rails convention and explained in the documention:

Rails has a collection of directories similar to $LOAD_PATH in which to look up post.rb. That collection is called autoload_paths and by default it contains:

  • All subdirectories of app in the application and engines present at boot time. For example, app/controllers. They do not need to be the default ones, any custom directories like app/workers belong automatically to autoload_paths.

Therefore our custom app/calculators directory is automatically added.. as we saw earlier!

How about 2 namespaces? ()

Can we add 1 more namespace to our current PriceCalculator (2 namespaces / modules)?

# test/product_calculators/product_discount_calculator_test.rb
require 'test_helper'

class PriceCalculatorTest < ActiveSupport::TestCase
  test 'cheap widget is reduced by 15%' do
    widget = products(:one)
    calculated_price = Pricing::WidgetPricing::PriceCalculator.get_price(widget)

    assert_equal((widget.price * 0.85), calculated_price, 'cheap widget should have a 15% price reduction')
  end
end
# app/calculators/pricing/price_calculator.rb
# Simple calculator that takes a Product and returns a price with a 15% discount
module Pricing
  module WidgetPricing
    class PriceCalculator
      def self.get_price(product)
        product.price * 0.85
      end
    end
  end
end

then move it into a directory with the same namespace structure:
mkdir app/calculators/pricing/widget_pricing && mv app/calculators/pricing/*.rb app/calculators/pricing/widget_pricing/
test again..
spring stop && rails test test/product_calculators/product_discount_calculator_test.rb
and..

# Running:

.

Finished in 0.378896s, 2.6392 runs/s, 10.5570 assertions/s.
1 runs, 4 assertions, 0 failures, 0 errors, 0 skips

3 Namespaces? ()

2 works, now let’s try 3 namespaces. Same as above, first we alter the test, then the file, and then the file’s location..

# test/product_calculators/product_discount_calculator_test.rb
require 'test_helper'

class PriceCalculatorTest < ActiveSupport::TestCase
  test 'cheap widget is reduced by 15%' do
    widget = products(:one)
    calculated_price = Pricing::WidgetPricing::TwoThousand::PriceCalculator.get_price(widget)

    assert_equal((widget.price * 0.85), calculated_price, 'cheap widget should have a 15% price reduction')
  end
end
# app/calculators/pricing/two_thousand/price_calculator.rb
# Simple calculator that takes a Product and returns a price with a 15% discount
module Pricing
  module WidgetPricing
    module TwoThousand
      class PriceCalculator
        def self.get_price(product)
          product.price * 0.85
        end
      end
    end
  end
end

spring stop && rails test test/product_calculators/product_discount_calculator_test.rb

# Running:

.

Finished in 0.047112s, 21.2260 runs/s, 21.2260 assertions/s.
1 runs, 1 assertions, 0 failures, 0 errors, 0 skips

4 Namespaces ()

That also works - how about 4 namespaces?

# test/product_calculators/product_discount_calculator_test.rb
require 'test_helper'

class PriceCalculatorTest < ActiveSupport::TestCase
  test 'cheap widget is reduced by 15%' do
    widget = products(:one)
    calculated_price = Pricing::WidgetPricing::TwoThousand::Nineteen::PriceCalculator.get_price(widget)

    assert_equal((widget.price * 0.85), calculated_price, 'cheap widget should have a 15% price reduction')
  end
end
# app/calculators/pricing/two_thousand/nineteen/price_calculator.rb
# Simple calculator that takes a Product and returns a price with a 15% discount
module Pricing
  module WidgetPricing
    module TwoThousand
      module Nineteen
        class PriceCalculator
          def self.get_price(product)
            product.price * 0.85
          end
        end
      end
    end
  end
end

.. and it still works - all without touching the Rails app configuration files.

5 Namespaces ()

Hmm.. so what happens if we move our PricingCalculator from app/calculators/... to app/lib/...:

spring stop && rails test test/product_calculators/product_discount_calculator_test.rb

# Running:

.

Finished in 0.052509s, 19.0444 runs/s, 19.0444 assertions/s.
1 runs, 1 assertions, 0 failures, 0 errors, 0 skips

And if we were to add a calculators sub-directory under app/lib/ like so:

# app/lib/calculators/pricing/widget_pricing/tow-thousand/nineteen/price_calculator.rb
# Simple calculator that takes a Product and returns a price with a 15% discount
module Calculators
  module Pricing
    module WidgetPricing
      module TwoThousand
        module Nineteen
          class PriceCalculator
            def self.get_price(product)
              product.price * 0.85
            end
          end
        end
      end
    end
  end
end

Why would we do this? Well, our use of this class within our code (and our tests) would be a little more self-documenting:

# test/product_calculators/product_discount_calculator_test.rb
require 'test_helper'

class PriceCalculatorTest < ActiveSupport::TestCase
  test 'cheap widget is reduced by 15%' do
    widget = products(:one)
    calculated_price = Calculators::Pricing::WidgetPricing::TwoThousand::Nineteen::PriceCalculator.get_price(widget)

    assert_equal((widget.price * 0.85), calculated_price, 'cheap widget should have a 15% price reduction')
  end
end

That’s clearer

Summary

Rails convention over configuration lets us organise our classes into nested namespace / module hierarchies as long as they match the directory or folder structure, starting one subdirectory in from app/ - e.g. app/lib.

There are 3 things to remember:
  1. Our application's code should live in /app/.. - personally I recommend /app/lib/{sub_directory}/.. so we can use {sub_directory} as the start of our namespacing: e.g. app/lib/calculators/... -> Calculators::...
  2. When moving files around or creating new directories we need to restart Spring (spring stop)
  3. To reference a class that isn't in a namespace (or in the root namespace), such as a Rails Model in its default location of app/models, we can use the namespace separator (::) as a prefix - e.g.: ::Product
  4. - this is useful if we also have a class of the same name in a different namespace.

Foreign Keys to custom Primary Key caveats in Ruby on Rails

0 Comments

Image Credit: S Egger, 2007

The Ruby on Rails model convention of automatically including primary keys named id and foreign keys named {primary_key_model}_id works well for the vast majority of models, but what if the object we’re modelling already has a unique numerical property? It would make sense to use this property instead of id, and Rails allows custom primary keys but there are some gotchas..

Let’s walk through an example using Books and Chapters

First of all we need a new Rails app which we’ll call Bookshelf: (I’m using Rails 5.2.1 here)
rails new bookshelf
cd bookshelf
rails db:create

Now we can create our Book model - but instead of id we’ll specify the International Standard Book Number (ISBN) as the primary key 1
rails generate model Book isbn:integer title:string genre:string

We can’t specify the primary key change with the command-line generator so we need to edit the migration Rails created for us, from this:

# db/migrate/20180825101955_create_books.rb
class CreateBooks < ActiveRecord::Migration[5.2]
  def change
    create_table :books do |t|
      t.integer :isbn
      t.string :title
      t.string :genre

      t.timestamps
    end
  end
end

to this:

# db/migrate/20180825101955_create_books.rb
class CreateBooks < ActiveRecord::Migration[5.2]
  def change
    create_table :books, id: false, primary_key: :isbn do |t|
      t.primary_key :isbn
      t.string :title
      t.string :genre

      t.timestamps
    end
  end
end

Then we can apply it and check the primary key is as expected by creating a Book:
rails db:migrate

== 20180825101955 CreateBooks: migrating ======================================
-- create_table(:books, {:id=>false, :primary_key=>:isbn})
   -> 0.0024s
== 20180825101955 CreateBooks: migrated (0.0025s) =============================

rails c
irb(main):001:0>Book.create(isbn: 9780099518471, title: 'Brave New World', genre: 'Science Fiction')
irb(main):002:0>Book.find(9780099518471)

  Book Load (0.3ms)  SELECT  "books".* FROM "books" WHERE "books"."isbn" = ? LIMIT ?  [["isbn", 9780099518471], ["LIMIT", 1]]
=> #<Book isbn: 9780099518471, title: "Brave New World", genre: "Science Fiction", created_at: "2018-08-25 10:36:37", updated_at: "2018-08-25 10:36:37">

So far so good. Now let’s create our Chapter association: (type quit to exit rails c)
rails generate model Chapter title:string no:integer book:references

If we run the Chapter migration now it will apply, but if we try to create a Chapter and reference our Book we get a rollback transaction error:
rails db:migrate
rails c
irb(main):001:0>Chapter.create!(title: "Chapter 1", no: 1, book: Book.find(9780099518471))
  Book Load (0.5ms)  SELECT  "books".* FROM "books" WHERE "books"."isbn" = ? LIMIT ?  [["isbn", 9780099518471], ["LIMIT", 1]]
   (0.1ms)  begin transaction
  Chapter Create (0.7ms)  INSERT INTO "chapters" ("title", "no", "book_id", "created_at", "updated_at") VALUES (?, ?, ?, ?, ?)  [["title", "Chapter 1"], ["no", 1], ["book_id", 9780099518471], ["created_at", "2018-08-25 11:26:44.984648"], ["updated_at", "2018-08-25 11:26:44.984648"]]
   (0.0ms)  rollback transaction
ActiveRecord::StatementInvalid: SQLite3::SQLException: foreign key mismatch - "chapters" referencing "books": INSERT INTO "chapters" ("title", "no", "book_id", "created_at", "updated_at") VALUES (?, ?, ?, ?, ?)
There's no way to create a Chapter because the foreign key constraint is trying to enforce values against books.id which doesn't exist so we will never get a foreign key match

There are a couple of manual steps which are pointed to in the foreign key documentation to get this working properly:

If the column names can not be derived from the table names, you can use the :column and :primary_key options

Let’s throw in a little complication at this point - we don’t want our foreign key column name to be chapters.book_id - we’d like it to be chapters.book_isbn as that’s in keeping with the Books model and looks better in the database too.

We need to make a few changes to our Chapters migration, from this:

# db/migrate/20180825111413_create_chapters.rb
class CreateChapters < ActiveRecord::Migration[5.2]
  def change
    create_table :chapters do |t|
      t.string :title
      t.integer :no
      t.references :book, foreign_key: true

      t.timestamps
    end
  end
end

to this:

# db/migrate/20180825111413_create_chapters.rb
class CreateChapters < ActiveRecord::Migration[5.2]
  def change
    create_table :chapters do |t|
      t.string :title
      t.integer :no
      t.references :book_isbn, references: :books, null: false # creates 'book_isbn_id'

      t.timestamps
    end

    rename_column :chapters, :book_isbn_id, :book_isbn
    add_foreign_key :chapters, :books, column: 'book_isbn', primary_key: 'isbn'
  end
end

As you can see we’re taking the default naming convention for the foreign key references column, renaming it, and then adding the foreign key constraint with the column: and primary_key: options as per the documentation.

Now we can apply our migration:
rails db:migrate

== 20180825111413 CreateChapters: migrating ===================================
-- create_table(:chapters)
   -> 0.0032s
-- rename_column(:chapters, :book_isbn_id, :book_isbn)
   -> 0.0328s
-- add_foreign_key(:chapters, :books, {:column=>"book_isbn", :primary_key=>"isbn"})
   -> 0.0000s
== 20180825111413 CreateChapters: migrated (0.0366s) ==========================
If we try to create a Chapter and reference our Book now we get a missing attribute error:
rails c
irb(main):001:0>Chapter.create!(title: "Chapter 1", no: 1, book: Book.find(9780099518471))
  Book Load (0.2ms)  SELECT  "books".* FROM "books" WHERE "books"."isbn" = ? LIMIT ?  [["isbn", 9780099518471], ["LIMIT", 1]]
ActiveModel::MissingAttributeError: can't write unknown attribute `book_id`
We still can't create a Chapter because Rails doesn't know we're using custom column and primary key names - the error message is telling us there's a problem with our Chapter model

The final peice is to tell Rails about our custom primary key and while we’re editing the model, we can add our associations at the same time.
Change the Book from this:

# app/models/book.rb
class Book < ApplicationRecord
end

to this:

# app/models/book.rb
class Book < ApplicationRecord
  self.primary_key = 'isbn'
  has_many :chapters, primary_key: 'isbn', foreign_key: 'book_isbn'
end

And then change the Chapter from this:

# app/models/chapter.rb
class Chapter < ApplicationRecord
  belongs_to :book
end

to this:

# app/models/chapter.rb
class Chapter < ApplicationRecord
  belongs_to :book, foreign_key: 'book_isbn'
end

Now let’s try adding a Chapter or two..

rails c
irb(main):001:0>Chapter.create!(title: "Chapter 1", no: 1, book: Book.find(9780099518471))

  Book Load (0.5ms)  SELECT  "books".* FROM "books" WHERE "books"."isbn" = ? LIMIT ?  [["isbn", 9780099518471], ["LIMIT", 1]]
   (0.1ms)  begin transaction
  Chapter Create (1.8ms)  INSERT INTO "chapters" ("title", "no", "book_isbn", "created_at", "updated_at") VALUES (?, ?, ?, ?, ?)  [["title", "Chapter 1"], ["no", 1], ["book_isbn", 9780099518471], ["created_at", "2018-08-27 07:29:24.381030"], ["updated_at", "2018-08-27 07:29:24.381030"]]
   (4.0ms)  commit transaction
=> #<Chapter id: 1, title: "Chapter 1", no: 1, book_isbn: 9780099518471, created_at: "2018-08-27 07:29:24", updated_at: "2018-08-27 07:29:24">

Success!

And because we’ve added the associations, we can also do:
irb(main):001:0>bravenewworld = Book.find(9780099518471)
irb(main):002:0>bravenewworld.chapters.create!(title: "Chapter 2", no: 2)

And to show the association works the other way..
irb(main):003:0>chapter2 = Chapter.where(book_isbn: 9780099518471, no: 2).first
irb(main):004:0>chapter2.book



  1. In reality ISBN probably isn’t a good choice for a primary key as the specification states the 10-digit versions can start with a zero, and leading zeros are dropped by integer datatypes. 

Setting up Ubuntu on Digital Ocean

0 Comments

Digital Ocean* is an internet hosting service that makes it trivial to spin up virtual servers called Droplets. While the base Ubuntu image Droplets are configured for the job, there are a couple of extra steps I take with new Ubuntu Droplets that I’m documenting here as much for my own future reference as to elicit your feedback

SSH Keys

I’ll typically create a new SSH key pair for a each Droplet. Digital Ocean’s community guide is comprehensive if you need a refresher or haven’t done it before.

ssh-keygen -t rsa -b 4096 -C "[email protected]"

Droplet creation

After logging into Digital Ocean (or signing up - use this link for an extra $10 USD credit), we click Create Droplet and follow the wizard.

Here are the typical base settings I use:

Distributions Ubuntu, latest LTM, x64
Size As per requirements (usually the smallest $5/mo)
Datacenter region Best to pick the one closest to the majority of our expected userbase. That might only be us
Select additional options As per requirements (usually just Monitoring)
Add your SSH Keys Click New SSH Key and paste in the public part of the SSH Key generated earlier
Finalise and create As per requirements

Then we click Create and wait less than a minute while Digital Ocean performs its magic

Configuration

For convenience we can give our new Droplet a friendly SSH name by adding the following to our local ~/.ssh/config file (I usually make this the same as the Droplet’s name):

# ~/.ssh/config
...
Host {droplet-name}
    User root
    HostName {droplet-ip-address}
    IdentityFile "~/.ssh/{our-new-ssh-private-key}"
...

Now we can SSH into our new Ubuntu Droplet with
ssh {droplet-name}

Set the timezone

dpkg-reconfigure tzdata

Ensure all packages are up-to-date

apt-get update; apt-get -y upgrade; apt-get -y clean

Configure automatic security patches (documentation here and here)

apt-get -y install unattended-upgrades; dpkg-reconfigure unattended-upgrades
Follow the prompts and accept the defaults.

Lock SSH to keys-only

Edit sshd_config to prevent root SSH login with a password - change PermitRootLogin from yes to without-password like so:

# /etc/ssh/sshd_config
...
# Authentication:
LoginGraceTime 120
PermitRootLogin without-password
StrictModes yes
...

And finally, reboot the Droplet to ensure our settings are loaded, current and it comes back to us before we start installing or configuring our application stack of choice..
reboot

Is there anything you’d add to this list of initial Ubuntu server setup steps? - Please let us know in the comments!