Categories
Programming

How to ignore PostgreSQL tables when restoring a database

At my current workplace, we use a copy of the backup database which is created by pg_dump to restore in our development machines. Since the database has a lot of audit tables which adds up a lot of space, I needed an easier way to ignore these tables when restoring the database.

Apparently pg_restore can generate a list of tables in the dump file archive. The advantage is that we can use the same list to specify what content we need to restore.

pg_restore options

Here are the pg_restore options we are going to use for this.

-l --list
List the contents of the archive. The output of this operation can be used as input to the -L option. Note that if filtering switches such as -n or -t are used with -l, they will restrict the items listed.

-L list-file --use-list=list-file
Restore only those archive elements that are listed in list-file, and restore them in the order they appear in the file. Note that if filtering switches such as -n or -t are used with -L, they will further restrict the items restored.

list-file is normally created by editing the output of a previous -l operation. Lines can be moved or removed, and can also be commented out by placing a semicolon (;) at the start of the line. See below for examples.

Steps

Let’s generate the list while ignoreing the content for the tables we don’t need. Note that I use an inverse grep statement to ignore the content of the tables where the table names that ending with _audits and create the output in restore.list file.

pg_restore -l PostgreSQL.sql | grep -v 'TABLE DATA public .*_audits' > restore.list

Then simply use -L to tell pg_restore to use the restore.list file.

pg_restore -U #{USERNAME} -O -L restore.list -v -c -d mydb_development PostgreSQL.sql

Ref
pg restore options: https://www.postgresql.org/docs/10/static/app-pgrestore.html

Categories
Programming

Vagrant – Running Rails server in the background

Let’s say when you boot up or restart your virtual machine, you want to run the Rails server in the background for your Rails application.

It is pretty easy. First, let’s create a function for the script block. Put following code on the top of the Vagrant file.

def start_rails_server
<<-SHELL
cd /vagrant
# start rails server as a daemon
# to kill: sudo kill `cat tmp/pids/server.pid`
rails s -d
SHELL
end

Now in the Vagrantfile configuration block, use the script function and set it to run always. 

Vagrant.configure("2") do |config|
...
config.vm.provision "shell", inline: start_rails_server, run: 'always'
...
end
Categories
Programming

Setting up Ruby SDK for RubyMine with Vagrant

This article is about how you can configure to use the Ruby SDK in your Vagrant image with RubyMine.

For this, I have used the  joshfng/railsbox Vagrant image because I was using that image for a Rails project. 

First you have to find where is your Ruby interpreter located for your project. Let’s assume your project lies in the same folder where you have Vagrantfile. In this case, by default the /vagrant folder in the vagrant image will contain your code.

To find the ruby path, here type `which ruby`. Then it will spit out the location. You have to remember this location. In my it was ‘/home/vagrant/.rbenv/shims/ruby’.

Now, to link to the vagrant ruby interpreter, you have to open the preferences dialogue and then select  Languages & Frameworks > Ruby SDK and Gems.

Then click on + and select remote location as the location and select vagrant.

For the Vagrant path, give the folder where your Vagrantfile is located. For the Ruby interpreter path, give the path you found before.

Click Ok.

Now Rubymine should load up the necessary gems to itself which will take some time. But after that, you should be good to go!

Categories
Programming

Think differently when the current tests fails

This is a lesson learned from work. SO we use Rials in our project at the workplace. SP for a new requirement, we had to have a validation for a particular attribute of a model.   The requirements were specific on where the validation should happen. Say for example for what controller actions.

However, the initial idea was tho have the validation at the model so that it will be validated on create and update. The idea was that every controller action that matters the new attribute validation would automatically be validated.

The approach was working great. However, after the existing tests suite has run, the results were about 70% failure. So we looked into the test code and noticed that the attributes set for the models were not following the rule. Therefore the initial thought was to fix the specs and make them pass. So we spent several hours altering the specs and running them to make sure they pass. Then we came across a test that acutally indicated our approach caused an existing functionality to break. Meaning that the rule should not be validated when performing that particular functionality.

So then we thought about a different approach and ended up using contexts in Rails. There we were able to tell the model only to validate the attribute when it is in the given context. As a result, there were only few tests that failed at controller level testing. We quickly fixed them and added specific test scenarios to cover the new requirements.

So lessons learned:

When the code change breaks a lot of existing test, it is most likely because that the app functionality is broken. So think of the bigger picture of the code change and come up with a solution that would impact less part of the application as a whole. 

Categories
Programming

Be careful when using loops in Jasmine specs

Your test automation code is an important tool to evaluate your application.  You must ensure your test code is working/testing as you want it to be. the automated tests give confidence for the developer that the application behaves as expected and things are not broken after they change the code.

Recently, at our QA phase, the QA found a bug that should have been caught from our test automation. Therefore being suspicious I looked into the code.  Interesting enough, the way we used loops along with test blocks were not testing as we wanted. Below is an example of a test block with the bad approach we had. Then I listed down a working approach using loops.

describe('Bad Spec', function () {
 for (var i = 0; i < 3; i++) {
   console.log('Iteration value outside test: ' + i);
   it ('bar', function() {
     console.log('Test is evaluated. Iteration value inside test: ' + i);
     expect(true).toEqual(true)
   });
 }
});

The following is the console output. Notice that the value inside the test block is always the same value.

Iteration value outside test: 0
Iteration value outside test: 1
Iteration value outside test: 2
Test is evaluated. Iteration value inside test: 3
Test is evaluated. Iteration value inside test: 3
Test is evaluated. Iteration value inside test: 3

 

Here a solution is to use a function

describe('Good Spec', function () {
  function goodRunner(itemValue) {
    it ('foo', function() {
      console.log('Test is evaluated. Iteration value inside test: ' + itemValue);
      expect(true).toEqual(true)
    });
  }
  for (var i = 0; i < 3; i++) {
    console.log('Iteration value outside test: ' + i);
    goodRunner(i);
  }
});

Now the output looks as follows.

Iteration value outside test: 0
Iteration value outside test: 1
Iteration value outside test: 2
Test is evaluated. Iteration value inside test: 0
Test is evaluated. Iteration value inside test: 1
Test is evaluated. Iteration value inside test: 2

 

 

 

 

 

 

Categories
Programming

Go Programming Language: An Introductory Tutorial

What’s the Go Programming Language?

Go is a recent language which sits neatly in the middle of the landscape, providing lots of good features and deliberately omitting many bad ones. It compiles fast, runs fast-ish, includes a runtime and garbage collection, has a simple static type system and dynamic interfaces, and an excellent standard library.

Go and OOP

OOP is one of those features that Go deliberately omits. It has no subclassing, and so there are no inheritance diamonds or super calls or virtual methods to trip you up. Still, many of the useful parts of OOP are available in other ways.

*Mixins* are available by embedding structs anonymously, allowing their methods to be called directly on the containing struct (see embedding). Promoting methods in this way is called *forwarding*, and it’s not the same as subclassing: the method will still be invoked on the inner, embedded struct.

Embedding also doesn’t imply polymorphism. While `A` may have a `B`, that doesn’t mean it is a `B` — functions which take a `B` won’t take an `A` instead. For that, we need interfaces, which we’ll encounter briefly later.

Meanwhile, Go takes a strong position on features that can lead to confusion and bugs. It omits OOP idioms such as inheritance and polymorphism, in favor of composition and simple interfaces. It downplays exception handling in favour of explicit errors in return values. There is exactly one correct way to lay out Go code, enforced by the gofmt tool. And so on.

Go is also a great language for writing concurrent programs: programs with many independently running parts. An obvious example is a webserver: Every request runs separately, but requests often need to share resources such as sessions, caches, or notification queues. This means skilled Go programmers need to deal with concurrent access to those resources.

While the Go language has an excellent set of low-level features for handling concurrency, using them directly can become complicated. In many cases, a handful of reusable abstractions over those low-level mechanisms makes life much easier.

In today’s Go programming tutorial, we’re going to look at one such abstraction: A wrapper which can turn any data structure into a transactional service. We’ll use a Fund type as an example – a simple store for our startup’s remaining funding, where we can check the balance and make withdrawals.

In this introduction to programming in Go, we’ll build the service in small steps, making a mess along the way and then cleaning it up again. Along the way, we’ll encounter lots of cool Go features, including:

  • Struct types and methods
  • Unit tests and benchmarks
  • Goroutines and channels
  • Interfaces and dynamic typing

A Simple Fund

Let’s write some code to track our startup’s funding. The fund starts with a given balance, and money can only be withdrawn (we’ll figure out revenue later).

This graphic depicts a simple goroutine example using the Go programming language.

Go is deliberately not an object-oriented language: There are no classes, objects, or inheritance. Instead, we’ll declare a struct type called Fund, with a simple function to create new fund structs, and two public methods.

fund.go

package funding

type Fund struct {
    // balance is unexported (private), because it's lowercase
    balance int
}

// A regular function returning a pointer to a fund
func NewFund(initialBalance int) *Fund {
    // We can return a pointer to a new struct without worrying about
    // whether it's on the stack or heap: Go figures that out for us.
    return &Fund{
        balance: initialBalance,
    }
}

// Methods start with a *receiver*, in this case a Fund pointer
func (f *Fund) Balance() int {
    return f.balance
}

func (f *Fund) Withdraw(amount int) {
    f.balance -= amount
}

Testing with benchmarks

Next we need a way to test Fund. Rather than writing a separate program, we’ll use Go’s testing package, which provides a framework for both unit tests and benchmarks. The simple logic in our Fund isn’t really worth writing unit tests for, but since we’ll be talking a lot about concurrent access to the fund later on, writing a benchmark makes sense.

Benchmarks are like unit tests, but include a loop which runs the same code many times (in our case, fund.Withdraw(1)). This allows the framework to time how long each iteration takes, averaging out transient differences from disk seeks, cache misses, process scheduling, and other unpredictable factors.

The testing framework wants each benchmark to run for at least 1 second (by default). To ensure this, it will call the benchmark multiple times, passing in an increasing “number of iterations” value each time (the b.N field), until the run takes at least a second.

For now, our benchmark will just deposit some money and then withdraw it one dollar at a time.

fund_test.go

package funding

import "testing"

func BenchmarkFund(b *testing.B) {
    // Add as many dollars as we have iterations this run
    fund := NewFund(b.N)

    // Burn through them one at a time until they are all gone
    for i := 0; i < b.N; i++ {
        fund.Withdraw(1)
    }

    if fund.Balance() != 0 {
        b.Error("Balance wasn't zero:", fund.Balance())
    }
}

Now let’s run it:

$ go test -bench . funding
testing: warning: no tests to run
PASS
BenchmarkWithdrawals    2000000000             1.69 ns/op
ok      funding    3.576s

That went well. We ran two billion (!) iterations, and the final check on the balance was correct. We can ignore the “no tests to run” warning, which refers to the unit tests we didn’t write (in later Go programming examples in this tutorial, the warning is snipped out).

Concurrent Access

Now let’s make the benchmark concurrent, to model different users making withdrawals at the same time. To do that, we’ll spawn ten goroutines and have each of them withdraw one tenth of the money.

How would we structure muiltiple concurrent goroutines in the Go language?

Goroutines are the basic building block for concurrency in the Go language. They are green threads – lightweight threads managed by the Go runtime, not by the operating system. This means you can run thousands (or millions) of them without any significant overhead. Goroutines are spawned with the go keyword, and always start with a function (or method call):

// Returns immediately, without waiting for `DoSomething()` to complete
go DoSomething()

Often, we want to spawn off a short one-time function with just a few lines of code. In this case we can use a closure instead of a function name:

go func() {
    // ... do stuff ...
}() // Must be a function *call*, so remember the ()

Once all our goroutines are spawned, we need a way to wait for them to finish. We could build one ourselves using channels, but we haven’t encountered those yet, so that would be skipping ahead.

For now, we can just use the WaitGroup type in Go’s standard library, which exists for this very purpose. We’ll create one (called “wg”) and call wg.Add(1) before spawning each worker, to keep track of how many there are. Then the workers will report back using wg.Done(). Meanwhile in the main goroutine, we can just say wg.Wait() to block until every worker has finished.

Inside the worker goroutines in our next example, we’ll use defer to call wg.Done().

defer takes a function (or method) call and runs it immediately before the current function returns, after everything else is done. This is handy for cleanup:

func() {
    resource.Lock()
    defer resource.Unlock()

    // Do stuff with resource
}()

This way we can easily match the Unlock with its Lock, for readability. More importantly, a deferred function will run even if there is a panic in the main function (something that we might handle via try-finally in other languages).

Lastly, deferred functions will execute in the reverse order to which they were called, meaning we can do nested cleanup nicely (similar to the C idiom of nested gotos and labels, but much neater):

func() {
    db.Connect()
    defer db.Disconnect()

    // If Begin panics, only db.Disconnect() will execute
    transaction.Begin()
    defer transaction.Close()

    // From here on, transaction.Close() will run first,
    // and then db.Disconnect()

    // ...
}()

OK, so with all that said, here’s the new version:

fund_test.go

package funding

import (
    "sync"
    "testing"
)

const WORKERS = 10

func BenchmarkWithdrawals(b *testing.B) {
    // Skip N = 1
    if b.N < WORKERS {
        return
    }

    // Add as many dollars as we have iterations this run
    fund := NewFund(b.N)

    // Casually assume b.N divides cleanly
    dollarsPerFounder := b.N / WORKERS

    // WaitGroup structs don't need to be initialized
    // (their "zero value" is ready to use).
    // So, we just declare one and then use it.
    var wg sync.WaitGroup

    for i := 0; i < WORKERS; i++ {
        // Let the waitgroup know we're adding a goroutine
        wg.Add(1)
        
        // Spawn off a founder worker, as a closure
        go func() {
            // Mark this worker done when the function finishes
            defer wg.Done()

            for i := 0; i < dollarsPerFounder; i++ {
                fund.Withdraw(1)
            }
            
        }() // Remember to call the closure!
    }

    // Wait for all the workers to finish
    wg.Wait()

    if fund.Balance() != 0 {
        b.Error("Balance wasn't zero:", fund.Balance())
    }
}

We can predict what will happen here. The workers will all execute Withdraw on top of each other. Inside it, f.balance -= amount will read the balance, subtract one, and then write it back. But sometimes two or more workers will both read the same balance, and do the same subtraction, and we’ll end up with the wrong total. Right?

$ go test -bench . funding
BenchmarkWithdrawals    2000000000             2.01 ns/op
ok      funding    4.220s

No, it still passes. What happened here?

Remember that goroutines are green threads – they’re managed by the Go runtime, not by the OS. The runtime schedules goroutines across however many OS threads it has available. At the time of writing this Go language tutorial, Go doesn’t try to guess how many OS threads it should use, and if we want more than one, we have to say so. Finally, the current runtime does not preempt goroutines – a goroutine will continue to run until it does something that suggests it’s ready for a break (like interacting with a channel).

All of this means that although our benchmark is now concurrent, it isn’t parallel. Only one of our workers will run at a time, and it will run until it’s done. We can change this by telling Go to use more threads, via the GOMAXPROCS environment variable.

$ GOMAXPROCS=4 go test -bench . funding
BenchmarkWithdrawals-4    --- FAIL: BenchmarkWithdrawals-4
    account_test.go:39: Balance wasn't zero: 4238
ok      funding    0.007s

That’s better. Now we’re obviously losing some of our withdrawals, as we expected.

In this Go programming example, the outcome of multiple parallel goroutines is not favorable.

Make it a server

At this point we have various options. We could add an explicit mutex or read-write lock around the fund. We could use a compare-and-swap with a version number. We could go all out and use a CRDT scheme (perhaps replacing the balance field with lists of transactions for each client, and calculating the balance from those).

But we won’t do any of those things now, because they’re messy or scary or both. Instead, we’ll decide that a fund should be a server. What’s a server? It’s something you talk to. In Go, things talk via channels.

Channels are the basic communication mechanism between goroutines. Values are sent to the channel (with channel <- value), and can be received on the other side (with value = <- channel). Channels are “goroutine safe”, meaning that any number of goroutines can send to and receive from them at the same time.

Buffering

Buffering communication channels can be a performance optimization in certain circumstances, but it should be used with great care (and benchmarking!).

However, there are uses for buffered channels which aren’t directly about communication.

For instance, a common throttling idiom creates a channel with (for example) buffer size `10` and then sends ten tokens into it immediately. Any number of worker goroutines are then spawned, and each receives a token from the channel before starting work, and sends it back afterward. Then, however many workers there are, only ten will ever be working at the same time.

By default, Go channels are unbuffered. This means that sending a value to a channel will block until another goroutine is ready to receive it immediately. Go also supports fixed buffer sizes for channels (using make(chan someType, bufferSize)). However, for normal use, this is usually a bad idea.

Imagine a webserver for our fund, where each request makes a withdrawal. When things are very busy, the FundServer won’t be able to keep up, and requests trying to send to its command channel will start to block and wait. At that point we can enforce a maximum request count in the server, and return a sensible error code (like a 503 Service Unavailable) to clients over that limit. This is the best behavior possible when the server is overloaded.

Adding buffering to our channels would make this behavior less deterministic. We could easily end up with long queues of unprocessed commands based on information the client saw much earlier (and perhaps for requests which had since timed out upstream). The same applies in many other situations, like applying backpressure over TCP when the receiver can’t keep up with the sender.

In any case, for our Go example, we’ll stick with the default unbuffered behavior.

We’ll use a channel to send commands to our FundServer. Every benchmark worker will send commands to the channel, but only the server will receive them.

We could turn our Fund type into a server implementation directly, but that would be messy – we’d be mixing concurrency handling and business logic. Instead, we’ll leave the Fund type exactly as it is, and make FundServer a separate wrapper around it.

Like any server, the wrapper will have a main loop in which it waits for commands, and responds to each in turn. There’s one more detail we need to address here: The type of the commands.

A diagram of the fund being used as the server in this Go programming tutorial.

Pointers

We could have made our commands channel take *pointers* to commands (`chan *TransactionCommand`). Why didn’t we?

Passing pointers between goroutines is risky, because either goroutine might modify it. It’s also often less efficient, because the other goroutine might be running on a different CPU core (meaning more cache invalidation).

Whenever possible, prefer to pass plain values around.

In the next section below, we’ll be sending several different commands, each with its own struct type. We want the server’s Commands channel to accept any of them. In an OOP language we might do this via polymorphism: Have the channel take a superclass, of which the individual command types were subclasses. In Go, we use interfaces instead.

An interface is a set of method signatures. Any type that implements all of those methods can be treated as that interface (without being declared to do so). For our first run, our command structs won’t actually expose any methods, so we’re going to use the empty interface, interface{}. Since it has no requirements, any value (including primitive values like integers) satisfies the empty interface. This isn’t ideal – we only want to accept command structs – but we’ll come back to it later.

For now, let’s get started with the scaffolding for our server:

server.go

package funding

type FundServer struct {
    Commands chan interface{}
    fund Fund
}

func NewFundServer(initialBalance int) *FundServer {
    server := &FundServer{
        // make() creates builtins like channels, maps, and slices
        Commands: make(chan interface{}),
        fund: NewFund(initialBalance),
    }

    // Spawn off the server's main loop immediately
    go server.loop()
    return server
}

func (s *FundServer) loop() {
    // The built-in "range" clause can iterate over channels,
    // amongst other things
    for command := range s.Commands {
    
        // Handle the command
        
    }
}

Now let’s add a couple of struct types for the commands:

type WithdrawCommand struct {
    Amount int
}

type BalanceCommand struct {
    Response chan int
}

The WithdrawCommand just contains the amount to withdraw. There’s no response. The BalanceCommand does have a response, so it includes a channel to send it on. This ensures that responses will always go to the right place, even if our fund later decides to respond out-of-order.

Now we can write the server’s main loop:

func (s *FundServer) loop() {
    for command := range s.Commands {

        // command is just an interface{}, but we can check its real type
        switch command.(type) {

        case WithdrawCommand:
            // And then use a "type assertion" to convert it
            withdrawal := command.(WithdrawCommand)
            s.fund.Withdraw(withdrawal.Amount)

        case BalanceCommand:
            getBalance := command.(BalanceCommand)
            balance := s.fund.Balance()
            getBalance.Response <- balance

        default:
            panic(fmt.Sprintf("Unrecognized command: %v", command))
        }
    }
}

Hmm. That’s sort of ugly. We’re switching on the command type, using type assertions, and possibly crashing. Let’s forge ahead anyway and update the benchmark to use the server.

func BenchmarkWithdrawals(b *testing.B) {
    // ...

    server := NewFundServer(b.N)

    // ...

    // Spawn off the workers
    for i := 0; i < WORKERS; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for i := 0; i < dollarsPerFounder; i++ {
                server.Commands <- WithdrawCommand{ Amount: 1 }
            }
        }()
    }

    // ...

    balanceResponseChan := make(chan int)
    server.Commands <- BalanceCommand{ Response: balanceResponseChan }
    balance := <- balanceResponseChan

    if balance != 0 {
        b.Error("Balance wasn't zero:", balance)
    }
}

That was sort of ugly too, especially when we checked the balance. Never mind. Let’s try it:

$ GOMAXPROCS=4 go test -bench . funding
BenchmarkWithdrawals-4     5000000           465 ns/op
ok      funding    2.822s

Much better, we’re no longer losing withdrawals. But the code is getting hard to read, and there are more serious problems. If we ever issue a BalanceCommand and then forget to read the response, our fund server will block forever trying to send it. Let’s clean things up a bit.

Make it a service

A server is something you talk to. What’s a service? A service is something you talk to with an API. Instead of having client code work with the command channel directly, we’ll make the channel unexported (private) and wrap the available commands up in functions.

type FundServer struct {
    commands chan interface{} // Lowercase name, unexported
    // ...
}

func (s *FundServer) Balance() int {
    responseChan := make(chan int)
    s.commands <- BalanceCommand{ Response: responseChan }
    return <- responseChan
}

func (s *FundServer) Withdraw(amount int) {
    s.commands <- WithdrawCommand{ Amount: amount }
}

Now our benchmark can just say server.Withdraw(1) and balance := server.Balance(), and there’s less chance of accidentally sending it invalid commands or forgetting to read responses.

Here is what using the fund as a service might look like in this sample Go language program.

There’s still a lot of extra boilerplate for the commands, but we’ll come back to that later.

Transactions

Eventually, the money always runs out. Let’s agree that we’ll stop withdrawing when our fund is down to its last ten dollars, and spend that money on a communal pizza to celebrate or commiserate around. Our benchmark will reflect this:

// Spawn off the workers
for i := 0; i < WORKERS; i++ {
    wg.Add(1)
    go func() {
        defer wg.Done()
        for i := 0; i < dollarsPerFounder; i++ {

            // Stop when we're down to pizza money
            if server.Balance() <= 10 {
                break
            }
            server.Withdraw(1)
        }
    }()
}

// ...

balance := server.Balance()
if balance != 10 {
    b.Error("Balance wasn't ten dollars:", balance)
}

This time we really can predict the result.

$ GOMAXPROCS=4 go test -bench . funding
BenchmarkWithdrawals-4    --- FAIL: BenchmarkWithdrawals-4
    fund_test.go:43: Balance wasn't ten dollars: 6
ok      funding    0.009s

We’re back where we started – several workers can read the balance at once, and then all update it. To deal with this we could add some logic in the fund itself, like a minimumBalance property, or add another command called WithdrawIfOverXDollars. These are both terrible ideas. Our agreement is amongst ourselves, not a property of the fund. We should keep it in application logic.

What we really need is transactions, in the same sense as database transactions. Since our service executes only one command at a time, this is super easy. We’ll add a Transact command which contains a callback (a closure). The server will execute that callback inside its own goroutine, passing in the raw Fund. The callback can then safely do whatever it likes with the Fund.

Semaphores and errors

In this next example we’re doing two small things wrong.

First, we’re using a `Done` channel as a semaphore to let calling code know when its transaction has finished. That’s fine, but why is the channel type `bool`? We’ll only ever send `true` into it to mean “done” (what would sending `false` even mean?). What we really want is a single-state value (a value that has no value?). In Go, we can do this using the empty struct type: `struct{}`. This also has the advantage of using less memory. In the example we’ll stick with `bool` so as not to look too scary.

Second, our transaction callback isn’t returning anything. As we’ll see in a moment, we can get values out of the callback into calling code using scope tricks. However, transactions in a real system would presumably fail sometimes, so the Go convention would be to have the transaction return an `error` (and then check whether it was `nil` in calling code).

We’re not doing that either for now, since we don’t have any errors to generate.

// Typedef the callback for readability
type Transactor func(fund *Fund)

// Add a new command type with a callback and a semaphore channel
type TransactionCommand struct {
    Transactor Transactor
    Done chan bool
}

// ...

// Wrap it up neatly in an API method, like the other commands
func (s *FundServer) Transact(transactor Transactor) {
    command := TransactionCommand{
        Transactor: transactor,
        Done: make(chan bool),
    }
    s.commands <- command
    <- command.Done
}

// ...

func (s *FundServer) loop() {
    for command := range s.commands {
        switch command.(type) {
        // ...

        case TransactionCommand:
            transaction := command.(TransactionCommand)
            transaction.Transactor(s.fund)
            transaction.Done <- true

        // ...
        }
    }
}

Our transaction callbacks don’t directly return anything, but the Go language makes it easy to get values out of a closure directly, so we’ll do that in the benchmark to set the pizzaTime flag when money runs low:

pizzaTime := false
for i := 0; i < dollarsPerFounder; i++ {

    server.Transact(func(fund *Fund) {
        if fund.Balance() <= 10 {
            // Set it in the outside scope
            pizzaTime = true
            return
        }
        fund.Withdraw(1)
    })

    if pizzaTime {
        break
    }
}

And check that it works:

$ GOMAXPROCS=4 go test -bench . funding
BenchmarkWithdrawals-4     5000000           775 ns/op
ok      funding    4.637s

Nothing But transactions

You may have spotted an opportunity to clean things up some more now. Since we have a generic Transact command, we don’t need WithdrawCommand or BalanceCommand anymore. We’ll rewrite them in terms of transactions:

func (s *FundServer) Balance() int {
    var balance int
    s.Transact(func(f *Fund) {
        balance = f.Balance()
    })
    return balance
}

func (s *FundServer) Withdraw(amount int) {
    s.Transact(func (f *Fund) {
        f.Withdraw(amount)
    })
}

Now the only command the server takes is TransactionCommand, so we can remove the whole interface{} mess in its implementation, and have it accept only transaction commands:

type FundServer struct {
    commands chan TransactionCommand
    fund *Fund
}

func (s *FundServer) loop() {
    for transaction := range s.commands {
        // Now we don't need any type-switch mess
        transaction.Transactor(s.fund)
        transaction.Done <- true
    }
}

Much better.

There’s a final step we could take here. Apart from its convenience functions for Balance and Withdraw, the service implementation is no longer tied to Fund. Instead of managing a Fund, it could manage an interface{} and be used to wrap anything. However, each transaction callback would then have to convert the interface{} back to a real value:

type Transactor func(interface{})

server.Transact(func(managedValue interface{}) {
    fund := managedValue.(*Fund)
    // Do stuff with fund ...
})

This is ugly and error-prone. What we really want is compile-time generics, so we can “template” out a server for a particular type (like *Fund).

Unfortunately, Go doesn’t support generics – yet. It’s expected to arrive eventually, once someone figures out some sensible syntax and semantics for it. In the meantime, careful interface design often removes the need for generics, and when they don’t we can get by with type assertions (which are checked at runtime).

So we’re done, right?

Yes.

Well, okay, no.

For instance:

  • A panic in a transaction will kill the whole service.
  • There are no timeouts. A transaction that never returns will block the service forever.
  • If our Fund grows some new fields and a transaction crashes halfway through updating them, we’ll have inconsistent state.
  • Transactions are able to leak the managed Fund object, which isn’t good.
  • There’s no reasonable way to do transactions across multiple funds (like withdrawing from one and depositing in another). We can’t just nest our transactions because it would allow deadlocks.
  • Running a transaction asynchronously now requires a new goroutine and a lot of messing around. Relatedly, we probably want to be able to read the most recent Fund state from elsewhere while a long-running transaction is in progress.

In the next Go programming tutorial, we’ll look at some ways to address these issues.

This article originally appeared on Toptal.

Categories
Programming

Trying out Ansible on local mode

So you might be interested running ansible to configure your local machine. In this article, I’m going to show you the minimum steps that need to install Vim  using Ansible on an Ubuntu host machine.

1. Install latest Ansible version from apt

$ sudo apt-get install software-properties-common
$ sudo apt-add-repository ppa:ansible/ansible
$ sudo apt-get update
$ sudo apt-get install ansible

2. Create a folder (let’s call this “ansible”) under home directory

3. Create a file called ‘inventory‘ under the new folder and put following content in the file

localhost   ansible_connection=local

4. Create a file called ‘vim.yml‘ under the new folder and put following content in the vim.yml file

---
- hosts: localhost
 tasks:
 - name: Install vim
 apt:
 name: vim
 update_cache: yes
 sudo: yes

5. Open terminal inside the folder and type

ansible-playbook -K -i inventory vim.yml

-K To ask for password

References:

  • Intro to Playbooks, http://docs.ansible.com/ansible/playbooks_intro.html#intro-to-playbooks
  • Running ansible playbook in localhost, http://ansible.pickle.io/post/86598332429/running-ansible-playbook-in-localhost
Categories
Programming

Trailing whitespaces: what you gonna do about it?

What is a trailing whitespace?

Trailing whitespace is any spaces or tabs after the last non-whitespace character on the line until the newline.

Background

Few months ago, it was raised that PR submitted code tend to have trailing whitespace changes in code diff along with other “real” code changes. This whitespace change is a diff where the change was removing one or more trailing whitespaces from the existing code. We then noticed that the editors we were using were automatically removing trailing whitespaces that were already existed in the code when the file get saved.
The agreed outcome was:

  • Disable this automatic feature
  • If must put removal of trailing whitespace changes to its own commit

Current Situation

Then it appeared that trailing whitespaces are slowly leaking into the code with new commits. Possibly mainly due to disabling the automatic trailing whitespace removal feature in the editors. See below example on how a new trailing whitespace could end up in a plain text file (e.g. .rb, .coffee). Let’s say the editor does not remove trailing whitespaces automatically on file save.

  var foo;
  # when you press ENTER after the (;), you will have two spaces to keep the indentation due to the editor’s behavior. The indentation reference will be the line it was earlier on. So following line has unnecessary trailing whitespaces as you freely moved to type var bar
  
  var bar;

Since you can’t see whitespaces (because of the nature of character), your code looks clean. But is it? Then why would editors worry about it? It seems like something to concern.
See this article on what Emacs offers to handle trailing whitespaces: http://www.gnu.org/software/emacs/manual/html_node/emacs/Useless-Whitespace.html

What could possibly go wrong having some few trailing whitespaces?

  • Jumping to END of the lines will put the cursor after the trailing spaces in the editor. But you didn’t see that you would end up there 🙁
  • Line wrapping in the editor would behave weirdly because of extra spaces (yeah they are also characters!)
  • String literals that span multiple lines can make the output look incorrect
  • Increase the file size unnecessary. Not a big deal, but still why?
  • The parser has to skip an extra character (or hundreds) depending on how many spaces you left out. (Only if the parser could talk, we would know how it feels…)

what we can do?

  • Have a Git hook to stop committing with trailing whitespaces
  • Have your editor show trailing whitespaces mode, so you see what you are committing 😉
  • Think Bold: Remove all existing trailing whitespaces in your code and re-enable automatic trailing whitespace removal feature in your editor.

References/Interesting Thoughts

Categories
Programming

Having a newline character at end of file

Background/problem/question:
When you create a newfile, and push it to Github, in code diff view, it will say no newline at end of file. Sometimes text editors would also add an EOF newline, if they noticed that the file being edited does not end in a newline, when they are saved (usually it is configured that way by default).

Reasoning:
In POSIX, a line is a sequence of zero or more non-newline characters plus a terminating <newline> character. Therefore some Unix tools would expect text files to end with a newline character, especially when they process text line by line.
For example ‘wc’ will expect every line to end with a newline.

test.txt:

first line
second line

Then…

$ wc -l test.txt
1 test.txt

We only get one line.

Conclusion:
This is only a concern. The modern platforms and applications we use should be ok for a text file not to end with a newline. However as long as we use the tools that would expect each line to end with a newline, we might not see all the results.

References/Resources:

Categories
Blog Programming

Use different conditions to test methods

Let’s say that you have a function that takes an HTML <ul> structure and check items.

For simplicity, our method would check if each item in the list has a link pointing to type of resource (A).

So here can write a test that takes a list with items containing resource A links. Here we expect the function to return true.

But this does not guarantee that the function is working as there is another outcome from the function which false.  So we have to test this by using a list having no item pointing to resource A.

Then we have to think about the situation where we find both type of links that contains non-resource A and resource A links and expect the function to return false.

The importance of this that you test different scenarios. This increases the code reliability and your understanding of the code.