Blog | Tombola Careers

< back

Jul
26th

Docker in Production – A Year (and a bit) Later

It’s been around 2 years since tombola started to investigate the use and potential benefits of container technology, specifically Docker. At times since then progress has been slow and frustrating, but that’s no shock with new technologies.

We’ve been running containers in a production environment for just over a year now and I wanted to look back at how this move has affected the business.

N.B. This isn’t going to be overly technical, just a general review.

The team started by converting two API services to run in containers. The work not only required rewriting these services from .Net to NodeJS, but also to creat a new solution to build and deploy them. A combination of Team City (which we already used) and Terraform (more on that choice later) was what was settled upon.

To run and manage our containers we chose to use Amazon’s Elastic Container Service. This provides an orchestration tool to manage a cluster of instances and a private container image repository. At the time it was the easiest solution that met our needs.

So, why Terraform? Two words – Placement strategies. We needed to set placement strategies for the containers within the ECS cluster. These needed to be set upon deployment of the application. These strategies could be set through the AWS console, but not using Cloudformation at the time, which was what we originally wanted to use. However, Terraform did support this, and so the choice was pretty simple. Setting the placement strategies was an important part to maintaining a highly available service.

Since the migration of those 2 initial services the teams have added almost a dozen more. All of these services are deployed using the same pipeline mentioned earlier, which has simplified and sped up the process of getting new services live.

The move has simplified some of the server management in Operations. At least 8 of the services would previously have been built to run on their own stacks of 2-4 instances, leaving us with at least 16 instances running. Instead we have 3 in our main cluster, with room to spare. This saves us money, time and effort.

Since the services have gone live they have been rock solid. Even patching of the live cluster instances is non-disruptive.

Has it all been sunshine and rainbows? Not exactly. There have been technical challenges along the way, but nothing that hasn’t been overcome (eventually). You can’t really argue against the success of this work.

Where do we go from here?

There’s always work to be done and there’s no exception here. Recently AWS released their managed Kubernetes service, which warrants investigating. Is it a better solution for us than ECS? We’ll have to find out.

Monitoring has always been a bit of a difficulty, but we will soon be implementing a new monitoring solution across all of our infrastructure that will handle container monitoring far better than our current solution.

Security is a constant battle and everyone using containers can step up here. Visiting Dockercon this year opened my eyes to just how much more work is required from a security point of view across the community. For example, seeing a major distro’s official docker image being exploited quickly was a little scary. Container security definitely presents a new set of challenges.

So there is plenty more work to be done and maybe next time I’ll be writing about a migration to Kubernetes, who knows?

Jul
3rd

Want organisational change? Tear down the fences

Posted by tombola ops

Any business larger than a start-up will be composed of groups of people, each performing a distinct job. These groups may be called teams, divisions, sections or task groups. The words don’t matter. The important thing is that they are pockets of human beings, gathered together to do a certain thing.

Introduction

An American poet once wrote that ‘Good fences make good neighbours‘. In the corporate environment, the exact opposite is true. The boundaries we construct between teams actually damage relationships.

Us and Them

These groups are positive in many ways. They provide employees with support and a sense of belonging. They concentrate expertise and knowledge. They are the logical building blocks of corporate hierarchy. Unfortunately, they have a drawback. When you create a team, you build a fence. Everything inside that fence is us, everything outside is them.

War at the borders

Teams can find themselves in a state of cold war. The spaces between them are like hostile borders. The lines of communication are tenuous and strained. Why is this? More often than not, it is the inevitable result of misaligned goals. Team A wants something that Team B cannot provide without breaking rules. Division C needs something quickly that Division D has to make slowly. Section 1 needs approval from Section 2, who are understaffed and stressed. The groups are driven to conflict by circumstance and mismatched business needs. Inevitably, there are casualties. People get hurt. Grievances, written warnings, hushed departures. Employees who spend their careers secretly at war.

The root of this problem is understanding, or rather the lack of it. Relationships between team members are stronger than non-team members. This makes perfect sense. When you work in a group, you have more time to bond. You suffer through the same meetings. You swap personal histories. You construct a shared vision. You spend your time in the same building, sometimes the same room. Because of these shared experiences, you understand each other. Any minor disagreements are seen through the lens of your shared history. You assume goodwill. This is not true of non-group members, who might need (because of their role) to obstruct you from time to time, to get in way, to say No. When goals collide, the understanding isn’t there. This is the soil in which problems grow.

Possible solutions

Friction between groups is a difficult nut to crack. Tribalism seems to be baked into us. But you are certainly not powerless. You can alter your own attitudes, if you really want to. Here are three suggestions:

Police yourself. Stop being negative about other teams, or members of other teams. No more jokes. No more type casting. No more gossip. Try to change the underlying patterns. You may find this difficult initially. It’s hard to put down old ideas.

Proximity. Try shared projects, shadowing, secondments, anything that gets you in the same room as the other group. Try to break it down into individuals. It is harder to be annoyed at someone when you know the names of their kids.

Compassion. When you really, really need something, and that person is holding you up again, try to step back from the brink. Take a breath. Perhaps that person is unwell, upset, run off their feet. We cannot know the hidden lives of others.

Conclusion

Improve inter-group relationships and you will improve the business. This is obvious, so obvious that it does not really deserve an article. And yet it often gets ignored. In the rush of deadlines, targets, goals and sprints, we can all fall into the trap of us and them. We can all participate in the grumbling narrative of office dispute. But you can make a difference. You can try to change. We can’t tear down all the fences just yet. But maybe we can make a few doors.