tombola careers
tombola careers

#tombolalife

Blog | Tombola Careers

tombola blog
Jul
3rd

Want organisational change? Tear down the fences

Posted by tombola ops
Tear down the fences
Any business larger than a start-up will be composed of groups of people, each performing a distinct job. These groups may be called teams, divisions, sections or task groups. The words don’t matter. The important thing is that they are pockets of human beings, gathered together to do a certain thing.

Introduction

An American poet once wrote that ‘Good fences make good neighbours‘. In the corporate environment, the exact opposite is true. The boundaries we construct between teams actually damage relationships.

Us and Them

Any business larger than a start-up will be composed of groups of people, each performing a distinct job. These groups may be called teams, divisions, sections or task groups. The words don’t matter. The important thing is that they are pockets of human beings, gathered together to do a certain thing.

These groups are positive in many ways. They provide employees with support and a sense of belonging. They concentrate expertise and knowledge. They are the logical building blocks of corporate hierarchy. Unfortunately, they have a drawback. When you create a team, you build a fence. Everything inside that fence is us, everything outside is them.

War at the borders

Teams can find themselves in a state of cold war. The spaces between them are like hostile borders. The lines of communication are tenuous and strained. Why is this? More often than not, it is the inevitable result of misaligned goals. Team A wants something that Team B cannot provide without breaking rules. Division C needs something quickly that Division D has to make slowly. Section 1 needs approval from Section 2, who are understaffed and stressed. The groups are driven to conflict by circumstance and mismatched business needs. Inevitably, there are casualties. People get hurt. Grievances, written warnings, hushed departures. Employees who spend their careers secretly at war.

The root of this problem is understanding, or rather the lack of it. Relationships between team members are stronger than non-team members. This makes perfect sense. When you work in a group, you have more time to bond. You suffer through the same meetings. You swap personal histories. You construct a shared vision. You spend your time in the same building, sometimes the same room. Because of these shared experiences, you understand each other. Any minor disagreements are seen through the lens of your shared history. You assume goodwill. This is not true of non-group members, who might need (because of their role) to obstruct you from time to time, to get in way, to say No. When goals collide, the understanding isn’t there. This is the soil in which problems grow.

Possible solutions

Friction between groups is a difficult nut to crack. Tribalism seems to be baked into us. But you are certainly not powerless. You can alter your own attitudes, if you really want to. Here are three suggestions:

  1. Police yourself. Stop being negative about other teams, or members of other teams. No more jokes. No more type casting. No more gossip. Try to change the underlying patterns. You may find this difficult initially. It’s hard to put down old ideas.
  2. Proximity. Try shared projects, shadowing, secondments, anything that gets you in the same room as the other group. Try to break it down into individuals. It is harder to be annoyed at someone when you know the names of their kids.
  3. Compassion. When you really, really need something, and that person is holding you up againtry to step back from the brink. Take a breath. Perhaps that person is unwell, upset, run off their feet. We cannot know the hidden lives of others.

Conclusion

Improve inter-group relationships and you will improve the business. This is obvious, so obvious that it does not really deserve an article. And yet it often gets ignored. In the rush of deadlines, targets, goals and sprints, we can all fall into the trap of us and them. We can all participate in the grumbling narrative of office dispute. But you can make a difference. You can try to change. We can’t tear down all the fences just yet. But maybe we can make a few doors.

See also

https://www.management-issues.com/opinion/7236/changing-a-culture-starts-with-changing-behaviours/

https://culturesloanreview.mit.edu/article/from-shooks-prize-winning-article-change-behavior-if-you-want-to-change-culture/

The line of poetry in the introduction is taken from ‘Mending Wall’ by Robert Frost, published in the collection ‘North of Boston’ in 1914.

(Published on LinkedIn 03/07/2018)

read more
Jun
29th

tombola @ codegarden18

Posted by james conway
This year we were lucky enough to be invited to umbraco codegarden to tell the unique story of how we used umbraco CMS to deliver content to our players during I’m a Celebrity… Get me out of here!

Umbraco is an excellent CMS and as a project/company are very similar to tombola in terms of culture – users and community first always.

This year we were lucky enough to be invited to umbraco codegarden to tell the unique story of how we used umbraco CMS to deliver content to our players during I’m a Celebrity… Get me out of here!

Umbraco is an excellent CMS and as a project/company are very similar to tombola in terms of culture – users and community first always.

You can check out our talk below

read more
Jun
27th

Alexa Skill – from Hackathon to Production

Posted by phil atkinson
I’ve worked for many companies over the years (people say I’m older than I look) that just don’t bother with innovation, at least not to the extent that tombola does it. Recently I got a chance to take part in my first ever Hackathon. For those that don’t know what it is, you get to sit your projects to one side for a few days, and experiment with tech and ideas that you think would be interesting to try out and maybe work with people that you may not usually work with.

Hackathon

I’ve worked for many companies over the years (people say I’m older than I look) that just don’t bother with innovation, at least not to the extent that tombola does it. Recently I got a chance to take part in my first ever Hackathon. For those that don’t know what it is, you get to sit your projects to one side for a few days, and experiment with tech and ideas that you think would be interesting to try out and maybe work with people that you may not usually work with.

Here at tombola we love all things AWS and my team mates love the range of echo devices, they’ve been raving about them since their launch. I’m sure everyone on my team, myself included, now has at least one in their home. I thought it would be cool to build something with Alexa for my Hackathon so I gave it a shot.

My team had recently just finished our new social feed news stream and consumed it on our website https://www.tombolaarcade.co.uk/socialfeed so I thought it would be good if Alexa could read this out to the players.

the hotspot

Building a skill

When you start to build a skill you will quickly realise there are loads of useful documentation (https://developer.amazon.com/docs/custom-skills/steps-to-build-a-custom-skill.html) to walk you through the process, and an Alexa skill is simply an AWS Lambda function designed to convert spoken intents into API calls.  This is now even easier by using one of the many Alexa skills SDKs https://github.com/alexa/alexa-skills-kit-sdk-for-nodejs simplifying the amount of code you need to write.
The information feeds for the skill’s functionality just comes from API calls, which depending on your architecture you may already have existing, like we did here at tombola.
AWS even cater for developers to test their skill without the need to actually buy an echo device, you can interact with your skill using the Alexa simulator:

alexa simulator

Once you’re happy, publish it and wait until AWS approve it. If you are lucky enough Amazon will send you some free swag. :)

alexa bag

 

Production

When we demonstrated the prototype to our CEO he immediately recognised its potential, a new method of interacting with the players and a new way of delivering social content at no cost to the players.  It wasn’t long before a real production project was kicked off, with an enhanced functionality specification, and now we have released the “tombola news” Alexa skill: https://www.amazon.co.uk/tombola-Ltd-news/dp/B07D18H75B/ref=sr_1_1?s=digital-skills&ie=UTF8&qid=1530092410&sr=1-1&keywords=tombola+news

Hackathons are a good way to let developers experiment and see what they come up with. You never know, your next hackathon idea may make it to production too.

read more
Jun
14th

Tech Brown Bag – Runtime Configuration Management

Posted by phil atkinson
Recently I took part in a tombola guild to investigate managing our application and system configuration. We have a lot of static configuration baked into our applications and various tools which means a config change is a deployment change. With our rapid growth of applications we need a better solution that lets us manage our configuration at runtime.

Fortunately it seems AWS already have the answer, SSM Parameter Store..

Recently I took part in a tombola guild to investigate managing our application and system configuration.  We have a lot of static configuration baked into our applications and various tools which means a config change is a deployment change.  With our rapid growth of applications we need a better solution that lets us manage our configuration at runtime.

Fortunately it seems AWS already have the answer, SSM Parameter Store..

SSM parameter store

Once the guild had a working prototype it was time to share the knowledge with the rest of the tech team.
Here’s the full presentation in all it’s glory: Click Here To View.

read more
Feb
2nd

We’ve open sourced a tool – Introducing ECS Monitor

Posted by marc costello
At tombola, we use Amazon’s ECS (Elastic Container Service) to schedule and orchestrate many of our services. While this has many benefits to traditional application hosting, it also brings new challenges – Not least of all monitoring.

Picture this scenario: You have an application running on a cluster of machines, those machines are completely redundant; and so are the application processes themselves. Instances of your application process can be ran on any machine in the cluster, you could have multiple processes of the application running on the very same machine – Don’t care. What we DO care about, is making sure everything is healthy, and when it isn’t we want to know about it – Monitoring, right?

I’m not talking about application monitoring as in: logging, transaction performance, stack traces etc… I’m instead referring to container and infrastructure monitoring.

At tombola, we use Amazon’s ECS (Elastic Container Service) to schedule and orchestrate many of our services. While this has many benefits to traditional application hosting, it also brings new challenges – Not least of all monitoring.

Picture this scenario: You have an application running on a cluster of machines, those machines are completely redundant; and so are the application processes themselves. Instances of your application process can be ran on any machine in the cluster, you could have multiple processes of the application running on the very same machine – Don’t care. What we DO care about, is making sure everything is healthy, and when it isn’t we want to know about it – Monitoring, right?

I’m not talking about application monitoring as in: logging, transaction performance, stack traces etc… I’m instead referring to container and infrastructure monitoring.

We made a tool for that.

An AWS ECS monitoring tool, would you believe. It’s open source and developed on github. It essentially offers service aggregation over the AWS API’s which allows us to view metrics, logs and events which happen on the ECS scheduler and in containers as quickly as possible.

Why did we make a tool?

Currently, Amazon don’t offer anything like this. ECS itself is nothing more than a scheduler (and an agent running on EC2 instances) and leave much of the tooling up to you. There are many commercial options which promise great insight into your Docker applications but we weren’t ready for another piece of software which needs to run on our agents, and really; AWS has all the raw data we need, it’s just not in the best format to be used as a dashboard.

What is the ECS monitor good for?

The monitor was built with dashboarding in mind, we want to put this on the TV for the whole team to see, so there were a several things we wanted to monitor which we found valuable:

Aggregation

As I mentioned earlier one of the primary benefits is aggregation. Amazon currently don’t have a place where you can view service events all together in one place. So this was one feature we wanted, an aggregated service event stream.

service events

Task Churn

The ECS scheduler always tries to keep the number of running tasks to the desired count. It will do this forever, regardless of whether or not the containers are unhealthy and being killed frequently; ECS will keep creating more in an attempt to reach and maintain that desired count. When this happens, it can be pretty tricky to track down. It’s all too easy to look in the AWS console and see a healthy service, when in reality it’s spamming tasks.

Task churn is a component in the monitor which detects this. The monitor listens for task start events on each service and applies a time buffer to the count, if it’s over a threshold an alert is raised and the service is flagged as being sick – It should be investigated.

task churn alert

Agent visualisation

Being able to visualise your tasks running on a cluster is useful, particularly if you have specific placement strategies and constraints set on your services. We’ve found it valuable to track the spread of tasks across a cluster. Another scenario where this is useful is when an instance becomes sick, you can quite easily detect which one is playing up and kill it. Like all the features in the ECS monitor this updates with the latest state as often as AWS allows. For example, when the scheduler registers a new task, that task will be displayed on this cluster visualisation indicating a pending state without having to refresh or take any action.

agents view

Logs

If you are using CloudWatch to hold container logs, the ECS monitor lets you view and query log streams. It doesn’t offer anything over the AWS console, only it’s quicker to access, as 90% of the time you’re only going to be diving in there for debugging purposes.

Metrics

Cluster and service metrics are another thing we want to keep an eye on. So we have put these in too, with cluster metrics over time represented on charts and realtime counts (as close to realtime as AWS allows anyway). The monitor requests this data as frequently as the Amazon API allows.

metrics

Deployments

When a new deployment is detected that is also raised as an alert and displayed in the monitor too.

deployments alert

Coming up.

You can check out the github issues to see upcoming features, but if I were to list a few of the big hitters:

  • Alerts – If we leave the monitor on overnight, it would be nice to come in on the morning and quickly see if any alerts were raised overnight which might indicate a problem.
  • Compact mode – When there are many services, the current layout gets out of hand. We want to offer an alternative view – A ‘not so detailed’/‘essentials only’ view but one that puts more emphasis on alerts rather than showing all of the details. Much better suited to accounts with many services. All in aid of making this a good tool for dashboards.
  • EC2 data – Data and metrics on the EC2 agents themselves, stats such as CPU credits remaining/usage; network in/out and more. We’ve found it’s generally not that important to be watching these metrics constantly, but it’s another great piece of data to diagnose the health of a cluster.

Open source and taking contributions

We find this very useful internally, hopefully others may find a use for it and would like to contribute to it’s evolution, issues and pull requests are always welcome!

read more