Q&A: Paddy Power Betfair on DevOps

In the complex and creative world of IT, the concept of business structure is often overlooked. Instead of a well-polished, strategically ordered set-up of developers and back-end operators we often envision a dark room filled with buttons and rows of hunched over techies.

Over the last three years the DevOps concept sprang into motion, revolutionising the way tech-driven departments communicate with each other.

Steven Armstrong, head of PPB’s DevOps team, is somewhat of an expert on the subject, having written extensively on it. Talking to EGR Technology, he is keen to spread the message that DevOps is the future of the egaming industry.

EGR Technology: Can you explain DevOps as a concept?

Steven Armstrong (SA): I think there’s a common misconception that it means automation, but DevOps is really about organising your business to collaborate better. Generally teams are divided out into different silos in organisations so you’ll have an operations department, a development department, and testing department who won’t talk to each other daily.

What routinely happens over time is that they just throw stuff over the wall to each other and they don’t talk or collaborate.

What DevOps is really doing is trying to break down those boundaries between the teams by reorganising those siloed teams into project teams that carry out pieces of work.

So when you’re going to put together a new project, you will assemble a team made up of people from those previously siloed disciplines, who will work closely together to build a new piece of infrastructure or help put a new product out to market.

There’s lots of different concepts that cross-functional teams can benefit from to speed up that implementation process, such as continuous integration and delivery.

EGR Technology: Where did the concept come from?

SA: The main origin of it is just treating code like a production line you’d have in a factory. So the deployment pipeline is really developers building code with people then building infrastructure and networking that goes into a pipeline and that delivers an end product, which could be something like a web application or mobile application to end users.

EGR Technology: How is your team at Paddy Power Betfair structured?

SA: Generally when we were building our i2 project which is based on OpenStack, we set up new cross-functional teams that worked together to build the platform.

What we were really doing is creating T-shaped teams where members have a deep dive knowledge (depth of the T) on one particular area and then you share that knowledge with others like developers and network engineers to widen the breadth of knowledge (breadth of the T).

The hope is really through sharing that knowledge the breadth of the T is widened for each team member so they become more well-rounded and can understand multiple software engineering disciplines such as development, infrastructure, networking etc.

EGR Technology: How long have you been working under this structure?

SA: We were generally doing this when we started the pilot project in 2015, when we were building out the i2 (next generation infrastructure) platform. That was at the heritage Betfair organisation before the merger, which restructured the way we were working.

That was very key to the initiative we did with i2 because the way OpenStack is structured is it encompasses infrastructure, storage, networking and development disciplines so you really need a multi-disciplined team to make it successful.

EGR Technology: Do you find it difficult to recruit new team members?

SA: Sometimes it’s quite difficult to recruit the right people but generally what we’ve done is ask the good people we hire if they know anyone first. We also have specific technical recruiters that we go through.

Our HR department basically has a group of specialist recruiters that recruit for DevOps, Linux etc, so we use them whenever possible. Generally, if you hire someone good they’ll bring good people with them. That’s been quite successful for us.

EGR Technology: How big is your team?

SA: We have a core team of six people that look after the core infrastructure platform on OpenStack. That team does upgrades and maintenance to make sure the Open-Stack API layer for the infrastructure is in a workable state.

We automate everything in the whole data centre using Ansible. We have teams helping developers on board to the platform in each country, including Romania, Ireland, the UK and Portugal.

There are around six team members in each country that help development teams fill in the self-service config files that allow them to deploy their applications, while the core team looks after the core infrastructure.

Because we’ve automated everything we can, we don’t need huge teams to operate it.

EGR Technology: Has this concept taken off across the entire industry?

SA: I think in London specifically, where we have attended DevOps conferences, a lot of people think it just means automation but it really means restructuring the business and departments to work in this new way.

I think they think that because they are doing automation, they’re doing DevOps, but when you talk to them generally they haven’t done a restructure and they are still working in silos.

With public cloud I think they are skipping out the blockers that they’re hindered with internal IT processes, but they are not fixing the issues by doing this. It is often a quick fix. So developers will just go to AWS and then skip out actually having to talk to the test team or the operations team.

I don’t think a lot of companies are structured and set up correctly to be successful. You really need CTO and COO buy-in for this to work and reorganise your company in the best way to succeed, and that’s often hard because you’ve got managers that have been in place for a long time.

They generally like their niche spaces and see the restructure as losing control of a particular department. I think it’s a new way of thinking that is required.

EGR Technology: How is it going to revolutionise online gambling?

SA: I think it’s taking a step back and really looking at what the industry is doing, setting up a business and streamlining pieces of it to reduce waste. When I say waste, I mean using a ticketing system to talk to another department, for example.

That doesn’t make sense because a lot of people, in banks for instance, can be blocked for months from doing certain things just because they have different modes of management and approvals they need to go through.

DevOps is changing the industry in that you have to get products out to market as quickly as possible, so if you have a very slow process to do that, then you just won’t be competitive and will lose market share. If competitors can get products out faster than you then they are going to win the share of the market. That’s how business is changing.

You set out your teams in this way, so that you can optimise the throughput.

EGR Technology: In what ways does DevOps affect core areas of the business?

SA: You’ve got more collaborative teams and people enjoy work more. For instance, you’re prototyping new ideas and you’re not blocked internally from getting those ideas out as quickly as possible.

Really it’s changing the way that people think because you’re working with multidiscipline teams so you have different views. A developer might not consider something that an operations person thinks of because they think differently.

That generally raises the bar on people’s abilities and they build new skills.

EGR Technology: Tell me about your award winning i2 OpenStack project.

SA: This project is split across two data centres. Generally that’s for disaster recovery purposes, so we’ll deploy our applications in both data centres.

Also, we set up a full self-service framework where developers can control infrastructure and networking programmatically.

This enables developers to consume the core infrastructure, networking and the storage very quickly and not have to wait on ticketing systems. We allow development teams to fill in these self-service fi les and then they get the infrastructure that they can deploy their applications on.

This has allowed us to speed up the time it takes to put a product to market, so we’re doing around 1,000 deployments a day on the platform which means every time a developer will check in code, a new deployment occurs and then that goes through a deployment pipeline all the way to production.

It goes through all the different test phases and then if it’s successful, it’s a released candidate that can be deployed to production.

EGR Technology: What other industries have picked up on DevOps as an innovative business structure?

SA: I think DevOps is becoming more widespread. Other organisations that have done it include Netflix and the other pioneers such as Google and Facebook. A lot of the market leaders are doing this and people have looked deeper into the way they’re doing it and openly presented its benefits. A lot of other businesses are trying to replicate that model so that they can deliver as fast as these companies and be as successful as them.

EGR Technology: Do other operators outsource this work?

SA: There are a lot of DevOps consultancies that come in and help the business to write automation and do this sort of thing. I think the benefit of doing it in-house is you have skin in the game. A lot of the time consultancies could potentially promise a lot and not deliver it.

It is hard to get the right people to do this. You really do have to have the in-house engineering capabilities to have success when going it alone.

If you do it in-house people generally have more domain knowledge because they work in the business, as opposed to a consultancy that will have a pre-written framework that isn’t business specific that they will try to fit into the particular business.

That might not suit the needs of the business as much. It is useful if you don’t have that in-house knowledge and you need help to get started on it.

There are good consultancies out there that can help too.

EGR Technology: What’s the future of DevOps, where is it going?

SA: It’s just giving people the choice on how they deploy tools such as Kubernetes, which allows you to distribute workloads across private and public clouds via Google’s framework, and are being heavily adopted.

I think generally people will want to place workloads across public and private clouds and that’s the end goal. You’ll have tools that analyse the cost of running on each cloud and can pick based on cost. You could have Microsoft Azure cloud, Google cloud or AWS and then run OpenStack in the private cloud.

Then there are customer facing web applications that could run in private clouds such as Google or AWS and keep your data in private cloud. Kubernetes will allow you to orchestrate that across different clouds without locking into each vendor’s bespoke API.

EGR Technology: What are the biggest threats to this system?

SA: In public clouds you really don’t know where your data is going. That’s the main concern for a lot of people. Generally with some regulatory requirements they need to know where the data is, so that’s another concern. Security is the main consideration for people.