Business as usual: how to stop drowning and learn to swim | Jonathan Stott | #LeadDevLondon


Hello, it’s amazing to be here. This is my third year at the Lead Dev conference
in London. And it’s even more amazing that this is my
first time presenting at a tech conference, and I’ve got the opportunity to do that here. So, today, I’m going to tell you a story about
how my team and I evolved the way that we handled our business-as-usual workload. The story is split into eight parts. If you’re all sitting comfortably, I will
begin. Part one is the one with the problem. So what is business as usual – BAU? There is a whole pile of things that it can
be, and here are just a few of those things that tell me what it is to me. So things like bug fixing, customer support,
infrastructure maintenance, manual processes, legacy code or legendary code, as some people
call it! The database administration. But there are some other things that it can
be as well. It’s not interesting, so quite often you’re
being asked to do things you’re not interested in. It is not interesting at all. It’s the a distraction from the fun stuff. It’s beneath me. You’re possibly a senior or a lead developer
and being asked to run some basic processes and to not write code, or something like that. Keeps the lights on. It makes the money come into our business,
and it means that we are able to have your salaries paid. Finally, it’s possibly urgent, and hopefully
not for anybody, but someone might die if you don’t do this stuff. So hopefully for you, it doesn’t feel like
you’re trying to single-handedly push a double-decker bus. I saw this in Manchester a while back. I’m not sure what he was trying to do! [Laughter]. So why is business as usual a challenge? So, for me, these are all things that I have
experienced, you might resonate with some of these as well. This is a word that was mentioned a couple
of times yesterday. It can overwhelm. You can feel like there’s no way to see a
way beyond the business as usual. Very frequently, it’s overlooked. So, you don’t allocate enough time or enough
brain power to this business as usual because you want to do the other stuff. And it takes your time away. You need to really focus on doing this otherwise
you’ll get distracted by the interesting stuff. As I said said when it was beneath me earlier,
it’s a poor use of skills. You need to make sure that the people doing
the business as usual are engaged enough with their skills. But also importantly, it can be a cause of
stress. So, you can be really having to do a whole
pile of extra work just to keep on top of your BAU workload and do the other stuff that
you need to do. As a result of that, as a potential, it could
increase your turnover, so you might have a higher rate of people leaving your team
than you would otherwise expect. And worse-case scenario, it can feel insurmountable,
and you can’t see a way beyond the problem that you’re solving. How do I know? Why am I talking to you about this? As Meri said, I work for the Royal Pharmaceutical
Society. I’ve been there for about five years. I started as a senior developer, and now I
have moved up to a technical architect, and I’m responsible for two teams. In that time, I’ve been working on business
as usual and I’ve experienced all the things that that means to us. So some context about what we will be the
RPS, do. We produce drug information, so if you go
into your doctor’s surgery in the UK and other places around the world, you will find your
doctors or nurses, or pharmacists consulting our reference information. So the books here, the BNF, are in every doctor’s
surgery in this country, and they are checking things like drug doses and side effects, and
interactions. Those are really important things for them
to be sure about. We also published this information online. We don’t just print books, and we also have
some apps and APIs, and other digital formats as well. So this is a big part of our team’s time. We have to make sure that this content is
correct, and support our editorial teams who are writing it. BAU is a large part of our team’s time. We usually spend around about 20 per cent,
sometimes 100 per cent of our time, on BAU, so it is a really big piece of what we do. We’ve got a whole pile of systems and services
for support. Some of those are old. We’ve got one in the basement. I’m sure other people have got old systems
in the basement they want to get rid of but they can’t. We also have regular processes, so we have
a monthly content release process. We have regular big milestones, so the BNF
is printed twice a year and we are starting that for the next edition now. Because of the problem domain we are working
in, quality and precision are super important to us. So critical issues need to be dealt with very,
very quickly so that we don’t have incorrect information out there and possibly bad decisions
being made for patients. But also, we have been inundated at times. There was a time when we could not start a
project for twelve months because we had too much BAU to do and we had to get a way to
work around that. Even today I get involved in this even though
I’m the technical architect. We’ve tried a whole pile of things, and I’m
going to show you the things that we’ve tried today. Spoiler alert! I’m not going to give you one single solution,
Eiffel. So, you will have to pick and choose what
I talk about and maybe something works for you, and you might have other ideas, so see
what works, and then you can come and talk to me later in speaker hours. Part two is the one with the kiwi. And she’s come along! I’m going to pop her down here. She’s going to watch the rest of the talk. Why have we got a kiwi? A weird thing to have? If it was a token that sat on someone’s desk
and she was the marker that said you could come to that person and ask them about problems
and they would be dealing with our BAU. This is the simplest approach that we took. It was the approach that was there when I
started five years ago and the idea was that the kiwi would sit on someone’s desk and then
move to the next desk, and the next desk around in a rota system. As a result of that, everyone got to experience
the kiwi, the BAU work, and it was a great way for them to learn stuff and for people
to be able to share knowledge around a team. The thing that is really important to remember
and one of the things I had trouble with when I was trying to do this was that you had to
support the person doing this. They can’t just go away and work it all out
themselves, especially if your documentation isn’t good. I’m sure everyone’s is perfect! But they need to have the support from other
team members, and that has to be available to them. One thing we learned is you should never plan
non-BAU tasks for the person who is doing this role. If you try to plan stuff, then they will end
up doing that rather than the BAU, however much you try to stop them, so if they have
free time, which hopefully they will, then anything extra that you can give them is a
bonus and shouldn’t be seen as something you plan ahead of time. So you also have to ensure that you’re fair. So I remember a case where the person on the
desk next to me went away for two weeks and missed their kiwi spot, and then the kiwi
came to my desk and then it just carried on around, so they just missed out. You have to make sure that everyone takes
their fair share of this process. But because of the sometimes lack of support
we had, it kind of became a feared thing. You have to make sure that you provide that
support so that if people have trouble, then they can get help when they need it. Especially when you’re new. So, part 3: the one with the project that
will not start. So there are probably times when you feel
like you’re drowning in your BAU, and you can’t see a way to do anything else but this,
and sometimes you probably got more than you can even do with the number of people in your
team. So we had to go through a process where we
said no to a whole pile of stuff which meant we had to have a really rigorous triage process. So the way that we did that was to look at
the stuff that is important, the stuff that was urgent, and not just in a casual way but
really, really dig into the things that you need to do and determine what is really important
and what is really urgent. When you have done that, do it again and make
sure that you’re absolutely sure, because the more often that you look at things, the
more likely you are to find things that you don’t actually need to do. Another way is to do a risk assessment of
the work. If you don’t do this, what is the impact of
that not being done? How likely is it to happen? Don’t get distracted doing things that are
only a problem once in a blue moon. Only do the stuff that’s really likely and
the stuff that has high risk and high impact. We also ring-fenced someone in our team to
do this. And the way that we did this was to ring-fence
just enough people to, or just enough one person, actually, for us, and that wasn’t
quite enough to do all the BAU that we needed to do. So that kind of imposes an artificial ceiling
of the amount of work that’s possible for the team to take on, and it means this people
kind of have to – it means that people have to think more deeply about what is urgent
rather than implicitly having a rate limit on your team. There were occasions when we had to flex a
little bit, but there’s got to be some flexibility otherwise people would be really annoyed about
things that have to be done and aren’t done. You might be wondering what the difference
is here between the kiwi – she’s still there – and just having one person ring-fenced in
a team. This approach was that there’s no rota, and
that person is kind of dedicated to doing the BAU, or rather the rota is a longer period
of time, so they might stay doing BAU for two, three, or six months, maybe. Another approach in this situation, and one
we tried which works for us, was to spend some money. So we contracted some of our work out. Actually, we discovered it’s very hard to
contract out BAU. Which mainly because the documentation wasn’t
good enough, and they still needed the support that – they still needed to be supported by
people in our team. So the things that worked for us when we were
looking at what we could contract out were things like automation of processes, or maybe
fixing some of our long-standing problems. Bear in mind, though, your contractors should
not do all the interesting work. It’s important that you keep your knowledge
in your team and don’t let the contractors take it away from you. And make sure that your team are doing the
fun and the really cool stuff that is providing the value to your business. So there’s a challenge here with stakeholders. We obviously have a lot of people around our
business who care about the stuff that we do, and the way that we approached that was
to look for some common aims – so try to work out what the value is for doing this and,
in our case, the common aim is that we both wanted to start this big project. If you kind of look at it from that way rather
than bottom up rather than top down, then it’s a lot easier for people to see the value
of controlling the BAU work. This worked for us. We were able to start our project, and almost
exactly a year ago today, we released our new on line publication platform, and it’s
been absolutely great since then. There’s one coda, though, to this, and that
is to consider the morale of the people or person in your team who are ring-fenced into
this. They may feel like they’re missing out on
all the interesting stuff, so you need really to think about maybe providing a little bit
of variety for them so that they don’t feel like they’re doing all the drudge work. In retrospect, we should have done more about
this at the time so it’s important to think about if you’re doing this up front, and,
if you’re not, start. So, part 4: the one where everyone leaves. Sad face. So, this is a prepare-for-the-worst scenario. Imagine you’re in a situation where everyone
in your team but you leaves. How are you going to manage? Who is going to run your processes? What happens if you leave and the rest of
your team have to do the stuff that only you know? So, those are quite scary situations to think
of, but actually, just by doing the things I will talk about here, that can be applied
across everything, and I will come back to that a little bit later. So my advice is to think about the three “ations”:
documentation, simplification, and automation. They kind of say what they say on the tin,
but I will just talk a little bit about each of these in turn. So, documentation: literally write everything
down. No-one likes doing this. But it’s really important, and by involving
everybody in the team in this, it means that people really do have to think about lying
the documentation. For processes, we wrote the process. You write it out first. And then someone else, the next time the process
runs would follow the documentation and amend it and gradually it evolves into something
that works for everybody in the team. Another approach that we take is to create
play books or wayfinders for all of your products, or projects in your code. These are documents that would explain what
the project is, how you would go about setting it up to run it or work on it locally, and
then also some pointers into the code so that, for certain scenarios, you know where to look
to start working out a problem. And the document that we’ve produced literally
has links into the coding in GitHub so we can follow that and start to read through
the code. It gives people that head start they might
not be able to find out about themselves particularly if you’ve got so much code that no-one’s going
to be familiar with all of it. The second part is simplification. I think the maths are right on this, but you
will have to trust me. So this is a case where you should go about
questioning everything you’re doing. Does everything that you do as part of your
BAU have value? And also, just because it’s been done before,
it doesn’t mean that it has to be done again. So when we started to look at this for our
workload, we started to realise that parts, or even whole processes, weren’t actually
of any use. We discovered that some things had been deprecated
and de commissioned but we were still doing some process around that, and that was a complete
waste of time. This is just fundamentally about reducing
the quantity of stuff that your team has to deal with, and it requires a long, hard, deep
look into the stuff. Starting with the documentation is a great
approach. It then means that you can understand what
your processes are, and you can try to work out where the improvements need to go. I remember one situation where I looked at
a process which was about 12 or 16 pages long, and about 50 or 60 steps in it, and isle certain
there is stuff in there that could be simplified. So the third “ation” is automation. So you might want to think about when you
should be automating something. So the naīve approach is if the cost to automate
greater than the time to do it without the automation, then you should think about doing
that, and that is generally the approach that you take. But make sure that you consider the long-term,
so processes will stick around for longer than you expect, so take that into account
when you’re working out how much time you will save by automating something. Also, it’s not all about time-saving, it’s
about improving the quality, because, by having fewer human processes that you need to run,
you will have fewer errors, and then there’s fewer chances that you have to go and rerun
something because you made a mistake or someone made a mistake earlier in the process. It’s also about making these manual processes
run themselves. You see, I used a perfect tool for this. We have a number of our processes automated
in our CI tool, Jenkins, and we gradually add to that over time. You might be really short of time. Even automating one or two steps as part of
a process by creating a script, or something similar, is worth doing even if you don’t
automate the whole thing. Each time you run the process, you could automate
one extra step, say, I think magically, without even really thinking about it, you’ve automated
the whole process, which is a great way to approach it. So part 5: the one where the leopard changes
its spots. Unfortunately, I couldn’t find a leopard to
hand, so this is the closest I could get. This is one of our cats. Clawed said hello. – Claude. For us, we had a situation where BAU became
a toxic brand. Nobody in our team likes working on anything
to do with BAU, and it was badly received as a term, even outside our team. So all of our stakeholders understood that
the stuff that they were trying to get us to do wasn’t being done, and then they just
completely switched off. So we had a rebrand ing process for our BAU,
and we changed its name to “support and maintenance”. Actually, this name better reflects what it
is because BAU actually for a lot of the time for us was business as unusual rather than
business as usual. And this was an excellent opportunity for
us to consider our situation and to change our priorities about how we manage this work. So you need to think about how you actually
address the problem. Just changing the time or the logo of something
doesn’t convey a change of approach. So, one way to consider this is to think about
how you might go through a corporate rebranding process. Now, we’ve just done this at the RPS, so our
new brand was brought in last year, and as part of that, the whole business was involved
in looking at things like our mission, and our goals, and our department goals. What are our values and our tone of voice? All of those aspects can be looked at as part
of how you would deal with your BAU. Tone of voice is a really interesting one. It’s about how you communicate things, and
make sure that you communicate things in a way that other people that you need to talk
to can understand. So part 6 is the one where we fixed some things. In our case, we tried to fix all the things,
and we introduced another, I suppose it’s kind of another “ation” but a little bit more
general for us, was something called “consolidation”. So this was a process where the whole team
was involved in looking at what could be improved. I started off creating a list of ideas, of
things that we could work on as part of this, and then the team were involved in prioritising
that and deciding what they actually wanted to do. A good approach for this is to make the team
its own product owner for its consolidation work, and so that they have the last say in
what is actually done. But you should not neglect your stakeholders
and your colleagues in the rest of your business because the work that you do for consolidation
will have value for them. This is a mix of tasks. So there is a whole variety of things that
we picked up for our consolidation work. These included things like bug-fixing, refactoring,
more work on the three “ations” – documentation, simplification, and automation, and we looked
at the things that allowed the team to learn. Even artificially prioritising some things
that aren’t of high value but it means someone or some people in the team can actually learn
from doing those things. And you can imagine a situation where you
start with something simple and gradually get more complex as you level up in your knowledge. An example of one of the things that we did,
and we’ve just completed this work was to upgrade our code from Scala from 11 to 12. There was a lot involved in that. It is important that you repeat this process. It’s not a once-only thing. We are hoping to have regular consolidation
sprints through the year. Hopefully, one two-week sprint every quarter
which means we spend eight weeks of our time every year on this stuff. For us, that’s a good balance of time, I think. But it depends on where you work and what
the sort of things that you’re doing are as to how much time you would want to allocate
to this. Important note here: you still have to focus
on BAU while you’re doing this consolidation, so the BAU doesn’t go away straightaway, so
someone still needs to be able to address the BAU work. The work that you pick and the work that you
do must be achievable. So break it down into smaller tasks, and exactly
how you would do if you were planning a sprint. If you’re doing this and might not come back
for three months, it’s important to make sure that the stuff you start is going to be finished
in the time you’ve got. So part 7 is the one with the new role. So by this time, we are starting to mature
in the way that we handled our BAU, and we still had the problem where we were having
senior developers and myself doing a lot of the work. So we introduced a brand new role into our
team, and this role was called the technical support engineer. So obviously, their main role, their main
focus is to do the BAU is to also do some improvements to it over time as well. But we actually have an unwritten rule, an
unwritten goal for this job in our team. And that is to automate themselves out of
the role altogether. It sounds really bad but actually that’s all
about the transition from this role, technical support engineer, into a fully fledged developer,
and that means that as they learn how to do the automation and they learn how to programme
and code, and all that sort of stuff, and learn the technology and the code that we
have, they become fully fledged and they can start doing normal development work. It’s a great way to motivate this. It’s a fantastic job to do if you’re new and
you want to learn a whole pile of things. The . The earn doing this will get stuck into
processes, Java, Scala, C, and Perl, and there are a whole lot of things they will get experience
with. Because of that, it can be fun. We allow the person doing this role to have
some independent identification of problems and solutions. We allow them to have some level of autonomy
in selecting the things that they might want to improve and to do. And that’s a great way to motivate them and
also to generally improve the way that they deal with the work. The great effect for us is that it has unblocked
the team. It’s a better use of the team’s knowledge,
the people with the detailed knowledge and experience can focus on adding the value by
doing the new things, and the people who don’t have that yet but will get it eventually are
in a place where they can start to learn that. A final note on this new role that we introduced:
they still need to have the support. We have a couple of people in our team who
are essentially mentoring this person, and it’s a great role to have, if you want to
introduce a mentoring scheme into your work, it’s a great place to start. Part 8, the last part, the one with the cliffhanger. So, I haven’t given any silver bullets, there’s
no magic wand that you can wave to fix all of your problems, but hopefully some of the
ideas I’ve talked about will resonate with you, and you can go away and start think about
how you would introduce those into your teams. See what works for you. And remember that you shouldn’t stick with
one approach over forever, so we’ve talked through a whole pile of things here which
we’ve evolved our team through, and we even go back to doing some things that we did before
and stopped, so be treated to adapt your approach over time. Really important note: don’t lose heart. So you might be really frustrated about your
BAU, but you will get there in the end, I guarantee it. It just takes a little bit of patience and
a little bit of motivation to get this started and you’ll be absolutely fine. Let me know how you get on. I’m keen to find out what your experiences
are, if you’ve tried some of these things, let me know how you got on. If you try something different, come and talk
to me as well and you can do that straight after my talk in the office hours. Thank you very much!
[ Applause }

1 thought on “Business as usual: how to stop drowning and learn to swim | Jonathan Stott | #LeadDevLondon

  1. 02:10 It looks like he's successfully pushing a double decker bus?

    I used to drive those. Once had one which wouldn't start while I blocked the driveway to a school. Put it in neutral with the engine off and it did roll gently downhill until it was out of the way.

Leave a Reply

Your email address will not be published. Required fields are marked *