Andrew: Sure, we build software development and collaboration tools, so a lot about tooling is around for developer specific, so that is issue tracking, code, repositories, code views, that sort of things, but we also work on tools that are for less technical audience, so project management, workflow management, that sort of thing, and also make a wiki, for collaboration as well.
Jo: Yes, I’ve been in Sydney about 2 years now, I moved over here with ThoughtWorks, and then I moved to Atlassian just over a year ago, so I’ve been working on GreenHopper for most of that time, I also did a bit of work on Bonfire when I first started which was pretty fun, it’s been a pretty cool ride so far.
5. From a technology perspective you are in the JVM areas of development?
Jo: At Atlassian all of our products are built with Java and GreenHopper in particular is quite JavaScript heavy so I started as a front-end Java developer they call it which is a strange beast I think but it’s about 50% Java, 50% JavaScript.
Andrew: I sort of got attracted to the big end of town when I left University, working for the large IT consulting companies which was great because I got to travel the world and experience different environments, but the thing that I got frustrated with, when we worked on a project, pretty much every time before we actually deployed it to production, I’d get reallocated to another project, so I never actually got to see the results of my work or whether what I was actually doing, decisions I was making were the right ones, so I really wanted to work for a product company and I was lucky enough to find Atlassian.
Craig's full question: Excellent. So one of the things I guess I wanted to ask you about testing, and we’ve known each other for a while, we share the same kinds of ideals in testing and you have spoken in the Agile space quite a bit, how does someone fall into testing? What is your story? Because software development is something that you go to university and learn and it’s one of those things that you kind of go through, but what about testing, how you get there?
Andrew: It’s definitely a profession that people fall into and that has pros and cons because some people discover it and actually love it and they are the kind of people that we try to hire at Atlassian. Unfortunately there is a lot of people who fall into the profession for the lack of anything better to do and as a result they kind of bring the whole profession down, so they are a bit of a hindrance as well, personally I started off in support and we were a level 2 helpdesk and through that got domain knowledge about some systems and then as a result of that I got called in to do some acceptance testing and that is when I discovered this realm and discovered that I really loved it and it took off from there.
8. So it’s one of the things where you see it from the other side and then you take off when you do?
Andrew: Once the opportunity presents itself to actually take ok or bad software and make it fantastic, it’s a pretty seductive proposition, for those people who are passionate about it.
Jo: I tried it out at university it’s funny, you should say that you can go to University and study because I fell into it as well, I was working in a part time job, ended up doing their website, and thought: “This is actually kind of fun, I think I might do a little bit more of this” and I ended up doing a little bit at university and then found a job luckily enough where I could learn on the job, so it kind of went from there.
Jo: I don’t know, I like puzzles and I like logic problems and solving those kinds of things and I think that is probably common to a lot of developers. I don’t know, it’s more fun than most girls think it is. I think when I was at school it never occurred to me and I do think there is kind of a gap in education where girls just really aren’t introduced to that kind of stuff, just you don’t even think about going to do Computer Science and for some reason guys do. I wish there were more girls in Software Development because it would definitely make my team more fun, but that’s just the way it is.
Craig's full question: So we are here at the Agile Australia Conference here in Sydney and both of you have talks here at the conference but very different ones, so Andrew you are here talking about quality assistance which is interesting, it’s a different take on what most people think the term QA is and how you help Agile teams develop faster at scale, so tell us a little bit about that because it’s sort of an interesting journey you take people on in that talk.
Andrew: When we think about quality, there’s are a few factors that can either enhance or impede producing quality software, and the one that affects Atlassian the most is growth, scaling, we are a fast growing company, and what that means is our customers numbers grow as our organization grows as our codebases grow, it can become more challenging to release better quality software. And so the QA team, its function is to mitigate those challenges and essentially the mission is to make it safer for development teams to move faster. And that is quite different to the traditional approach to QA where test teams would go “Well we’re are not really caring about the speed of development here, we’re just caring about making sure that we don’t ship bad code to customers” and often times that means that it will take a long time or slow things down on purpose as a result and that is a very different to our way of thinking. Hence why we call our QA team Quality Assistance rather than Quality Assurance, the role is really to assist the development teams, ship better quality faster.
Andrew: Well it means, recognizing that quality is the responsibility of everyone on an Agile team not just the testers, and if you accept that then that also means that testing is not just the responsibility of your testers or QA people, it’s the responsibility of everyone. And so that means encouraging and sometimes enforcing developers to learn how to be better testers themselves, because if they have those skills and that knowledge, then the likelihood of the code they are producing being of sufficient quality is much higher.
Cragi's full question: So that is probably a good point to bring you in Jo, because your talk here is about automating your JavaScript tests, JavaScript is one of those languages I think that people sometimes forget as an important facet, there’s so much of that code in the codebase and I suspect something like GreenHopper has a lot of things, tell us about that talk and how that fits in with being a developer that has to test?
Jo: I think, you’ve said it, JavaScript makes up such a large portion of the codebase for so many people, but it gets forgotten and unlike C#, Java, Ruby and all those languages, JavaScript testing just isn’t as wide spread, people don’t for whatever reason don’t seem to do it as much and I think partly you end up with this codebase that hasn’t been written to be testable and it’s just to much of: “Where the hell do I start? I’m just going forget about it and not bother to do it.” So for my talk, I’m talking more about how we can do kind of integration level tests as opposed to unit tests in the JavaScript and the inspiration really came from, on GreenHopper where we had all these slow running WebDriver tests, that were running against the front-end, and I think the build was taking nearly an hour and the tests took 20-30 minutes on every single build, so we’ve tried to speed that up by replacing them with JavaScript tests and I wanted to share some of the experiences that we’ve had doing that some of the pitfalls we’ve gone down and some of the things that are working.
14. So is there a particular tool that stands out in that space that you guys have standardized on?
Jo: We use QUnit, it’s pretty standard in Atlassian, I used to use Jasmine before I came to Atlassian, and I think they’re both really good, I don’t know if you’ve used some other ones haven’t you?
Andrew: Yeah I have, like I think QUnit is good, I think personally my interest is more in the test runners and those headless browsers that are now available that really speed up test execution. On a slight tangent, one of the problems that we have at Atlassian is the cost of maintaining our automated tests. I think a lot of people feel like they need more automation, or need to start automating. A product like JIRA has around 30,000 automated tests and what we’ve found is that a lot of those tests were written quite some time ago using different technology stacks and the sort of work that Jo’s doing, means that we can actually write faster tests more efficiently with some of these new technologies.
Andrew: It’s partly that but I think also as we moved to more single page web applications and a lot of the logic and functionality is actually in JavaScript in the browser, that changes the way you want to test those applications and the ability to take advantage of some of the newer frameworks that target towards that.
Jo: I think with the slower tests what we’re trying to do with GreenHopper is just test the happy path, so we have a minimum number of tests that really test the whole application end to end, but we try and maximize the coverage in kind of integration and unit tests, so those real small fast ones, we have lots of, but the maintenance headache is another problem that we are trying to find ways to reduce the maintenance costs really, I guess.
Jo: With the tests that I have been writing recently we are trying to figure out the different user paths through the code, so I’ve been focusing more on testing the UI which I used to think was a really bad idea and I’m kind of coming around to the idea that you can, if you watch out for some of the pitfalls and you don’t try and replicate too much of the UI in the tests, then you can get quite a lot of value out of it, but it’s really thinking about what does the user do and what do they see? So if I click an issue and then I press a keyboard shortcut, what do I expect to see happen in the UI? So that I guess is where I’m focusing on.
Andrew: Some of these newer frameworks allow you to actually test all of your JavaScript in an actual real browser, but you pass in just the necessary HTML to get those JavaScript to execute, rather than loading up the whole application, so by using that we can get things to run much faster.
Craig's full question: But I guess you see it, Andrew, at a wider scale because Jo is obviously focused in on a product and talking, in this particular talk about JavaScript although I’m sure you do a lot of testing in the Java stack, I assume you see it at a much wider angle because it is not just the GreenHopper tests that you are trying to do, it is actually a whole stack and in fact the product suite, potentially products that need testing, so again I assume it must be very hard to figure out where to expend your limited effort that you have there?
Andrew: Well it can be but I think Agile offers a solution there, if you have people who are passionate and good at testing and QA embedded in the team with good collaboration and good oversight as to what everyone on that team is working on then they’re in an excellent position to be able to make those decisions about where to apply those limited resources. It’s in those situations where you have a separation between test teams and development teams and everything is happening behind closed doors, that is when that lack of visibility means you’ve got greater risk and tends to encourage people to mitigate those risks by trying to test all the things rather just the things that they need to test.
Craig's full question: Your role as the manager of a QA team, one of the things that if you read any of the Agile text books they say, the first thing you do is you dismantle the QA team. So obviously you still have that ideal of some sort of QA team at Atlassian, how does that work, how do you make that work in an Agile environment?
Andrew: If your QA team is made up of those people who fell into testing for all the wrong reasons and then by all means I encourage you to disband the team straight away, that is really not going to help you at all. What you really do want is find those people who are passionate about it, and one of the keys there is finding people who get frustrated about things being inefficient or slow, I mean you want people who are passionate about delivering great software to end users and getting satisfaction about those users getting value from that.
If you find those people, then they’re a great boon to have on your Agile development teams, because they can get rid of roadblocks, so the QA team at Atlassian spends quite a bit of time developing our own tools and providing solutions to some of those age old problems that bog down development teams. We have innovation hackathons every quarter and the QA team won not the most recent one but the one before with a tool that solves the “Why don’t developers test in IE problem”. We all like our Macs at Atlassian or Linux, very few people with Windows, how do we solve that problem. Traditionally the problem is developers will have VM’s with IE installed on them somewhere, but managing those is a problem.
So the QA team set this goal: “We want any person in Atlassian to be able to have access to their own virtual machine, with any version of IE connected through to their development or testing environment within 10 seconds” - that was the goal. So they built the tool that leverages EC3 to do exactly that, so from the command line now developers can just say: “Dev environment IE8”10 seconds later they’ve got a browser connecting back to their dev environment to do that testing.
Jo: And can I just get back to the original question as well, I just wanted to add something. So, on the GreenHopper team we have one QA, so although there is a QA team at Atlassian, they’re really the QA’s are distributed into the development teams. So I think we don’t have a QA team that we chuck stuff over the wall to, and I think that when you were talking about dismantling it, we do that and I think it’s works really well because our QA will work with the other QA’s to kind of bring these techniques into the GreenHopper team, so he is not just working on his own or reporting to the GreenHopper team manager for example, he’s part of this QA team and I think it brings the best of both worlds doing it that way.
Jo: It’s actually tomorrow and it’s aimed at people who at least have some knowledge of JavaScript, so developers or people who at least know a little bit. I think my goal for the talk is just to inspire at least one person in my audience to go and try something. So I’m going to talk a lot about the tools and some real code examples, I’m trying to use a lot of code examples from GreenHopper, so the real life situations and the real tests that kind of exist in our codebase to demonstrate things and I said before I’m using QUnit, using some stuff out of the Sinon which is a really cool JavaScript library for testing, and it has this idea of a fake Ajax server, so if you want to test JavaScript code that makes Ajax calls, you can set the server up and so it never goes out on to the network, it just returns a kind of canned response which has been so useful in the GreenHopper tests. So I want to kind of demonstrate those techniques and hopefully somebody the light bulb will go on for someone.
Craig's full question:So you mentioned Bonfire before and I know Andrew, I was speaking to you very early, I saw some of the very early versions of Bonfire when it was I don’t even think it was called that it was called something else I actually think, but how did that tool come about it was, I guess that was kind of a frustration of testing really and a bit like these innovations things trying to put something together that helps testers, right?
Andrew: So I think to preface that, my answer, if you look at the Agile literature there is a common response to what does QA do in an Agile team, it automates all the tests, that is what you do. And don’t get me wrong, automation is extremely important, in fact if you are doing any manual regression testing then I think you are doing it wrong. But when it comes to new features, in an Agile environment, you want to be able to adapt as you are building something and respond to those sorts of things, and you want to be focused on building the best possible product feature. Often times having to write tests at the same time can be constraining on that and what we find is that manual testing of new features allows greater exploration and less time with often times less setup. And often when a feature is finally finished that is the point when we can actually look at the automated tests created today and decide whether we need to extend those or actually cut them back.
So the point there is we find a lot of value in manual testing of new stories, but we don’t use test cases the traditional way, because things change and we’d rather spend a lot of time testing the products rather than actually writing documentation. So we use a technical exploratory testing and we want to manage that in a way, we want to have the ability to actually look at what exploratory testing sessions that took place, who did them what kind of issues they found, that sort of thing. And we went and looked out in the marketplace to see what tools were available and there was none, so we built our own which was Bonfire. So effectively it was a tool that was created for our internal use, but speaking to customers who were on the same Agile curve as us, we have a use for similar tools so that is why we commercialized it and now sell it as a product.
Andrew: It is actually a tool with two purposes: one is the management of exploratory test sessions and that is all sort of done through JIRA, the other part is a set of browser add-on’s because we make web applications, and these browser add-ons allow you to raise issues from your browser, so you don’t have to switch to JIRA and enter in all those details, and because it is in browser-based we can create screen shots which you can then annotate and we can use JavaScript to actually pull out some environment details automatically, so what we are doing is just making it a very low barrier for people to actually report issues, without actually breaking them out of that testing mindset because we’re forcing them to fill in some two page issue creation form.
Craig's full question: And Jo you, I guess we are at an Agile conference and it’s kind of ironic I guess that the product you work on is GreenHopper which is the JIRA add-on for Agile management, so I have to ask: working on GreenHopper as a tool, does the GreenHopper team work Agile, do you use your own product and eat your own dog food?
Jo: Yes, we absolutely do, and in fact the team used to work with Kanban and we switched to Scrum about this time last year maybe a little bit.
24. You are saying you switched back from Kanban to Scrum? Most people work the other way.
Jo: I know, and actually I think in some ways I would like to see us go back to Kanban maybe, but part of the reason for that was it was the point where they started building up the Scrum part of the product and there was a need for us to “dog-food” the product, if that’s a verb, and yes, so they wanted to be able to see how the Scrum product was building up and so using it day and day out, I think it gives you a much better idea of what you are building and making sure that we are kind of living by our values and giving the customers something that we like as well.
25. And is that pervasive all the way through Atlassian, Andrew?
Andrew: Every team is free to choose its process and methodology, so there are some teams that are hardcore Kanban, hardcore Scrum and everywhere in-between, there is a lot of latitude there, but we certainly don’t have any teams that you would classify as traditional in any way.
Craig's full question: There’s lots of interesting, and if you go and look at the blogs at the Atlassian website, there is lots of interesting articles I know you’ve written lots about this Andrew in the past and I’ve seen some of your posts as well Jo on different things, I mean even down to the way you guys do technical writing and all the way through the entire stack of software development innovative approaches to tackling software development problems. It’s interesting that apart from where you actually use the word Agile in relation to GreenHopper, I never actually hear the word Agile really mentioned at Atlassian though and do you think that is part of the success of how you guys deliver, because you just do stuff?
Andrew: I have a little theory about this, to me when I think about Agile, I think that’s being an adult and traditional ways being a child. Some people, for whatever reasons, sometimes in maybe good reasons, prefer to be told what to do, but I think if you are going to be successful with Agile development, then you have to step up and be: “I want to be an adult, and I want to choose my own direction and I want to be accountable for the decisions I make”. And once you have that attitude then all of a sudden you are looking at everything saying: “Well I’m accountable for this and I can change things”, so you don’t have to put up with bad processes or bad tools and that sort of thing, and you can just go on and improve it, and fix it. And I think then when you get to that point, it doesn’t really matter what labels you apply to these things, this has just become natural and as a result, those kind of terminology begins to fall away.
27. So it’s really just the way you develop at Atlassian?
Andrew: That is right, and I think also when you have two founders who dropped out of university and never worked anywhere else and never experienced say traditional Waterfall type approaches, then it was never that need to distinguish what we do with those other ways, it’s just the Atlassian one.
Andrew: Scott, one our founders, often likes to say: “Don’t ask for permission, ask for forgiveness”. I think he is worried that the company is all going to want to funnel every decision through him but I think that is an ideal that a lot of people in the company actually hold dear, that we can have that freedom and that opportunity and shouldn’t let it go to waste.
Jo: We do and we release GreenHopper every two weeks which is fairly continuous, I think for our customers that is enough, like if we would try to do every single day, then they would probably get a bit upset. I think one of the main things for me is that the team is very self-managing, so we decide who is going to work on which stories that come up in the sprint between the team members, the actual developers, it’s not like our manager sits us down and says: “Right Jo you are going to do this, Michael you can do that one”, and that is just a great way to work, you can kind of look at the story list and think what’s interesting “I just want to do that one” and then you see the other ones coming up, and it’s much more motivating, and then you have the responsibility at the end of the sprint that you actually did it, so I think that is one of the main things for me.
Andrew: We ask a lot of our QA engineers and we want them to be technical and we also want them to be good facilitators because they actually are trying to assist developers and that requires some soft skills that a lot of testers traditionally may not have. The technical side is a must have but it’s very difficult to find people strong in all those areas, so we do hire people who have good technical aptitude but perhaps they’ve never actually done anything with that from university. We will invest in courses, we are training to bring those skills up to speed, but yes, technical skills are a must. I mean for us, the ability for a QA engineer to actually just review the commits that the developer has made, and be able to say: “Right, I know what has changed, so therefore I know what to then test or what’s at risk”, it’s hugely powerful and enables us to make sure that we test the right things and not everything, and that way we can create that speed. In terms of finding people, it’s a big challenge, I think I haven’t counted recently because it’s a bit depressing but I think it’s, we hire one person in every 120 who apply for a QA role at Atlassian, but on the bright positive side, that is changing, I mean that is over a five year period. Five years ago these types of people with this experience were very rare, but as Agile gets more adoption, we are finding that there is more and more people who have now got the experience that we need.
Jo: Well I’d like to think not. Actually they are changing slightly because they are hiring a lot more people who are what they call “full stack” developers now, so people who want to work across Java through to JavaScript or real front-end, even CSS and HTML stuff as well, which I think is a change from traditionally they had Java developers and then they had front-end developers and it’s much more of an emphasis now on doing both. But they did a pretty innovative recruitment drive last year, where they just took a bus to Europe and they went around, what, 5 countries was it, 4 countries and the tagline was: “Hey Europe, we’re coming to steal your geeks” and so I think they must have hired 20-30 maybe more people from that who will start over the next six months and moved across, so you know, it is hard to recruit but they are taking steps, less conventional steps I guess to find good people.
Andrew: The area that I’m really excited about is analytics and finding ways to get analytics from SaaS customers and inserting that directly into the development process, I mean some of the things that are pretty common place I guess in terms of running A/B experiments that sort of thing. But I think there is also a possibility for us to actually test out more our decision making process in Atlassian. Sometimes when you are looking at something and ways to implement it, you might end up with multiple options and no clear winner as to which is the right decision. The ability to actually quickly run an experiment against a set of customers to get some information that actually allows you to answer that question, as you need it. I think could be hugely powerful, so that is the sort of things we want to start working on.
Jo: I’ve been playing around with some Scala so we’ve got some performance testing stuff for GreenHopper which is all in Scala, so that it’s a new language for me which is pretty exciting, and I’m getting quite involved in all the build stuff, obviously our builds are on Bamboo so I’m trying to skill up in that area, it’s pretty cool. And then, I don’t know really, other than that, that’s mostly it for Atlassian.
Andrew: Well the Atlassian developer blog is the place to go that is where we publish all our insights and thoughts, and that is the place to go.