At last AnDevCon, Doug Bateman, the director of training for NewCircle, moderated a panel focused on what it takes to build Android apps that scale up to millions of global users. This included team management, testing and design for testability, feature and release management, support, open source contributions, alternative architectures, and more.
The event was organized by NewCircle Training and gathered together the following experts to share their insights: Howard Harte from Cyanogen, Inc.; Jake Wharton from Square; Ty Smith from Twitter's Fabric team; Juan Gomez from Eventbrite; Mike Hines from Amazon; Larry Schiefer from HiQES; and Dave Smith from Possible Mobile.
How do you manage teams responsible for a huge number of features?
Ty described shortly how the Twitter Fabric team went from originally 15 people after Twitter acquired Crashlytics to a 60 people team responsible for many features. Anyway, their organization remained the same, based on creating smaller teams around individual features and then making sure that those teams communicate at least on a weekly basis. Juan stressed the importance of automation testing and other processes to ensure that contributions from new team members do not break anything.
How do you do the experimentation before rolling out new features to massive user bases?
According to Mike, you don't have to push a set of changes to all users at once. You can just push a change to a small set of users. Amazon uses an internal framework that allows to do A/B testing to find out what features falls over, so it is possible to decide what to push to all users. Ty, on the other hand, remarks that A/B testing is not a good option when developing an SDK, since developers expect some kind of stability and they would not like that new features get pushed and removed after a while. So, in his case, it all comes down to developers' feedback and working with developers, focus groups, etc.
What are your strategies for testing? Which tools do you use?
In Howard's case, being CyanogenMod supported by more than a hundred devices, testing heavily relies on people trying it out on their devices and reporting back. Jake explains that at Square three separate types of tests are in place: unit tests (in the thousands) that are enforced before committing to the main repo, slower instrumentation tests that run asynchronously to exercise various flows of the app, and finally manual testing. Interestingly, developers at Square try to use their app to simulate the payment process for all coffees, lunches, etc. Juan remarked once again the importance of automated tests, although they also rely on a dedicated QA team. They used to have Robotium as a tool, but are switching over to Espresso.
Answering a question from the audience, Jake provided his view on Robotium saying that it is implemented as a wrapper around the basic Android instrumentation API, and thus it does not solve the basic issues that you have when testing async operations. You should not, e.g., perform a new test until you are sure that all tasks related to the previous one have completed. Espresso, on the other hand, takes a novel approach whereby it allows to perform a new test only when the main event loop is clear of all messages. Beyond that, he says, if you are using a library that does networking, you can say something like: do not perform new tests, until there is no network activity. Espresso also has got a rather declarative API that allows you to write higher-level tests.
At Fabric, Ty says, they are in the process of migrating from JUnit, to Robolectric. Robolectric provides the fastest way to test code that does not require heavy interaction with Android frameworks. Where the Android framework is required, it does a good job at creating shadow classes to interact with contexts and activities, and so on. Robolectric allows tests to run much faster because it runs on the local VM and this is extremely important since the Fabric team runs their unit tests with each commit inside of their CI pipeline. On the design side of things, Ty explains their effort to architect code so that it is highly testable, basically by following the single responsibility principle, and mocking classes, such as network communication classes, to test everything in isolation.
Jake chimes in again to say that they also use Robolectric for those cases where some code has to interact with a bundle or an intent or some UI views. In such cases, the usual practice is to wrap that code into an interface, so it can be tested with JUnit. Robolectric, instead, makes it possible to call that code directly through shadow classes. Square, Jake goes on, has been in touch with Robolectric and helped to reduce the number of shadow classes in an upcoming Robolectric release, with the aim of using as much as possible real Android code. The main benefit of this is that testing gets closer to actual behaviour on a real Android device. Finally, Jake notices how people always think of Robolectric as a way to run all their tests on the JVM, because it is faster, but this is not what Robolectric is actually for. For Jake, the big deal with it is that it provides a safety net and some more bits of code so you can easily run your code through some Android superclass.
How have you been designing your app for testability?
Ideally, says Juan, you want to architect things so you can automate tests with the help of some mocking. There are patterns and libraries that can be used to decouple your code and make it more testable. At Eventbrite, they use their own framework for doing mocking and dependency injection, and they are looking into Mockito as well. Sometimes, says Juan, it's not that easy to get a clean unit to test when there are interactions with the Android framework, but there are ways to make the process less painful.
Being Fabric an SDK, follows on Ty, they cannot use the dependency injection pattern, basically due to the intent of being as lightweight as possible, which makes undesirable using a beautiful dependency injection tool like Dagger. What Fabric's developers do, when there are interactions with Android activity or fragments, is trying to decouple the business logics from the activity itself so the former can be tested in isolation. According to Ty, you want Activities as much as possible out of your tests, and that is what they do by binding activities to their dependencies through a singleton controller.
Finally, Jake offers a slightly different view by observing that design for testability is important, but there is also a balance between the abstractions you use to test, their number, the mocks, and the final aim you have, which is shipping your app. So you want also to do things that simply get the thing done. You want to abstract, you want a clean separation, but sometimes you have just put some code in there to ship and maybe you can come back later to improve it. So design for testability could be seen as an incremental process.
What about testing at the embedded levels?
If you work at the embedded level, says Larry, with kernel level drivers, or HAL, you will not have the same frameworks or patterns that are available on the JVM. At his company, they try to use the same modular principles to construct drivers or libraries and abstract them into components that can be extracted out into a proprietary test framework. For drivers, they leave them in the kernel space and rely on a series of application space tests, so they know the drivers are reliable inside of kernel space.
What alternative design approaches are you using?
Jake recounts how they jumped on the Fragments bandwagon to manage their UI, but found that it tried to be too many things for too many people, as in not being enough "opinionated" about how to do things, and being complex and buggy. They removed it, finally, when they understood that they were actually abusing the system by trying to have a single activity holding a bunch of fragments. This brought the system to a limit, so they decided to implement their own code to manage views. Indeed, Fragments are nothing more than a way to handle your views and animating then in and out. This allowed Square engineers to simplify their code and they could see their crash rate going down with every new release, basically thanks to their getting rid of fragments. Fragments are maybe appropriate when you deal with just a few of them at a time, says Jake.
Is anybody using reactive style programming or reactive Java for their apps?
Once again Jake takes the lead and expresses his experience with RxJava. Succinctly, the problem is that RxJava is a framework, not a library, which means you are going to do a lot of work to integrate its API across all of your app and you almost never want it in your activities, views, services, because it is simply too much, says Jake. At Square, they use internal helpers and also RxAndroid, which is part of the same package as RxJava. RxAndroid will make it more easy to connect the asynchronous observable model of RxJava to the Android world.
How do you deal with Android at scale when you have a worldwide customer base?
Mike gives his view about the importance of localization and internationalization to get support for a global audience right. This comes down to being prepared for things such as right-to-left writing support, or languages like German that can have extremely long words that are able to defeat any UI, etc. Also important is abstracting things and using resource files. An interest side of supporting a large user base comes from the interaction between support people and engineers, says Mike. Support people usually can come up with interesting user stories for the features you built, but as scale increases they will also suggest ways of using those features that were never intended to be. Sometimes, though, those new stories are really great!
For Ty, when supporting a global user base, device and OS fragmentation becomes an important factor, because you'll want your SDK to support the widest range of devices, but you do not choose them. Therefore, it is important to be able to incrementally degrade your features and also to be able to run with the minimum amount of resources, what is especially relevant for emerging markets. Juan reinforces the same concept by explaining that he makes always sure to give an app a spin on low resolution or slow devices because that is what many users outside of the US will have. Furthermore, many times, says Howard, devices out there have not got much memory either.
Can you touch on how your organization engages with the open source community?
At Amazon, says Mike, they have a group responsible for dealing with open source contributions and also giving back to open source community. There is a process in place where engineers submit what they would like to contribute back to Amazon's lawyers so they can check any involved IP. This is a very tricky business, says Mike, so don't let engineers be attorneys!
Jake provides quite a different view: at Square, the process is entirely engineer-driven. They also have a legal or license-review, but there is nobody saying: "Hey, this looks nice, you should open source that." Square has no formal process around, besides using peer reviews to evaluate a candidate for open sourcing, and ultimately, if it is deemed appropriate, it is pushed open source. The relation between Square and the open source community is strong, both giving to Square's many contributions, but also thank to external users sending pull requests in. Jake remarks how they even hired a couple of people thanks to their pull requests.
Cyanogen also has a great relation with open source, says Howard, with about 9,000 people that have contributed from the community. One nice thing about open source is you can review it to ensure that it is safe and is not open to exploits.
What are some of the open source libraries you consider indispensable?
Dave explains that his company does not contribute a lot to open source, though they consume a lot of it. Of course, they pay a lot of attention to code license, so there are cases where they are not comfortable with the license and have to reimplement the thing. Dave says that they heavily use many libraries from Square, such as Picasso, Retrofit, and Dagger.
In Larry's case, it is very important to make customers know about the possible use of a given open source library, so they understand what implications that has, and give them a chance to decide what is best for them.
How do analytics help in building better apps?
Ty points out that there are a lot of different analytics. E.g. the A/B testing that was mentioned earlier allows you to make important decisions about your product's features. Fabric is a very opinionated analytics platform, says Ty, which is not as flexible as Google Analytics, but gives you enough information to make high-level decisions, e.g., number of users, number of crashes, etc.
To Jake, analytics in the app are obviously great, but to him it's also important to collect them for support. For survey support calls, he explains, they record the frequency of calls for each feature, so when they detect a high call rate for a given feature, they can try to improve it, thus improving the app and support. This approach also helps to keep the support level constant in face of an app's user base growth, without requiring hiring more support people.
Finally Dave adds his view, observing that analytics are hugely important for their customers, basically because analytics help drive revenues and they get to use up to seven analytics packages in some apps. Analytics is where they get the data from to base the decisions about where their apps should go.
What are your favourite resources for learning Android
Android documentation is fantastic, according to Ty, who also believes that there are a lot of great books available, one favourite of his being "The Busy Coder's Guide to Android Development". Also plentiful are presentations about several topics that are available online, says Howard.
For Jake, it is really hard to find very good advanced level content. There is a lot of good stuff for beginners, he says, but sometimes there is not much information about newer parts of the framework or going deeper into it. So, it is difficult when you get to an intermediate level to find the material to move one step further.
Larry, Mike, and Dave all agree on the importance of mentoring to take a new developer up to speed. What they do at their companies is pairing junior developers with senior developers to get fast improvements.
How can we get our designers to understand Android? They only seem to understand iOS...
Dave has a quite radical recipe for this, suggesting to have several persons in the design department carry an Android phone for a period of time. That is the only way they are going to get to understand the interaction model beyond what screenshots look, he believes. Juan, on the other hand, stresses the importance of developers making the effort to give the right information to the designers. He recalls a case when he bought a few copies of Google's material design book and gave them to their designers.
A full-length recording of the panel is available online.
About the Author
Sergio de Simone is iOS Independent Developer and Consultant. Sergio has been working as a software engineer for over fifteen years across a range of different projects and companies, including such different work environments as Siemens, HP, and small startups. Currently, his focus is on development for mobile platforms and related technologies.
