At WWDC 2014, Apple announced the integration of TestFlight within the iOS 8 development tool suite. This aims at giving developers more options to set up a beta testing program and make the whole process a lot easier, thus promising to fix a longstanding complaint from developers about the hardness of testing iOS apps before launch.
Apple's announcement follows by a few months the acquisition of Burstly, owner of TestFlight, last February. Some of the first consequences of the acquisition were the termination of Android support and the discontinuation of TestFlight SDK for iOS, allowing to gather usage analytics. Now, TestFlight has been integrated within iTunes Connect along with some new features.
In a nutshell, what the new TestFlight Beta Testing will bring is:
-
Easier beta testers enrollment
The basic mechanism to install an app outside of the App Store requires adding each device identifier (UDID) where the app should be installed to a developer's account and generating a provisioning profile including them all.
In the new TestFlight, UDIDs are no longer required and new testers can be added to a beta program by simply sending an email to their Apple IDs. This will allow them to install the app on any of their devices.
-
Larger beta tester base
A longtime complaint of iOS developers have always been the reduced number of device UUIDs that could be added to a developer account, thus limiting the number of users that could participate in a beta program. From now on, developers will be able to add the Apple IDs for up to 1,000 beta testers to their app.
In addition to this, developers will also be allowed to have up to 25 internal testers; such testers will be required to have an iTunes Connect account and will have access to the latest build of the app.
-
Symbolicated crash reports
Crash reporting has long been a feature available to developers on iTunes Connect. Though, that feature has not always worked satisfactorily for developers due to several factors. The introduction of TestFlight should improve this and allow developers to see all of their accumulated crash logs, but only starting later next year. Furthermore, crash logs should appear automatically symbolicated.
All of these new features come with some caveats.
First and foremost, before being available to beta testers, apps will be required to pass through a review process. This could be a problem for some developers: Ole Begemann notes that review times of a few days would not make for an "an efficient beta release process." And indeed Apple seems to be aware of this being a potential issue for developers and has allowed for a less strict policy allowing "to distribute updates ... to Beta Testers without Apple’s review, unless such an update includes significant changes". In case an update includes significant changes the developer shall "inform Apple in iTunes Connect and have such Application re-reviewed."
Additional concerns are also raised by the fact that it seems that only the latest build of an app will be available to testers. Nick Arnott, writing for iMore, notices that in Apple's demonstration, the video shows all builds except for the latest being marked as "Inactive". It is not yet clear if beta testers get access to older builds too, and this could be a "deal breaker for many," since beta testers could need to go back to previous versions of an app in case a showstopper bug affects the latest.
Finally, it must be noted that TestFlight will only be available for iOS 8 when it's released, so developers still supporting previous iOS versions or Android won't be able to rely on TestFlight for those. For all of them, two alternatives are HockeyApp and Crashlytics.
