Amazon has announced a new testing tool to allow developers to make their Alexa skills available to closed groups of beta testers before they are published.
The new Skills Beta Testing feature can be enabled for any skills by the development account admin or any developer. The only requirements are that the skill run either on AWS Lamdba or on a server with an SSL certificate from a trusted certification authority, and that all the skill metadata be provided, including publishing and privacy information. Custom skills running on servers using a self-signed SSL certificates will not be allowed to use the new beta testing features.
Once Beta Testing is enabled for a skill, it becomes possible to enter a list of email addresses of testers and send out the invitations, thus starting the testing phase. Up to 2000 testers can be invited in total. A test will remain active for maximum 90 days after its start. At any time, you can add or remove new testers, send reminders to testers who have not accepted their invitation yet, request feedback, or stop the test. The Beta Testing dashboard also provides access to metrics such as the number of active testers and their status.
On the same account a live skill and a beta skill can co-exist, meaning that users that have access to the beta version will not have access to the live version. Testers can provide their feedback via email, and no special support is provided yet for doing surveys or for other feedback collection techniques.
Before the introduction of Amazon Skills Beta Testing, the only way for developers to get feedback on their not-yet-published skills was by adding all beta testers as members of the main account used for the skill development. While effective in providing early access to skills, this method did not allow, crucially, to selectively filter skills available on that account – meaning that all invited testers had visibility and access to all skills – and, worse, they could even modify their metadata, incuding the skill interaction model. As an alternative, a few developers resorted to creating special accounts only for beta testing, although that meant replicating the whole skill definition, which could add some unpredictability to the process.