Android: The Sad State of integration testing for Open Source projects
Hello. It’s currently 2019 and testing open source Android projects is extremely hard. The reason for this is the state of Android Emulator and it’s dependencies.
I am going to focus on the official Android Emulator images provided via the Android SDK
Before Android 26 we had:
- x86 images, provided your system supported hardware vm acceleration
- arm-based images. If your system didn’t support hardware acceleration then you could still run an emulator. Very-very slowly
By hardware acceleration I’m referring to the HAXM/Hypervisor.framework on MacOS, KVM on Linux and HAXM/WHPX on Windows
This status quo changed when Google stopped building arm images. Why does this matter you might ask?
Well, for enterprise companies it doesn’t matter. Everyone has their own choice of a public mobile farm (Firebase Test Lab/AWS Devices/etc). Or they have an internal mobile farm.
For open source world this is a kill-zone now. Don’t get me wrong: it’s still possible to use pre Android 26 images on public CI systems. But it’s completely impossible to test your code against a 26+ emulator. There is a technical reason for this though. Build agents are executing your builds on Travis/Circle/etc in an ephemeral environment. This means that your build is running in a VM or in a container. Giving the access to the actual hypervisor by exposing KVM for example is a serious risk in this case. Unless you triple check security for this sandbox.
What does it mean in practice? Well, unless a developer of an open source project has access to some kind of company-internal CI, it’s impossible to make sure everything still works on new versions of Android. And the list will grow as the time goes by.
Someone might mention that Firebase Test Lab has a free plan. Well, the limit of tests is 10 per day for virtual devices. I doubt that this high number will give you full test coverage. This means I can’t check that my code works, I can’t prove to my users that my code works, and as a result, the quality of my work decreases.
Another problem with any public cloud offering on the market is that you don’t get direct adb access. I want to pay attention to this one since I’m developing open source tooling for Android testing. I need direct access to adb and emulator to test my tools. These tools are integrating with Android’s tooling stack and it’s not exposed in public farm solutions. Even as a user who runs tests you might need direct access to a device if you’re using a custom test runner for example.
All this leads me to believe that Google doesn’t pay that much attention to the open source community. CI vendors can’t solve this problem either. The effort to build secure and performant nested virtualisation is huge.
Recently I enrolled into the Github Actions Beta and I see some positive direction here. You can use Windows, Linux and MacOS environments here for your actions in case you haven’t yet tried it. Windows and Linux are backed by Azure (kudos to Microsoft). The Linux offering doesn’t support nested VMs so still no luck here. MacOS on the other hand is provided by MacStadium. Here is where things start to look interesting. I’ve noticed that the Hypervisor.framework is enabled on the build agents. I started experimenting.
$ sysctl kern.hv_support
kern.hv_support: 1After a couple of days of work I got a basic setup working. One day I received a link to a post about Continuous testing with new Android emulator tools from Google. I got excited. But as I read more, it became obvious that it’s focusing on the wrong problem. It’s not hard to start an emulator on public CIs. There are GitHub snippets and whole projects that are doing that for the community, so we don’t need help there. What we do need is to be able to test with all the available versions of Android on public CIs.
I don’t want this to be only ranting, so I though about some suggestions:
- If what I’ve found with MacStadium is not a bug, then please-please-please don’t remove support for this. I don’t mind testing Android code on Apple’s hardware, but I mind not testing my code at all
- Google, please bring back support for arm images. This won’t fix the performance, but slow CI is better than nothing
- A project with focus to sandbox nested virtualisation. This might help public CI solutions to support Android CI needs
- Help projects that try to run Android in a real container (without
—-privilegedand-v /dev/kvm:/dev/kvm). They don’t have enough traction. One of them is anbox
If you’re still here, then thank you for reading this. Maybe you can help me raise attention to this? You can either comment on an open issue for API 26 ticket or spread this and hopefully someone from the Android team at Google will notice this.
Links:
