AWS open sources Neo-AI to push machine learning to the edge

AWS has made its Neo-AI project available under the Apache Software License, despite not being known as a prolific gifter to the open source world. According to the release announcement, the project should “eliminate the time and effort needed to tune machine learning models for deployment on multiple platforms”.

Neo-AI is made up of a compiler that helps developers optimise their machine learning models (“up to twice the speed of the original model with no loss in accuracy”) for various hardware, and a runtime for model execution. The latter is supposed to use less resources than frameworks such as TensorFlow would, which could for example facilitate the use of complex models on edge devices with constrained compute power and storage. To avoid compatibility issues, Neo-AI converts models into a common format.

At the moment, the project seems to be able to optimise models generated with TensorFlow, MXNet, PyTorch, ONNX, and XGBoost, and work on Intel, NVIDIA, and – maybe the first architecture that springs to mind when talking about resource-constrained devices – ARM platforms. Support for Xilinx, Cadence, and Qualcomm systems is already planned. 

Judging by the introductory blog post, the Neo-AI repo basically contains the code of Amazon’s SageMaker Neo, which the company announced back in November 2018 at its re:Invent conference. Since then, not that much seems to have changed, so the open sourcing might be a way to make collaboration with the hardware providers mentioned easier.

But maybe AWS just regarded it as a decent thing to do, given that the project is heavily reliant on versions of deep learning compiler stack TVM and compiler and runtime project Treelite – both of which are open source. The projects serve as a basis for the DLR runtime, which provides Python and C++ APIs for loading and running models on a range of devices.

In a canned statement, Naveen Rao, General Manager of the Artificial Intelligence Products Group at Intel, described the opportunities offered by a project such as Neo-AI quiet simply: “To derive value from AI, we must ensure that deep learning models can be deployed just as easily in the data center and in the cloud as on devices at the edge.” Whether the project is able to help with that or drown in between all the other AI noise remains to be seen.