Obviously, we have been discussing AI a lot, and looking closely at AI on the edge. When we think of AI on devices that consumers pay for, we tend to think of in terms of inference. In this framework, someone trains an AI model on a giant GPU cluster in the cloud, and then downloads that model to run queries against on our devices. Lately, we have come to realize that there is a deviation in that framework, edge devices are actually going to play an important part in training as well. No one is going to train a foundational model on their laptop, but edge AI chips are going to play an important role in gathering data to feed into the core model, and may very well do a lot of fine tuning locally as well. And this has some important implications across a variety of subjects from power consumption and battery life to privacy and security.
We recognize that people have been aware of this for a long time, we are coming to it a bit late. At the same time, this topic does not get the attention it merits. In recent weeks, we have heard the subject come up, but always as an aside, some minor feature, but it carries much more significance.
There are two factors behind the edge’s role in model building. First, everyone who builds a model is hungry for data, especially data about real world usage of those models. Every query we run on Chat GPT or every driving decision in a Tesla, gets recorded and sent back to the Cloud to help those companies improve their models. This can be used as the genesis of the next version of a model, or more likely as a way to fine tune an existing model. The second factor is the drive to personalization, which is a very specific mode of fine tuning. We all want our AI applications, and eventually agents, to know our habits and usage of those applications so they can cater to our needs better.
So why will this matter?
Let’s get privacy out of the way first. We are all accustomed to having all of our digital actions captured in some way. We may not like it, but it happens. Putting AI data capture deeper into our end devices is just one more step in that process. It is possible to construct all sorts of doomsday scenarios around this, but in reality, this is just an extension of forces that are already well underway. Admittedly, AI data collection is likely to delve even deeper into our online experiences, capturing ever more sensitive data. But rather than view this as some new existential threat with Skynet privy to our most private thoughts, this should be viewed as just one more reason we need sensible privacy policies.
Human nature as it is, the most direct impact we are going to reject about edge data collection will be the impact on battery life. Some of the AI applications we hear are in the works will essentially be ‘always on’, running in the background, gathering data. And as much as all the AI chip vendors are going out of their way to explain how power efficient their semis are, this type of usage model is going to end up sucking a lot of power. Many of the advances in extending battery life from the PC and smartphone makers in recent years have rested on letting the device power down, turning off processors when not in use. With model data collection, those circuits are never to turn off. Our guess is that over time, the industry will find some sort of balance, and re-optimize the systems, but we can almost guarantee the early days of AI PCs are going to see some scary press stories about reduced battery life.
So far this all sounds pretty bad, but there will be some positives to this. For starters, it is clear that AI systems can do some really useful things, and we are still in very early stages of looking for things for them to do. Having the ability to capture a lot more data will make the models more useful. Having an on-device AI agent that learns our preferences will be very useful, eventually, probably.
We also think this opens up the door for some fairly exciting advances in chip design. We already know several companies designing IP for neural cores that can be slotted into chip designs in fairly compelling ways. This includes some system which could actually radically reduce power usage in products we already use.
AI, or more precisely, transformer-based machine learning systems, does not guarantee a dystopian robot-overlord future. But we are in very early stages, and there are going to be significant growing pains that we should all be aware of.
Leave a Reply