Vertical Integration Becoming Pervasive

Key Takeaways:

• Hardware and software development have traditionally been disconnected, creating little opportunity to optimize system-level performance and energy consumption.
• Development swings between specialized and generalized solutions based on the rate of technology advancement.
• Energy and thermal concerns are forcing more companies to create specialized solutions even during a time of rapid change.

Thirty years ago, there was a lot of interest in hardware/software co-design, including the mapping of software onto predefined hardware platforms. At that time, the technology was not good enough to make a significant difference. The industry went in the direction of using assemblies of pre-verified IP blocks, which were integrated rather than a top-down design flow based on partitioning.

Much of the technology developed at that time has long been abandoned, but remnants do remain, such as virtual prototyping. Coupled with that are untimed and approximately timed SystemC models. Today, those virtual prototypes encompass many more engines, but they are still not viewed as being an adequate solution for realistic workloads encountered today.

This has extended into more areas today. “Long gone are the days of a hardware team building something then throwing it over the wall to an embedded software team,” says Steve Roddy, chief marketing officer at Quadric. “That era died more than two decades ago. In the past decade, virtual prototyping has extended to modeling of physical inputs and sensor interfaces in what is known as virtual twin modeling.”

When an application becomes stable, such as an audio processor or video encoder, customized hardware is more efficient than general-purpose processors. This led to a limited market for co-design and development tools. However, the development of hardware and software was not aligned. The software was relatively fixed, and the hardware was developed to optimize the running of that software.

Today’s systems are massive compared to those of several decades ago, designed for specific workloads as defined by software. At the same time, software iteration time is now much faster than hardware, making co-design a catch-up game for hardware. New tools are being developed to improve this situation. This may sound like déjà vu for industry veterans, because the tide has again turned toward custom ASICs. But this one has its own special twist.

“Historically, we have constructed abstraction layers between different parts of the software stack and different parts of the hardware stack to manage complexity for developers, with the ultimate goal of development efficiency and reliability,” says Arvind Srinivasan, design verification solutions engineer at Normal Computing. “What we have seen change is the need to squeeze optimization out of every part of the tool chain, particularly for highly performance-sensitive workloads like AI.”

A significant overlap between the needs of large AI processors and small embedded controllers is the ability to quickly and effectively create custom processors. “IP companies are focusing on making sure they are designing hardware around customer workloads and application focus,” says Sean Murphy, senior director of product at MIPS/GlobalFoundries. “This is how architectures like RISC-V are finding their differentiation in these application spaces.”

While significant progress has been made in the way that individual processors are designed, tools have more problems addressing system-level issues. “It’s all about the application or workload that you are trying to execute and understanding aspects of that — how that application affects the architecture,” says Andy Meier, principal product marketing manager for Siemens EDA. “Architecture is overloaded in that it includes both the macro-level hardware architecture, and the micro-architecture, really understanding the instructions and how certain instructions might help. In addition, it includes the software architecture, and it’s really understanding that software workload, that application, and what it is trying to accomplish.”

We remain a long way from that ultimate goal of designing hardware and software together. “We still have software teams working in their own way with their own tools, and hardware teams working with their own,” says Sam Grove, head of software and tools at MIPS/GlobalFoundries. “There is a fundamental divide that exists, and it needs to change. How do you bring the software teams and hardware teams closer together and make sure that their design flows translate to and from each other?”

Makimoto’s Wave
Back in the 1990s, Tsugio Makimoto, then CTO at Sony, made an observation about how the industry appeared to go in waves between specializing and generalizing. Since then, the market has become a lot more fragmented, but that wave continues to happen in an asynchronous manner between sectors. A more concerted move started with new requirements for edge processors that suddenly demanded greater processing power while remaining within power envelopes associated with batteries. This focus on power has spread to data centers, where thermal has become a major concern, but here it is about power densities restricting what is possible in AI.

“Twenty-five years ago, there was an ASIC for every little thing,” says Siemens’ Meier. “Then we moved to the era of general-purpose processors, and the industry collapsed down to just a few architectures. Today, we are seeing a lot of purpose-driven, specific silicon being developed. From a shift-left perspective, instead of things being serial in a typical silicon lifecycle, those phases are overlapping — and they’re overlapping more.”

Data centers used to be packed with x86 processors. Similarly, all mobile phones used Arm processors. Then AI demanded a different type of compute, and the data centers became packed with GPUs. “There’s conventional hardware, such as GPUs, and there’s the question of whether GPUs are the end of the story,” says Patrick Coles, chief scientist at Normal Computing. “There’s increasing evidence that they aren’t, and the world is going to be full of ASICs. It’s going to be heterogeneous data centers with lots of different chips inside, and we’ll compile workloads to the different chips inside the data centers.”

When software becomes stable, economics attempts to drive down costs, but when software evolves rapidly, it calls for more innovation in hardware to keep up. “It is a balance between saying we need it faster, versus building the best combination of product for the best power consumption, for the best performance,” says Frank Schirrmeister, executive director of strategic programs for system solutions at Synopsys. “The hyperscalers talk a lot about this, and one of the main reasons for them to build their own chips and their own accelerators is for co-optimization of KPIs. It may be performance. It may be power. We may be at a point where that dynamic shifts. They say we need it fast, but it’s now equally important, or even more important, that it has the right power and the right performance.”

The challenge is determining how much customization is worthwhile, economically and from a power/performance standpoint, and for whom, and sometimes that line isn’t always clear. “The specs are getting very complicated,” said Purna Mohanty, CEO of SignatureIP. “Sometimes, the customer may not be able to even articulate what they want, because they want everything in the world. That’s not possible, of course. But at the same time, you have to make sure you are in alignment with the customer. We initially wanted to sell off-the-shelf IP, because you can’t reinvent the wheel for every customer. But the flip side is that it’s rewarding to work with leading-edge customers who bring a lot of value to you, and you’re adding new features to your IP. So it’s kind of a give-and-take.”

Still, each time the pendulum swings, the demands on tools change. “We have been building better hardware to make the software run better,” says MIPS’ Grove. “There is an inversion of that model now because software is changing so fast; the workloads, whether it be different model architectures for these applications or otherwise, are outpacing the speed at which the hardware development flow takes place. Understanding the software as you build the hardware is fundamentally an inversion of that model. In that model, you just add more accelerators here or there to make that software work better. Now it becomes, ‘What is the software doing, and how do we partition and deploy this on different elements in the system?'”

Some of the changes can best be seen by looking at RISC-V. “Companies are optimizing silicon based on the workload that it’s targeting,” says Andrea Gallo, CEO for RISC-V International. “We are in the era of workload-designed silicon.”

But when you dig into the surrounding ecosystem, you see a tiny internal wave where companies develop extensions to the core to improve power or performance. This requires them to build parts from their own ecosystem, such as compilers. When they no longer get enough benefit from that proprietary extension to cover the cost of tool maintenance, they will seek to spread the costs by sharing it with everyone else and reverting to standardization.

“When you use the vendor or custom op codes, you own the complete cost of ownership,” adds RISC-V International’s Gallo. “It’s on you, because you have a custom toolchain and custom software integration. This gives you very fast innovation, but the next step would be to propose your custom extensions to become standards. Then the cost of ownership is shared across the entire ecosystem.”

When you know the workloads, you can do better than generic. “Knowing the expected software workload at the time of defining the functional specs for a chip has enabled specialization of the processing elements within SoCs,” says Quadric’s Roddy. “Processors didn’t need to be generic. They could be application-tailored. RISC-V didn’t invent the idea of tailoring a processor to a target workload. Tensilica began 15 years before RISC-V and has fully automated the creation of custom cores and accompanying toolsets. RISC-V adopted that concept and made it non-proprietary.”

In between, there is a mix of both. “If you try and run arbitrary software, you don’t have any of those insights into the workload,” says MIPS’ Murphy. “That’s where, especially on the AI compute side, we need to focus on making sure that the generic compute engines — the vector machines, the matrix machines, the things that actually execute AI-style workloads with big matrix operations — are built generically, and scalable enough that they can have good-enough performance on an arbitrary workload, even if it’s garbled slop from AI.”

Energy and thermal
A common theme pushing everyone toward co-design is energy. On the edge, that means extending battery life as much as possible. In the data centers, it is about managing the heat that comes from the consumption of that energy. That might make you think that energy has become a primary design factor. “The first thing anyone considers is whether it works functionally,” says Grove. “Then, does it work in the period of time or in the window that it needs to? Now there’s probably some heavy lifting that happens there. Finally, what does it mean to be deployed in the field? All of these are dramatically impacted by software. While software engineers probably understand issues about power, they are not equipped with the tools they need to understand it early enough. But power is probably step three on the journey of getting something out into the market.”

That needs to change. “If a software engineer is writing things purely for performance and does not know what power is, that is a problem,” says Murphy. “We are building tools and infrastructure to make sure that as they are developing that software alongside their hardware, they are getting power numbers and are optimizing their workloads. We need to make sure we are providing targeted hardware that’s optimizing for things like TOPS per watt instead of raw TOPS.”

Understanding performance and power is far from trivial. “We understand that it is a challenge, especially for heterogeneous applications with multi-level caching,” says Meier. “It is possible they will never fully understand it, but that’s where power/performance tradeoffs happen. We understand this is a problem, and there is a lot of analysis that goes on. It is ripe for EDA tooling to continue to look for ways to improve and help our customers design more efficient silicon.”

The problem continues to get tougher. “The complexity of what you can actually grasp with new tooling goes well beyond hardware and software,” says Synopsys’ Schirrmeister. “You may find hardware and software engineers uniting because they now have to deal with those pesky mechanical and electromagnetic and multiphysics guys taking software, taking hardware on the chip, and now looking at thermal. Now the hardware-software unit becomes the new element to which multiphysics is attached to assess the combination of both. Now we can deal with that complexity by virtue of organization bringing them together, but also by virtue of just complexity — being able to grasp in a long workload, based on all the activity data created by software, which parts of the chip I need to look further for IR drop aspects and things like that.”

To make life a little more complicated, hardware has to deal with workloads coming from software. There are other functions that hardware performs, and these also have to be factored into some aspects of hardware design. “If we think about it as a power performance analysis, how do we capture that in the context of a realistic workload?” says Meier. “It also works in areas outside of what we would consider a software workload. It’s still a workload, but it includes areas like DFT, understanding from a DFT context, power and performance, and what happens there.”

Some people have suggested that AI itself may be able to understand the complex relationships between hardware/software and power/performance. “If we look at the LLMs, everything that we work on today, we think about where the knowledge came from that they’ve been trained on,” says Grove. “Textbooks, repositories, public knowledge on the internet. How do we become power aware? How do these coding agents or other things actually become aware? Where does that data set live? Where is the data set for this software that ran and consumed this much power, whether it’s at the chip level, the server rack level, the box level, the car level, the data center level?”

Until AI-driven co-design is ready, solutions may again go back to the integration of IPs. “Consider matrix extensions,” says Gallo. “We have three working groups in parallel on three different options, because we believe that one matrix extension doesn’t fit all. We have one extension that is a set of operations on top of the vector extensions. It’s very tiny. They are not adding additional accumulators, and so the power consumption is very small. Then, on the other extreme case, there is a full-blown set of matrix registers, operations, and accumulators. Those provide the highest throughput and the fastest operations, because you have all the accumulators, and you can work on larger matrices. When you start adding that many registers for a matrix, die size and power consumption increase. Initially, I thought the small one was for IoT devices and the full-blown matrix accelerator was for supercomputers. Then the experts start telling me, it may even be the opposite. If you are designing an edge AI chipset, you have a few cores and possibly a single matrix accelerator that is shared, so you would use the large matrix accelerator shared among multiple cores for edge AI. With a mass replication of cores, maybe you want a small matrix capability in each core rather than being shared.”

[Editor’s Note: Part two will explore what co-design solutions look like today, the gaps that need to be filled, and the long-term implications of these changes.]

Related Articles
Specialization Vs. Generalization In Processors
What will it take to achieve mass customization at the edge, with high performance and low power?

Will Domain-Specific ICs Become Ubiquitous?
How shifts in end markets and device scaling could alter some fundamental assumptions in chip design.

Creating Agentic EDA Methodologies
Current approaches involve multiple tools, vendors, designs, data formats, and abstractions. Can agents really use them all?

The post Vertical Integration Becoming Pervasive appeared first on Semiconductor Engineering.