TWI900601B - Computer-implemented method, system, and non-transitory computer readable storage medium for neural architecture scaling for hardware accelerators - Google Patents

Abstract

Methods, systems, and apparatus, including computer-readable media, for scaling neural network architectures on hardware accelerators. A method includes receiving training data and information specifying target computing resources, and performing using the training data, a neural architecture search over a search space to identify an architecture for a base neural network. A plurality of scaling parameter values for scaling the base neural network can be identified, which can include repeatedly selecting a plurality of candidate scaling parameter values, and determining a measure of performance for the base neural network scaled according to the plurality of candidate scaling parameter values, in accordance with a plurality of second objectives including a latency objective. An architecture for a scaled neural network can be determined using the architecture of the base neural network scaled according to the plurality of scaling parameter values.

Description

Computer-implemented methods, systems, and non-transitory computer-readable storage media for neural architecture scaling for hardware accelerators

A neural network is a machine learning model that includes one or more layers of nonlinear operations to predict an output for a given input. In addition to an input layer and an output layer, some neural networks also include one or more hidden layers. The output of each hidden layer can be input to another hidden layer or an output layer of the neural network. Each layer of the neural network can generate an output from a given input based on the values of one or more model parameters in that layer. Model parameters can be weights or biases determined by a training algorithm to cause the neural network to produce accurate outputs.

A system implemented according to aspects of the present invention can reduce the latency of a neural network architecture by searching for candidate neural network architectures based on each candidate's computational requirement (e.g., FLOPS), operational intensity, and execution efficiency. Computational requirement, operational intensity, and execution efficiency are found to be a fundamental cause of a neural network's latency on target computational resources, rather than computational requirement alone affecting latency (including inference latency), as described herein. Aspects of the present invention provide techniques for performing neural architecture search and scaling, such as through latency-aware compound scaling and by expanding the space of candidate neural network candidates searched based on this observed relationship between latency and computational intensity and execution efficiency.

Furthermore, the system can perform composite scaling to consistently scale multiple parameters of a neural network according to multiple objectives, which can result in improved performance of the scaled neural network compared to methods that consider a single objective or that search for scaling parameters of a neural network separately. Delay-aware composite scaling can be used to quickly build a series of neural network architectures that are scaled according to different values from an initial scaled neural network architecture and that can be adapted to different use cases.

According to an aspect of the present invention, a computer-implemented method includes a method for determining an architecture of a neural network, comprising: receiving, by one or more processors, training data corresponding to a neural network task and information specifying target computing resources; using, by the one or more processors, a neural architecture search on a search space based on a plurality of first objectives to identify an architecture of a basic neural network; and identifying, by the one or more processors, a plurality of scaling parameter values for scaling the basic neural network based on the information specifying the target computing resources and a plurality of scaling parameters of the basic neural network. The identifying may include repeatedly performing the steps of: selecting a plurality of candidate scaling parameter values; and determining a performance metric of the base neural network scaled according to the plurality of candidate scaling parameter values, wherein the performance metric is determined based on a plurality of second objectives including a latency target. The method may include generating, by the one or more processors, an architecture of a scaled neural network using the architecture of the base neural network scaled according to the plurality of scaling parameter values.

The foregoing and other embodiments may include one or more of the following features, individually or in combination, as appropriate.

The plurality of first objectives for performing the neural architecture search may be the same as the plurality of second objectives for identifying the plurality of scaling parameter values.

The plurality of first objectives and the plurality of second objectives may include an accuracy target corresponding to the accuracy of the output of the basic neural network.

When the basic neural network is scaled according to the plurality of candidate scaling parameter values and deployed on the target computing resources, the performance metric may correspond at least in part to a latency measure between the basic neural network receiving an input and generating an output.

When the basic neural network is deployed on the target computing resources, the latency target may correspond to a minimum delay between the basic neural network receiving an input and generating an output.

The search space may include candidate neural network layers, each candidate neural network layer being configured to perform one or more respective operations. The search space may include candidate neural network layers, each candidate neural network layer including different respective activation functions.

The architecture of the basic neural network may include a plurality of components, each component having a plurality of neural network layers. The search space may include a plurality of candidate components of the candidate neural network layer, including: a first component of the candidate network layer, which includes a first activation function; and a second component of the candidate network layer, which includes a second activation function different from the first activation function.

The information specifying the target computing resources may specify one or more hardware accelerators; and wherein the method further includes executing the scaled neural network on the one or more hardware accelerators to perform the neural network task.

The target computing resources may include first target computing resources, the plurality of scaling parameter values may be a plurality of first scaling parameter values, and the method may further include: receiving, by the one or more processors, information specifying a second target computing resource different from the first target computing resources; and identifying a plurality of second scaling parameter values for scaling the basic neural network based on the information specifying the second target computing resources, wherein the plurality of second scaling parameter values are different from the plurality of first scaling parameter values.

The plurality of scaling parameter values are a plurality of first scaling parameter values, and wherein the method further comprises: generating a scaled neural network architecture from the base neural network architecture scaled using a plurality of second scaling parameter values, wherein the second scaling parameter values are generated based on the plurality of first scaling parameter values and one or more complex coefficients that uniformly modify the values of each of the first scaling parameter values.

The basic neural network may be a convolutional neural network, and the plurality of scaling parameters may include one or more of a depth of the basic neural network, a width of the basic neural network, and an input resolution of the basic neural network.

According to another aspect, a method for determining an architecture of a neural network includes: receiving, by one or more processors, information specifying target computing resources; receiving, by the one or more processors, data specifying an architecture of a basic neural network; and identifying, by the one or more processors, a plurality of scaling parameter values for scaling the basic neural network based on the information specifying the target computing resources and a plurality of scaling parameters of the basic neural network. The identifying may include repeatedly performing the following steps: selecting a plurality of candidate scaling parameter values; and determining a performance metric of the basic neural network scaled according to the plurality of candidate scaling parameter values, wherein the performance metric is determined according to a plurality of objectives including a latency target; and generating, by the one or more processors, an architecture of a scaled neural network using the architecture of the basic neural network scaled according to the plurality of scaling parameter values.

The plurality of objectives may be a plurality of second objectives; and receiving the data specifying the architecture of the basic neural network may include: receiving, by one or more processors, training data corresponding to a neural network task; and performing, by the one or more processors, a neural architecture search on a search space using the training data and based on the plurality of first objectives to identify the architecture of the basic neural network.

Other embodiments include computer systems, apparatus, and computer programs stored on one or more computer storage devices, each configured to perform the actions of the methods.

101: Basic Neural Network Architecture

103: Series

104A to N: Scaled Neural Network Architecture

107A to N: Candidate Networks

108: Coefficient Search Space

109: Scaled Neural Network Architecture

115: Data Center

116: Hardware Accelerator

200:Procedure

210: Block

220: Block

230: Block

240: Block

250: Block

300: Program

310: Block

320: Block

330: Block

340: Block

400: Neural Architecture Search Latency-Aware Composite Scaling (NAS-LACS) System

401: Training data

402: Target computing resource data

405: Neural Architecture Search (NAS) Engine

407: Basic Neural Network Architecture/Data

409: Scaled Neural Network Architecture

410: Performance Measurement Engine

415: Latency-Aware Composite Scaling (LACS) Engine

500: Environment

512: Client computing device

513: Processor

514: Memory

515: Server computing device

516: Processor

517: Memory

518: Instructions

519: Data

521: Instructions

523: Data

524: User input

526: User output

530: Storage device

550: Data Center

551A to N: Hardware Accelerator

560: Network

Figure 1 is a block diagram illustrating a series of scaled neural network architectures for deployment in a data center housing hardware accelerators on which the deployed neural network will execute.

FIG2 is a flow chart of an exemplary process for generating a scaled neural network architecture for execution on target computing resources.

Figure 3 shows an exemplary process for delay-aware complex scaling using a basic neural network architecture.

FIG4 is a block diagram of a Neural Architecture Search Latency-Aware Composite Scaling (NAS-LACS) system according to an aspect of the present invention.

FIG5 is a block diagram of an exemplary environment for implementing a NAS-LACS system.

Cross-reference to related applications

This application claims the benefit of U.S. Patent Application No. 63/137,926, filed January 15, 2021, under 35 U.S.C. §119(e), the disclosure of which is incorporated herein by reference.

Overview:

The techniques described in this specification generally relate to scaling neural networks for execution on different target compute resources, such as different hardware accelerators. Neural networks can be scaled according to multiple different performance targets, which can include separate targets for minimizing processing time (referred to herein as latency) and maximizing the accuracy of the neural network when scaled for execution on the target compute resource.

Generally speaking, a neural architecture search (NAS) system can be deployed to select a neural network architecture from a given search space of candidate architectures based on one or more objectives. A common objective is the accuracy of the neural network. Generally, a system implementing a NAS technique will favor networks that result in higher accuracy after training, rather than networks with lower accuracy. Following NAS selection of a base neural network, the base neural network can be scaled based on one or more scaling parameters. Scaling can include searching for one or more scaling parameter values for scaling the base neural network, for example, by searching for the scaling parameters in a coefficient search space of numbers. Scaling can involve increasing or decreasing the number of layers a neural network has or the size of each layer to efficiently utilize the computational and/or memory resources available for deploying the neural network.

A common concept related to neural architecture search and scaling is that the computational requirements required to process input through a neural network (e.g., measured in floating-point operations per second (FLOPS)) are proportional to the latency between sending an input to the network and receiving an output. In other words, a neural network with a low computational (low FLOPS) requirement is believed to produce output faster than a network with a higher computational (high FLOPS) requirement because fewer operations are performed overall. Consequently, many NAS systems choose neural networks with low computational requirements. However, because other characteristics of a neural network (such as operational intensity, parallelism, and execution efficiency) can affect a neural network's overall latency, the computational requirement and latency relationship have been determined to be disproportionate.

The techniques described herein provide latency-aware composite scaling (LACS) and an expansion of the search space for candidate neural networks from which a neural network is selected. In the context of expanding a search space, hardware accelerator-friendly operations and architectures can be included in the search space, where such additions can result in higher operational intensity, execution efficiency, and parallelism suitable for deployment on various types of hardware accelerators. These operations can include spatial-to-depth operations, spatial-to-batch operations, fused convolutional structures, and component-by-component search activation functions.

Latency-aware composite scaling of neural networks can improve neural network scaling compared to learning methods that do not optimize based on latency. Alternatively, LACS can be used to identify precise scaling parameter values for the scaled neural network that operate with low latency on target computational resources.

This technology further provides multi-objective scaling that is shared across targets used in searching for a neural architecture using NAS or similar techniques. Scaled neural networks can be identified based on the same targets used in searching for the underlying neural network. Therefore, the performance of scaled neural networks can be optimized in two phases (architecture search and scaling), rather than treating each phase as a task with a separate objective.

LACS can be integrated with existing NAS systems, at least because the same objectives for both search and scaling can be used to produce an end-to-end system for determining scaled neural network architectures. Furthermore, a range of scaled neural network architectures can be identified more quickly than with unscaled search methods, in which neural network architectures are searched but not scaled for deployment on target computing resources.

The techniques described herein can provide improved neural networks compared to those identified using learning search and scaling methods that do not use LACS. Furthermore, a range of neural networks with different trade-offs between objectives such as model accuracy and inference latency can be quickly generated for application in various use cases. Furthermore, the techniques can provide faster identification of neural networks for performing specific tasks, but the identified neural networks can perform better than the improved accuracy of neural networks identified using other methods. This is at least because the neural networks identified using search and scaling as described herein can consider characteristics that can affect latency (such as operational intensity and execution efficiency), rather than just the computational requirements of a network. In this way, an identified neural network can perform inferences faster without sacrificing network accuracy.

This technology further provides a generally applicable framework for rapidly migrating existing neural networks to improved computing resource environments. For example, LACS and NAS, as described herein, can be applied when migrating the execution of an existing neural network selected for a data center with specific hardware to a data center using different hardware. In this regard, a set of neural networks capable of performing the tasks of the existing neural network can be quickly identified and deployed on the hardware in the new data center. This application may be particularly useful in rapidly developing fields that require state-of-the-art hardware for efficient execution, such as networks performing tasks in computer vision or other image processing tasks.

Figure 1 is a block diagram illustrating a series 103 of scaled neural network architectures 104A-N deployed in a data center 115 housing a hardware accelerator 116 on which the deployed neural network will execute. Hardware accelerator 116 can be any type of processor, such as a CPU, GPU, FGPA, or ASIC, such as a TPU. According to aspects of the present invention, the series 103 of scaled neural network architectures can be generated from a base neural network architecture 101.

The architecture of a neural network refers to the properties that define the neural network. For example, the architecture can include the properties of the network's various neural network layers, how these layers process input, and how these layers interact with each other. For example, the architecture of a convolutional neural network (ConvNet) may define a discrete convolutional layer that receives input image data, followed by a pooling layer, followed by a fully connected layer that produces an output based on a neural network task (e.g., classifying the content of the input image data). The architecture of a neural network can also define the type of operations performed within each layer. For example, the architecture of a ConvNet may define the use of Reluctant Unit (ReLU) activation functions in the network's fully connected layers.

A base neural network architecture 101 can be identified based on a set of goals using NAS. Based on the set of goals, the base neural network architecture 101 can be identified from a search space of candidate neural network architectures using NAS. As described in more detail herein, the search space of candidate neural network architectures can be expanded to include different network components, operations, and layers, from which a base network that satisfies the goals can be identified.

The same set of objectives used to identify the base neural network architecture 101 can also be applied to identifying scaling parameter values for each neural network 104A-N in the series 103. The base neural network architecture 101 and the scaled neural network architectures 104A-N can be characterized by several parameters, where these parameters are scaled to varying degrees in the scaled neural network architectures 104A-N. In FIG1 , the neural networks 101 and 104A are illustrated using three scaling parameters: D indicates the number of layers in the neural network; W indicates the width or number of neurons within a neural network layer; and R indicates the size of the input processed by the neural network at a given layer.

As described in more detail herein, a system configured to perform LACS can search a coefficient search space 108 to identify multiple sets of scaling parameter values. Each scaling parameter value is a coefficient in the coefficient search space, which can be, for example, a set of positive real numbers. Candidate networks 107A through N are scaled from the base neural network 101 based on the candidate coefficient values identified as part of a search in the coefficient search space 108. The system can apply any of a variety of different search techniques to identify candidate coefficients, such as a Pareto front search or a grid search. For each candidate network 107A through N, the system can evaluate a measure of the candidate network's performance in performing a neural network task. Performance metrics can be based on a variety of objectives, including a latency objective that measures the delay between a candidate network receiving an input and producing a corresponding output as part of executing a neural network task.

After performing a search for scaling parameter values on the coefficient search space 108, the system may receive a scaled neural network architecture 109. The scaled neural network architecture 109 is scaled from the base neural network 101, where the scaling parameter values result in the highest performance metric for the candidate networks 107A-N identified during the search in the coefficient search space.

From the scaled neural network architecture 109 , the system can generate the series 103 of scaled neural network architectures 104A through N. The series 103 can be generated by scaling the scaled neural network architecture 109 by different values. The scaling parameter values of the scaled neural network architecture 109 can be scaled consistently to generate the different scaled neural network architectures in the series 103 . For example, the scaling parameter values of the scaled neural network architecture 109 can be scaled by increasing each scaling parameter value by a factor of 2. The scaled neural network architecture 109 can be scaled by different values or "compounding factors" that are consistently applied to each scaling parameter value of the scaled neural network architecture 109 . In some embodiments, the scaled neural network architecture 109 is scaled in other ways, for example, by separately scaling each scaling parameter value to generate one of the scaled neural network architectures in the series 103.

By scaling the scaled neural network architecture 109 by different values, different neural network architectures can be quickly generated to perform a task according to various use cases. Different use cases can be specified to identify different trade-offs between multiple objectives for the scaled neural network architecture 109. For example, one scaled neural network architecture can be identified as meeting a higher accuracy threshold at the expense of higher latency during execution. Another scaled neural network architecture can be identified as meeting a lower accuracy threshold but can be executed with lower latency on the hardware accelerator 116. Another scaled neural network architecture that balances the trade-off between accuracy and latency on the hardware accelerator 116 can be identified.

As an example, to perform a computer vision task such as object recognition, a neural network architecture may need to generate output in real time or near real time as part of an application that continuously receives video or image data and is used to identify a specific class of objects in the received data. In this exemplary task, the tolerance for accuracy may be lower, so a scaled neural network architecture that appropriately trades off lower latency and lower accuracy may be deployed to perform the task.

As another example, a neural network architecture may be tasked with classifying objects in a received scene from image or video data. In this example, if latency in performing this exemplary task is not considered as important as performing the task accurately, a scaled neural network with higher accuracy at the expense of latency may be deployed. In other examples, where no specific trade-off is identified or desired when performing the neural network task, a scaled neural network architecture that balances accuracy, latency, and other objectives may be deployed.

The scaled neural network architecture 104N is scaled using different scaling parameter values than the scaling parameter values used to scale the scaled neural network architecture 109 to obtain the scaled neural network 104A and may represent a different use case, such as one in which accuracy is desired rather than inference latency.

The LACS and NAS techniques described herein can generate families 103 for hardware accelerators 116 and receive additional training data and information specifying multiple different computational resources (e.g., different types of hardware accelerators). In addition to generating families 103 for hardware accelerators 116, the system can also search for a base neural network architecture and generate a family of scaled neural network architectures for different hardware accelerators. For example, given a GPU and a TPU, the system can generate separate families of models optimized for the GPU and TPU, respectively, with respect to the accuracy-latency tradeoff. In some embodiments, the system can generate multiple scaled families from the same base neural network architecture.

Illustrative methods

FIG2 is a flow chart of an exemplary process 200 for generating a scaled neural network architecture for execution on a target computing resource. The exemplary process 200 can be executed on a system of one or more processors in one or more locations. For example, a Neural Architecture Search-Latency-Aware Composite Scaling (NAS-LACS) system as described herein can execute process 200.

As shown in block 210, the system receives training data corresponding to a neural network task. A neural network task is a machine learning task that can be performed by a neural network. The scaled neural network can be configured to receive any type of data input to generate an output for performing a neural network task. For example, the output can be any type of score, classification, or regression output based on the input. Accordingly, the neural network task can be a scoring, classification, and/or regression task for predicting some output given some input. These tasks can correspond to a variety of applications that process image, video, text, speech, or other types of data.

The received training data can be in any form suitable for training a neural network according to one of a variety of different learning techniques. Learning techniques used to train a neural network can include supervised learning, unsupervised learning, and semi-supervised learning techniques. For example, the training data can include a plurality of training examples that can be received as input by a neural network. The training examples can be labeled with a known output corresponding to the output intended to be produced by a neural network trained to perform a specific neural network task. For example, if the neural network task is a classification task, the training examples can be images labeled with one or more classes that classify the objects depicted in the images.

As shown in block 220, the system receives information specifying a target computing resource. The target computing resource data may specify characteristics of a computing resource on which a neural network may be at least partially deployed. The computing resource may be housed in one or more data centers or other physical locations having any of a variety of different types of hardware devices. Exemplary types of hardware include central processing units (CPUs), graphics processing units (GPUs), edge or mobile computing devices, field programmable gate arrays (FPGAs), and various types of application specific integrated circuits (ASICs).

Some devices can be configured for hardware acceleration, including devices configured to efficiently perform specific types of operations. These hardware accelerators, which may include, for example, GPUs and tensor processing units (TPUs), may implement special features for hardware acceleration. Exemplary features for hardware acceleration may include configurations for performing operations commonly associated with executing machine learning models, such as matrix multiplication. For example, these special features may include matrix multiplication and accumulation units available in different types of GPUs and matrix multiplication units available in TPUs.

Target computing resource data may include data for one or more target sets of computing resources. A target set of computing resources may refer to a collection of computing devices on which a neural network is desired to be deployed. Information specifying a target set of computing resources may specify the type and quantity of hardware accelerators or other computing devices in the target set. Target sets may include devices of the same or different types. For example, a target set of computing resources may define the hardware characteristics and quantity of a particular type of hardware accelerator, including its processing power, throughput, and memory capacity. As described herein, a system may generate a series of scaled neural network architectures for each device specified in a target set of computing resources.

Additionally, the target computational resource data can specify different target sets of computational resources, for example, reflecting different potential configurations of computational resources housed in a data center. From this training and target computational resource data, the system can generate a series of neural network architectures. Each architecture can be generated from a base neural network identified by the system.

As shown in block 230, the system can use the training data to perform a neural architecture search on a search space to identify an architecture of a base neural network. The system can use any of a variety of NAS techniques, such as those based on reinforcement learning, evolutionary search, or differentiable search. In some embodiments, the system can directly receive data specifying an architecture of a base neural network, for example, without receiving training data and perform NAS as described herein.

A search space refers to candidate neural networks or portions of candidate neural networks that can potentially be selected as part of a basic neural network architecture. A portion of a candidate neural network architecture can refer to a component of a neural network. The architecture of a neural network can be defined in terms of multiple components of the neural network, where each component comprises one or more neural network layers. The properties of neural network layers can be defined in an architecture at the component level, meaning that the architecture can define specific operations performed in the component so that each neural network in the component implements the same operations defined for the component. Components can also be defined in the architecture by the number of layers in the component.

As part of performing NAS, the system can repeatedly identify candidate neural networks, obtain performance metrics corresponding to multiple objectives, and evaluate the candidate neural networks based on their respective performance metrics. As part of obtaining performance metrics (such as accuracy and latency of the candidate neural network), the system can train the candidate neural network using the received training data. Once trained, the system can evaluate the candidate neural network architecture to determine its performance metric and compare the performance metric against a current best candidate.

The system can repeatedly perform this search process by selecting candidate neural networks, training the networks, and comparing their performance metrics until a stopping criterion is reached. The stopping criterion can be a minimum predetermined performance threshold satisfied by a current candidate network. Additionally or alternatively, the stopping criterion can be a maximum number of search iterations, or a maximum amount of time allotted to perform the search. The stopping criterion can be a condition in which the performance of the neural network converges, for example, the performance of a subsequent iteration is less than a threshold value different from the performance of a previous iteration.

In the context of optimizing various performance metrics of a neural network (e.g., accuracy and latency), stopping criteria can specify a threshold range that is predetermined to be "optimal." For example, a threshold range for optimal latency can be derived from a theoretical or measured minimum latency achieved by a target computational resource. The theoretical or measured minimum latency can be based on a physical characteristic of the computational resource, such as the minimum amount of time required for a component of the computational resource to physically read and process incoming data. In some implementations, latency is maintained at an absolute minimum, e.g., as close to zero latency as practically possible, and is not based on a target latency measured or calculated from the target computational resource.

The system can be configured to use a machine learning model or other technique to select the next candidate neural network architecture, where the selection can be based at least in part on learned properties of different candidate neural networks that are more likely to perform well at the goal of a particular neural network task.

In some examples, the system can use a multi-objective reward mechanism for identifying basic neural network architectures as follows: Accuracy(m) is a performance metric for the accuracy of a candidate neural network m, and latency(m) is the latency of the neural network in producing an output on the target computational resource. As described in more detail herein, accuracy and latency can also be targets used by the system as part of scaling the basic neural network architecture based on the characteristics of the target computational resource. Target is a target latency value, measured in milliseconds, for example, and can be predetermined. The value w is a tunable parameter used to weight the impact of network latency on the overall performance of the candidate neural network. Different values of w can be learned or tuned based on different sets of targets for the computational resources. As an example, w may be set to -0.09, reflecting an overall larger factor for computational resources such as TPUs and GPUs, which are less sensitive to latency variations than, for example, mobile platforms (where w may be set to a smaller value, such as -0.07).

To measure the accuracy of a candidate neural network, the system can train the candidate neural network to perform a neural network task using a training set. The system can, for example, split the training data into a training set and a validation set based on an 80/20 split. For example, the system can apply a supervised learning technique to calculate the error between the output generated by the candidate neural network and the true data label of a training example processed by the network. The system can use any of a variety of loss or error functions that are appropriate for the type of task the neural network is trained for, such as cross-entropy loss for classification tasks or mean squared error for regression tasks. For example, a back propagation algorithm can be used to calculate the error gradient with respect to different weights of a candidate neural network, and the neural network weights can be updated. The system can be configured to train the candidate neural network until a stopping criterion is met, such as a number of iterations for training, a maximum time period, convergence, or when a minimum accuracy threshold is met.

In addition to other performance metrics, the system can also generate performance metrics for the accuracy and latency of the candidate neural network architecture on the target computing resource, including (i) the operational intensity of the candidate basic neural network when deployed on the target computing resource, and/or the execution efficiency of the candidate basic neural network on the target computing resource. In some embodiments, in addition to accuracy and latency, the performance metric of the candidate basic neural network is also based at least in part on its operational intensity and/or execution efficiency.

Latency, operational intensity, and execution efficiency can be defined as follows:

In (3), LATENCY is defined as , where W is the computational effort used to execute the measured neural network architecture (e.g., in FLOPS), and C is the computational rate achieved by the target compute resource processing inputs on the neural network architecture (e.g., in FLOPS/sec). E is the execution efficiency of the candidate neural network executed on the target compute resource, and E can be equal to C/C _ideal (and therefore, by algebra, C x C _ideal = E ). C _ideal is the ideal computational rate achieved on the target compute resource used to execute the measured neural network architecture. C _ideal is defined based on the operational intensity I , the memory bandwidth b of the target compute resource, and C _{max ,} which represents the peak computational rate of the target compute resource (e.g., the peak computational rate of a GPU).

Cmax and b are constants corresponding to the hardware characteristics of the _target computational resource, such as its peak computational speed and memory bandwidth, respectively. Operational intensity I refers to a measure of the amount of data processed by the computational resource on which a neural network is deployed, given the amount of computation W required to execute the neural network, divided by the memory traffic Q caused by the computational resource during the execution of the neural network ( , as shown in (3)).

The ideal computational rate for executing a neural network on a target compute resource is I xb when I < R , and C _max otherwise. R is the spine, or the minimum operational intensity required for the neural network architecture to achieve peak computational rate on the target compute resource.

Together, (3) and the corresponding description show that the inference latency of a measured neural network depends on the operation intensity I , the operations W , and the execution efficiency E , rather than on the operations W alone. In the context of hardware accelerators (specifically, hardware accelerators such as TPUs and GPUs commonly deployed in data centers), this relationship can be applied to expand the search space of candidate neural networks during NAS using "accelerator-friendly operations" and to improve how a base neural network is selected and subsequently scaled by the system.

The system can search for a base neural network architecture to improve the latency of a final neural network by simultaneously searching for candidate neural network architectures with improved operational strength, execution efficiency, and computational requirements, rather than searching solely for a computationally efficient network. The system can be configured to operate in this manner to reduce the overall latency of the final base neural network architecture.

Additionally, the search space of candidate architectures from which the system selects a base neural network architecture can be expanded to broaden the range of available candidate neural networks that are more likely to execute accurately and with reduced inference latency on the target compute resource (particularly in cases where the target compute resource is a data center hardware accelerator).

Expanding the search space as described can increase the number of candidate neural network architectures that are more suitable for data center accelerator deployment, which can result in identifying a base neural network architecture that may not yet be a candidate in a search space that has not been expanded according to aspects of the present invention. In instances where the target computational resources specify hardware accelerators such as GPUs and TPUs, the search space can be expanded using candidate architectures or portions of architectures, such as components or operations that improve operational strength, parallelism, and/or execution efficiency.

In one exemplary augmentation approach, the search space can be expanded to include neural network architecture components with layers implementing one of various types of activation functions. In the case of TPUs and GPUs, activation functions (such as ReLU or swish) have been found to typically have low operational intensity and are instead often memory-bound on these types of hardware accelerators. Because the execution of activation functions in a neural network is often limited by the amount of memory available on the target compute resource, the execution of these functions can have a significant negative performance impact on the end-to-end network inference speed.

One exemplary extension of the search space for excitation functions involves fusing excitation functions with their associated discrete convolutions. Because excitation functions typically operate element-wise and run on hardware accelerator units configured for vector operations, these excitation functions can be executed in parallel with the discrete convolutions, which are matrix-based operations typically performed on the matrix units of a hardware accelerator. These fused excitation function-convolution operations can be selected by the system as a candidate neural network component as part of the search for a basic neural network architecture as described herein. Any of a variety of different excitation functions can be used, including Swish, ReLU (rectified linear unit), Sigmoid, Tanh, and Softmax.

Different components comprising fused activation function-convolution layers can be added to the search space and can vary depending on the type of activation function employed. For example, one component of a activation function-convolution layer can include a Relu activation function, while another component can include a Swish activation function. It has been found that different hardware accelerators can perform more efficiently using different activation functions, so expanding the search space to include fused activation function-convolution with multiple types of activation functions can further improve the identification of a basic neural network architecture best suited for performing the neural network task in question.

In addition to the components with various activation functions described in this paper, the search space can be extended using other fused convolution structures to further enrich the search space with convolutions of different shapes, types, and sizes. These different convolution structures can be added as part of a candidate neural network architecture and can include 1x1 convolution dilation layers, depthwise convolutions, 1x1 convolution projection layers, and other operations such as activation functions, batch normalization functions, and/or skip connections.

As described in this paper, the root cause of the non-proportional relationship between computational requirements and latency is also demonstrated by the impact of parallelism on hardware accelerators. Parallelism can be critical for neural networks implemented on hardware accelerators such as GPUs and TPUs because such hardware accelerators may require large parallelism to achieve high performance. For example, a convolution operation in a neural network layer needs to have sufficient depth, batch size, and spatial dimensions to provide sufficient parallelism to achieve high execution efficiency E on the matrix units of the hardware accelerator. As described in reference (3), execution efficiency forms part of the overall picture of factors affecting network latency at inference.

Therefore, the NAS search space can be expanded in other ways to include operations that can exploit the parallelism available on a hardware accelerator. The search space can include one or more operations for fusing deep convolutions with neighboring 1x1 convolutions and operations for reshaping the input to a neural network. For example, the input to a candidate neural network can be a tensor. A tensor is a data structure that can represent values according to different ranks. For example, a rank-1 tensor can be a vector, a rank-2 tensor can be a matrix, a rank-3 matrix can be a three-dimensional matrix, and so on. Fusing deep convolutions can be beneficial because deep operations typically have a lower operational power, and fusing operations with neighboring convolutions can increase the operational power closer to the maximum capability of a hardware accelerator.

The search space may also include operations that reshape an input tensor by changing different dimensions of the tensor. For example, if a tensor has a depth, width, and height dimension, one or more operations in the search space that may form part of a candidate neural network architecture may be configured to change one or more of the depth, width, and resolution dimensions of the tensor. In one example, the search space may be expanded using one or more space-to-depth convolutions (such as 2x2 convolutions) that reshape the input tensor by increasing the depth of the input tensor while reducing other dimensions of the input tensor. In some embodiments, one or more operations using stride- n nxn convolutions are included, where n represents a positive integer that the system can use to reshape a tensor input. For example, if a tensor input to a candidate neural network is H x WC , then one or more added operations may reshape the tensor into One dimension.

Implementing spatial-to-depth convolutions as described above can have at least two advantages. First, convolutions are associated with relatively high computational intensity and execution efficiency, and therefore, more convolution options in the search space can enrich the overall search space when searching for a candidate neural network. High computational intensity and execution efficiency also facilitate implementation on hardware accelerators such as TPUs and GPUs. Furthermore, stride nxn convolutions can also be trained as part of a candidate neural network to contribute to the neural network's capacity.

The search space may also include operations that reshape the input tensor by moving elements of the input tensor to different locations in memory on the target compute resource. Additionally or alternatively, these operations may copy elements to different locations in memory.

In some embodiments, the system is configured to directly receive a base neural network for scaling without performing NAS or any other search for identifying the base neural network. In some embodiments, multiple devices individually perform at least a portion of process 200, for example by identifying the base neural network on one device and scaling the base neural network as described herein on another device.

The system can identify a plurality of scaling parameter values for scaling the base neural network based on information specifying a target computational resource and a plurality of scaling parameters, as shown in block 240 of FIG. 2 . The system can use latency-aware composite scaling (as currently described) to use the accuracy and latency of a candidate scaled neural network as targets for searching for scaling parameter values for the base neural network.

Generally speaking, scaling techniques are applied in conjunction with NAS to identify a neural network that is scaled for deployment on target computing resources. Model scaling can be combined with NAS to more efficiently search for a set of neural networks that support a variety of use cases. Within a scaling approach, various techniques can be used to search for different values of scaling parameters, such as the depth, width, and resolution of a neural network. Scaling can be performed by searching for values of each scaling parameter separately or by searching for a consistent set of values that adjust multiple scaling parameters together. The former is sometimes referred to as simple scaling, and the latter is sometimes referred to as compound scaling.

Scaling techniques that use accuracy as the sole objective can result in neural networks being deployed on dedicated hardware (such as data center accelerators) without properly accounting for the performance/speed impact of these scaled networks. As described in more detail in Figure 3, LACS can use both accuracy and latency objectives, which can be shared as the same objective for identifying underlying neural network architectures.

FIG3 illustrates an exemplary process 300 for delay-aware complex scaling of a basic neural network architecture. The exemplary process 300 may be executed on a system or device of one or more processors in one or more locations. For example, the process 300 may be executed on a NAS-LACS system as described herein.

As shown in block 310, the system selects a plurality of candidate scaling parameter values. Scaling parameter values are values of different scaling parameters of a neural network. As described herein, depth, width, and input resolution can be scaling parameters, at least because these parameters of a neural network can be scaled up or down. As part of selecting the scaling parameter values, the system can select a scaling coefficient tuple from a coefficient search space. The coefficient search space includes candidate scaling coefficient tuples from which scaling parameter values for a neural network can be determined. The number of scaling coefficients in each tuple can depend on the number of scaling parameters. For example, if the scaling parameters are depth, width, and resolution, the coefficient search space will contain candidate tuples of the form (α, β, γ), each with three scaling coefficients. The coefficients in each tuple can be numeric, e.g., integer or real values.

In a composite scaling method, the scaling coefficients for each scaling parameter are searched together. The system may apply any of a variety of different search techniques to identify a coefficient tuple, such as a Pareto frontier search or a grid search. However, when the system searches for a coefficient tuple, it may search for the tuple based on the same objectives used to identify the basic neural network architecture described herein with reference to FIG. 2 . In addition, the multiple objectives may include both accuracy and latency, which may be expressed as: Although the goal between the NAS implemented to identify the underlying neural architecture (shown in this paper at (1)) is the same as the goal shown in (2), a key difference in the computational performance metric under (2) is that the candidate scaled neural network m _scaled is evaluated rather than the underlying neural network m . The overall goal of the system can be to identify the scaling parameter value in a Pareto equilibrium for each of multiple objectives, such as accuracy and latency.

In other words, the system determines a performance metric of the base neural network scaled according to a plurality of candidate neural scaling parameter values, as shown in block 320. The system determines a performance metric of the base neural network based on different performance metrics that reflect the performance of the candidate scaled neural network scaled according to the candidate neural network scaling parameter values.

The base neural network architecture is scaled by its scaling parameters according to a scaling coefficient tuple searched in a coefficient search space. Accuracy(m _scaled ) can be a measure of the accuracy of the candidate scaled neural network on a validation set of examples obtained from the received training data. Latency(m _scaled ) can be a measure of the time between receiving an input and generating a corresponding output on the candidate scaled neural network when deployed on the target computing resource. In general, the system strives to maximize the accuracy of a candidate neural network architecture while minimizing latency.

As described in reference (3), the system can directly or indirectly obtain the performance metrics of the candidate scaled neural network's operation intensity, computational requirements, and execution efficiency. This is because LATENCY is based on these three other potential performance metrics. Therefore, in addition to the accuracy and latency of the scaled neural network architecture when deployed on the target computing resources, the system can also be configured to search for candidate scaling parameter values that optimize one or more of operation intensity, computational requirements, and execution efficiency.

As part of determining the performance metric, the system may use the received training data to further train or tune the candidate scaled neural network architecture. According to block 330, the system may determine the performance metric of the trained and scaled neural network and determine whether the performance metric meets a performance threshold. The performance metric and the performance threshold may be a composite of multiple performance metrics and performance thresholds, respectively. For example, the system may determine a single performance metric from both the accuracy and inference latency metrics of the scaled neural network, or the system may determine separate performance metrics for different objectives and compare each metric to a corresponding performance threshold.

If the performance metric meets the performance threshold, process 300 ends. Otherwise, the process continues, and according to block 310, the system selects a new plurality of candidate scaling parameter values. For example, the system may select a new scaling coefficient tuple from the coefficient search space based at least in part on previously selected candidate tuples and their corresponding performance metrics. In some embodiments, the system searches multiple coefficient tuples and performs a finer-grained search around each of the candidate tuples according to multiple objectives.

The system may implement any of a variety of different techniques for iteratively searching the space of coefficient candidates, such as using grid search, reinforcement learning, evolutionary search, and the like. The system may continue searching for scaling parameter values until a stopping criterion is reached, such as convergence or a number of iterations, as previously described.

In some embodiments, the system can be tuned based on one or more controller parameter values, which can be manually tuned, learned using a machine learning technique, or a combination of both. The controller parameters can affect the relative impact of various objectives on the overall performance metric of a candidate tuple. In some examples, specific values or relationships between values in a candidate tuple can be favorable or unfavorable based on learned characteristics of ideal scaling factors at least partially reflected in the controller parameter values.

At block 340, the system generates one or more sets of scaling parameter values from the selected candidate scaling parameter values based on one or more target tradeoffs. The target tradeoffs represent different thresholds for various objectives, such as accuracy and latency, and can be satisfied by different scaled neural networks. For example, one target tradeoff may have a higher network accuracy threshold but a lower inference latency threshold (i.e., a more accurate network with higher latency). As another example, one target tradeoff may have a lower network accuracy threshold but a higher inference latency threshold (i.e., a less accurate network with lower latency). As another example, a target tradeoff may balance accuracy and latency performance.

For each target tradeoff, the system can identify a respective set of scaling parameter values that the system can use to scale the base neural network architecture to meet the target tradeoff. In other words, the system can repeat the selection as shown in block 310, the determination of the performance metric as shown in block 320, and the determination of whether the performance metric meets a performance threshold as shown in block 330, where the difference between the performance thresholds is defined by the target tradeoff. In some embodiments, according to block 330, the system can search for a candidate tuple of scaling coefficients for the base neural network architecture scaled according to the selected candidate scaling parameter values that initially meet the multiple target performance metrics, rather than searching for a tuple of the base neural network architecture.

In some embodiments, after selecting a plurality of scaling parameter values and satisfying performance metrics, the system can be configured to generate a family of neural networks by rescaling the initially scaled neural network. In other words, from a tuple of scaling coefficients (α, β, γ), the system can scale the tuple by, for example, a common factor or "complex coefficient" to obtain tuples (α ' , β ' , γ ' ) and (α " , β " , γ " ) that are used to scale the base neural network by other factors. In this way, a family of models can be quickly generated, and it can include different neural networks suitable for various use cases.

For example, different neural networks in a series can be scaled according to one or more complex coefficients. and a scaling coefficient tuple (α, β, γ), a set of scaling parameters of a scaled neural network can be defined as follows: Where d , w , r are the scaling parameters of the neural network depth, width, and resolution respectively. In the case of the first scaled neural network architecture, Can be 1. In general, the compounding coefficient represents the delay budget used for network scaling, while α, β, and γ control how the delay budget is distributed to different scaling parameter values.

Returning to FIG. 2 , the NAS-LACS system can use the architecture of a base neural network scaled according to a plurality of scaling parameter values to generate one or more architectures of scaled neural networks, as shown in block 250 . The scaled neural network architecture can be part of a series of neural networks generated from the base neural network architecture and different scaling parameter values.

If the information specifying the target computing resources includes one or more sets of target computing resources, for example, multiple hardware accelerators of different types, the system may repeat process 200 and process 300 for each hardware accelerator to generate a scaled neural network architecture corresponding to each target set. For each hardware accelerator, the system may generate a series of scaled neural network architectures based on different target tradeoffs between latency and accuracy or latency, accuracy, and other targets (specifically, including operational intensity and execution efficiency, as described in reference (3) herein).

In some implementations, the system can generate multiple families of scaled neural network architectures from the same base neural network architecture. This approach can be helpful in situations where different target devices share similar hardware characteristics and can result in faster identification of a corresponding scaled family for each device, at least because the search for a base neural network architecture is performed only once.

Example system:

FIG4 is a block diagram of a Neural Architecture Search with Latency-Aware Composite Scaling (NAS-LACS) system 400 according to aspects of the present invention. System 400 is configured to receive training data 401 for executing a neural network and target computational resource data 402 specifying target computational resources. System 400 can be configured to implement the techniques for generating a series of scaled neural network architectures, as described herein with reference to FIG1 through FIG3.

System 400 can be configured to receive input data according to a user interface. For example, system 400 can receive data as part of a call to an application programming interface (API) that exposes system 400. System 400 can be implemented on one or more computing devices, as described herein with reference to FIG. 5 . For example, input to system 400 can be provided via a storage medium (including remote storage connected to one or more computing devices via a network) or as input through a user interface on a client computing device coupled to system 400.

System 400 can be configured to output a scaled neural network architecture 409, such as a series of scaled neural network architectures. The scaled neural network architecture 409 can be sent as an output, for example, for display on a user display, and optionally visualized according to the shape and size of each neural network layer as defined in the architecture. In some embodiments, system 400 can be configured to provide the scaled neural network architecture 409 as a set of computer-readable instructions, such as one or more computer programs, which can be executed by a target computing resource to implement the scaled neural network architecture 409.

A computer program can be written in any type of programming language and according to any programming paradigm, such as declarative, procedural, combinational, object-oriented, data-oriented, functional, or imperative. A computer program can be written to perform one or more different functions and operate within a computing environment (e.g., on a physical device, a virtual machine, or across multiple devices). A computer program can also implement functionality described in this specification, such as that performed by a system, engine, module, or model.

In some embodiments, system 400 is configured to forward data corresponding to scaled neural network architecture 409 to one or more other devices configured to convert the architecture into an executable program written in a computer programming language, optionally as part of a framework for generating machine learning models. System 400 can also be configured to send data corresponding to scaled neural network architecture 409 to a storage device for storage and subsequent retrieval.

System 400 may include a NAS engine 405. NAS engine 405 and other components of system 400 may be implemented as one or more computer programs, specially configured electronic circuits, or any combination thereof. NAS engine 405 may be configured to receive training data 401 and target computational resource data 402 and generate a basic neural network architecture 407, which may be sent to a LACS engine 415. NAS engine 405 may implement any of the various techniques used for neural architecture search described herein with reference to Figures 1-3. Systems according to aspects of the present invention may be configured to perform NAS using multiple targets, including the inference latency and accuracy of a candidate neural network when executed on the target computational resource. As part of determining performance metrics that the NAS engine 405 can use to search for basic neural network architectures, the system 400 can include a performance measurement engine 410.

The performance measurement engine 410 can be configured to receive a framework for a candidate basic neural network and generate a performance metric based on the objectives for performing NAS by the NAS engine 405. The performance metric can provide an overall performance metric for the candidate neural network based on multiple objectives. To determine the accuracy of the candidate basic neural network, the performance measurement engine 410 can execute the candidate basic neural network on a validation set of training examples, such as by reserving some of the training data 401 to obtain the validation set.

To measure latency, the performance measurement engine 410 may communicate with a computing resource corresponding to the target computing resource specified by the data 402. For example, if the target computing resource data 402 specifies a TPU as a target resource, the performance measurement engine 410 may send the candidate basic neural network to be executed on the corresponding TPU. The TPU may be housed in a data center that communicates with one or more processors of the implementation system 400 via a network as described in more detail with reference to FIG. 5 .

The performance measurement engine 410 can receive latency information indicating the delay between a target compute resource receiving an input and producing an output. This latency information can be measured directly on-site at the target compute resource and sent to the performance measurement engine 410, or measured by the performance measurement engine 410 itself. If the performance measurement engine 410 measures latency, the engine 410 can be configured to compensate for delays not attributable to processing the candidate basic neural network, such as network delays in communicating to and from the target compute resource. As another example, the performance measurement engine 410 can estimate the latency of processing the input through the candidate basic neural network based on previous measurements of the target compute resource and the hardware characteristics of the target compute resource.

Performance measurement engine 410 can generate performance metrics for other characteristics of candidate neural network architectures, such as their operational intensity and execution efficiency. As described herein with reference to Figures 1-3, inferred latency can be determined based on a combination of computational requirements (FLOPS), execution efficiency, and operational intensity, and in some embodiments, system 400 directly or indirectly searches for and scales a neural network based on these additional characteristics.

Once the performance metrics are generated, the performance measurement engine 410 may send the metrics to the NAS engine 405, which may then iterate a new search for a new candidate basic neural network architecture until a stopping criterion as described herein with reference to FIG. 2 is reached.

In some examples, NAS engine 405 is tuned based on one or more controller parameters to adjust how NAS engine 405 selects the next candidate base neural network architecture. The controller parameters can be manually tuned based on the desired properties of a neural network for a specific neural network task. In some examples, the controller parameters can be learned using any of a variety of machine learning techniques, and NAS engine 405 can implement one or more machine learning models trained to select a base neural network architecture based on various objectives, such as latency and accuracy. For example, the NAS engine 405 may implement a recurrent neural network that is trained to use characteristics of a previous candidate base neural network and multiple goals to predict a candidate base neural network that is more likely to meet the goal. The neural network may be trained using performance metrics and training data labeled to indicate a final base neural architecture selected based on a set of training data and target computational resource data associated with a neural network task.

The LACS engine 415 can be configured to perform latency-aware composite scaling according to aspects of the present invention. The LACS engine 415 is configured to receive data 407 specifying a base neural network architecture from the NAS engine 405. Similar to the NAS engine 405, the LACS engine 415 can communicate with the performance measurement engine 410 to obtain performance metrics for a candidate scaled neural network architecture. The LACS engine 415 can maintain a search space of different scaling factor tuples in memory and can also be configured to scale a final scaled architecture to quickly obtain a series of scaled neural network architectures, as described herein with reference to Figures 1-3. In some implementations, the LACS engine 415 is configured to perform other forms of scaling (e.g., simple scaling), but using multiple objectives (including latency) used by the NAS engine 405.

FIG5 is a block diagram of an exemplary environment 500 for implementing the NAS-LACS system 400. The system 400 can be implemented on one or more devices having one or more processors in one or more locations, such as in a server computing device 515. The client computing device 512 and the server computing device 515 can be communicatively coupled to one or more storage devices 530 via a network 560. The storage device(s) 530 can be a combination of volatile and non-volatile memory and can be in the same or different physical location as the computing devices 512, 515. For example, storage device(s) 530 may include any type of non-transitory computer-readable media capable of storing information, such as a hard drive, solid-state drive, tape drive, optical storage, memory card, ROM, RAM, DVD, CD-ROM, writable and read-only memory.

Server computing device 515 may include one or more processors 513 and memory 514. Memory 514 may store information accessible by processor(s) 513, including instructions 521 executable by processor(s) 513. Memory 514 may also contain data 523 that can be retrieved, manipulated, or stored by processor(s) 513. Memory 514 may be a type of non-transitory, computer-readable medium capable of storing information accessible by processor(s) 513, such as volatile and non-volatile memory. The processor(s) 513 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and/or application specific integrated circuits (ASICs), such as tensor processing units (TPUs).

Instructions 521 may include one or more instructions that, when executed by processor(s) 513, cause one or more processors to perform the actions defined by the instructions. Instructions 521 may be stored in object code format for direct processing by processor(s) 513, or in other formats, including interpretable scripts or independent source code modules that are interpreted on demand or pre-compiled. Instructions 521 may include instructions for implementing system 400 consistent with aspects of the present invention. System 400 may be executed using processor(s) 513 and/or using other processors located remotely from server computing device 515.

Data 523 may be retrieved, stored, or modified by processor(s) 513 in accordance with instructions 521. Data 523 may be stored in computer memory, a relational or non-relational database, or as a table with a plurality of different fields and records, or as a JSON, YAML, proto, or XML document. Data 523 may also be formatted in a computer-readable format, such as, but not limited to, binary values, ASCII, or Unicode. Furthermore, data 523 may include information sufficient to identify the relevant information, such as numbers, descriptive text, proprietary code, pointers, references to data stored in other memory (including other network locations), or information used by a function to calculate the relevant data.

The client computing device 512 may also be configured similarly to the server computing device 515, having one or more processors 516, memory 517, instructions 518, and data 519. The client computing device 512 may also include a user output 526 and a user input 524. The user input 524 may include any suitable mechanism or technology for receiving input from a user, such as a keyboard, mouse, mechanical actuator, soft actuator, touch screen, microphone, and sensor.

The server computing device 515 can be configured to transmit data to the client computing device 512, and the client computing device 512 can be configured to display at least a portion of the received data on a display as part of an implementation of user output 526. User output 526 can also be used to display an interface between the client computing device 512 and the server computing device 515. Alternatively or in addition, user output 526 can include one or more speakers, transducers, or other audio outputs, a haptic interface, or other haptic feedback that provides non-visual and non-auditory information to a platform user of the client computing device 512.

Although FIG5 depicts processors 513, 516 and memories 514, 517 as being within computing devices 515, 512, the components described herein (including processors 513, 516 and memories 514, 517) may include multiple processors and memories that may be located in different physical locations and not operate within the same computing device. For example, some instructions 521, 518 and data 523, 519 may be stored on a removable SD card, while others may be stored within a read-only computer chip. Some or all instructions and data may be stored in a location physically remote from processors 513, 516 but still accessible to processors 513, 516. Similarly, processors 513 and 516 may comprise a collection of processors capable of executing simultaneous and/or sequential operations. Computing devices 515 and 512 may each include one or more internal clocks that provide timing information that can be used to measure the timing of operations and programs executed by computing devices 515 and 512.

Server computing device 515 can be connected via network 560 to a data center 550 that houses hardware accelerators 551A through N. Data center 550 can be one of multiple data centers or other facilities where various types of computing devices, such as hardware accelerators, are located. The computing resources housed in data center 550 can be designated as part of the target computing resources for deploying a scaled neural network architecture, as described herein.

Server computing device 515 can be configured to receive requests from client computing device 512 to process data on computing resources in data center 550. For example, environment 500 may be part of a computing platform configured to provide various services to users through various user interfaces and/or APIs that expose platform services. One or more of these services may be a machine learning framework or a set of tools for generating neural networks or other machine learning models based on a specified task and training data. Client computing device 512 can receive and transmit data specifying target computing resources to be allocated for executing a neural network trained to perform a specific neural network task. According to aspects of the present invention described herein with reference to Figures 1 to 4 , the NAS-LACS system 400 can receive data and training data specifying a target computing resource and, in response, generate a series of scaled neural network architectures for deployment on the target computing resource.

As another example of a potential service provided by a platform implementing environment 500, server computing device 515 may maintain various families of scaled neural network architectures based on different potential target computing resources available at data center 550. For example, server computing device 515 may maintain different families for deploying neural networks on various types of TPUs and/or GPUs housed in data center 550 or otherwise available for processing.

Devices 512, 515 and data center 550 may be able to communicate directly and indirectly via network 560. For example, using a network socket, client computing device 512 may connect to a service operating in data center 550 via an internet protocol. Devices 515, 512 may establish listening sockets that accept an initial connection for sending and receiving information. Network 560 itself may include a variety of configurations and protocols, including the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local area networks, and private networks using one or more company-specific communication protocols. Network 560 may support a variety of short-range and long-range connections. Short-range and long-range connections can occur over different bandwidths, such as 2.402 GHz to 2.480 GHz (commonly associated with the Bluetooth® standard), 2.4 GHz and 5 GHz (commonly associated with the Wi-Fi® communication protocol), or using various communication standards, such as the LTE® standard for wireless broadband communications. Additionally or alternatively, network 560 can also support wired connections between devices 512, 515 and data center 550, including via various types of Ethernet connections.

Although a single server computing device 515, client computing device 512, and data center 550 are shown in FIG5 , it should be understood that aspects of the present invention can be implemented with a variety of different configurations and numbers of computing devices, including in scenarios for sequential or parallel processing, or across a distributed network of multiple devices. In some embodiments, aspects of the present invention can be executed on a single device connected to a hardware accelerator configured for processing a neural network, or on any combination thereof.

Example usage scenarios:

As described herein, aspects of the present invention provide an architecture for generating a neural network scaled from a base neural network according to a multi-objective approach. Below are examples of neural network tasks.

As an example, input to a neural network can be in the form of images or videos. As part of processing a given input, such as as part of a computer vision task, a neural network can be configured to extract, recognize, and generate features. A neural network trained to perform this type of neural network task can be trained to generate an output classification from a set of different potential classifications. Additionally or alternatively, the neural network can be trained to output a score corresponding to an estimated probability that a recognized object in an image or video belongs to a particular classification.

As another example, the input to a neural network can be a data file corresponding to a specific format, such as an HTML file, a word processing document, or formatted metadata derived from other types of data, such as metadata from an image file. In this context, a neural network's task can be to classify, score, or otherwise predict a characteristic of the received input. For example, a neural network can be trained to predict the probability that the received input contains text related to a specific object. Also as part of performing a specific task, a neural network can be trained to generate text predictions, such as as part of a tool for automatically completing text in a document as it is being written. A neural network can also be trained to predict a translation of text in an input document into a target language, such as when composing a message.

Other types of input files may be data related to the characteristics of a network of interconnected devices. These input files may include activity logs and records of the access privileges of different computing devices to different sources of potentially sensitive data. A neural network can be trained to process these and other types of files to predict ongoing and future network security breaches. For example, a neural network can be trained to predict a malicious actor's intrusion into a network.

As another example, the input to a neural network can be audio input, including streamed audio, pre-recorded audio, and audio that is part of a video or other source or media. A neural network task in the context of audio content can include speech recognition, including isolating speech from other identified audio sources and/or enhancing the characteristics of the identified speech to make it easier to hear. A neural network can be trained to predict an accurate translation of input speech into a target language, for example, in real time as part of a translation tool.

In addition to data input, which may include the various types of data described herein, a neural network can also be trained to process features corresponding to a given input. A feature is a value (e.g., a numeric value or a category) associated with a characteristic of the input. For example, in the context of an image, a feature of an image may be associated with the RGB value of each pixel in the image. A neural network task in the context of image/video content may be to classify the content of an image or video, such as the presence of different people, places, or things. A neural network can be trained to extract and select relevant features for processing to produce an output for a given input, and can also be trained to generate new features based on learned relationships between various characteristics of the input data.

Aspects of the present invention may be implemented in a digital circuit, a computer-readable storage medium, as one or more computer programs, or a combination of one or more of the foregoing. The computer-readable storage medium may be non-transitory, for example, as one or more instructions executable by a processor(s) and stored on a tangible storage device.

In this specification, the phrase "configured to" is used in various contexts relating to a computer system, hardware, or portion of a computer program. When a system is said to be configured to perform one or more operations, this means that the system has appropriate software, firmware, and/or hardware installed on the system that, when operated, causes the system to perform the one or more operations. When hardware is said to be configured to perform one or more operations, this means that the hardware includes one or more circuits that, when operated, receive input and produce output based on the input and corresponding to the one or more operations. When a computer program is said to be configured to perform one or more operations, this means that the computer program comprises one or more program instructions that, when executed by one or more computers, cause the one or more computers to perform the one or more operations.

Although the operations shown in the figures and described in the claims are presented in a particular order, it should be understood that the operations may be performed in a different order than shown, and that some operations may be omitted, performed more than once, and/or performed concurrently with other operations. Furthermore, the separation of different system components configured to perform different operations should not be construed as requiring the separation of the components. The described components, modules, programs, and engines may be integrated together as a single system or may be part of multiple systems.

Unless otherwise indicated, the foregoing alternative embodiments are not mutually exclusive and may be implemented in various combinations to achieve unique advantages. Because these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be considered by way of illustration, not by way of limitation, of the subject matter defined by the claims. Furthermore, the examples provided herein and clauses expressing "such as," "including," and the like should not be construed as limiting the subject matter of the claims to specific examples; rather, such examples are intended to illustrate only one of many possible aspects. Furthermore, identical element symbols in different figures may identify identical or similar elements.

101: Basic Neural Network Architecture

103: Series

104A to N: Scaled Neural Network Architecture

107A to N: Candidate Networks

108: Coefficient Search Space

109: Scaled Neural Network Architecture

115: Data Center

116: Hardware Accelerator

Claims (20)

A computer-implemented method for determining an architecture of a neural network, comprising: receiving, by one or more processors, information specifying target computational resources for deploying the neural network to perform a neural network task; receiving, by the one or more processors, data specifying an architecture of a base neural network; identifying, by the one or more processors, a plurality of scaling parameter values for scaling the base neural network based on the information specifying the target computational resources and a plurality of scaling parameters of the base neural network, wherein scaling the base neural network includes adjusting at least one of a number of neural network layers or a size of one or more neural network layers of the base neural network, wherein the identifying includes repeatedly performing the following steps: selecting a plurality of candidate scaling parameter values, determining a performance metric of the basic neural network scaled according to the plurality of candidate scaling parameter values, wherein the performance metric is determined according to a plurality of targets including a precision target and a latency target when operating the neural network using the target computing resources; and comparing the performance metric with a performance threshold to determine whether the performance metric meets the performance threshold, wherein the performance threshold includes a precision threshold and a latency threshold specific to the neural network task; generating, by the one or more processors, an architecture of a scaled neural network using the architecture of the basic neural network scaled according to the plurality of scaling parameter values, in response to the performance metric meeting the performance threshold; and The scaled neural network is deployed on the target computing resources to execute the neural network task. The method of claim 1, wherein receiving the data specifying the architecture of the basic neural network includes: receiving, by the one or more processors, training data corresponding to the neural network task, and performing, by the one or more processors, a neural architecture search on a search space using the training data and based on a plurality of objectives to identify the architecture of the basic neural network. The method of claim 2, wherein the search space comprises candidate neural network layers, each candidate neural network layer comprising a different respective activation function and configured to perform one or more respective operations. The method of claim 3, wherein the architecture of the basic neural network comprises a plurality of components, each component having a respective plurality of neural network layers. The method of claim 2, wherein the plurality of objectives used to perform the neural architecture search include the accuracy objective and the latency objective. The method of claim 1, wherein the accuracy target corresponds to a minimum accuracy of the output of the base neural network when deployed on the target computing resources. The method of claim 1, wherein the performance metric corresponds at least in part to a delay measure between the basic neural network receiving an input and generating an output when the basic neural network is scaled according to the plurality of candidate scaling parameter values and deployed on the target computing resources. The method of claim 1, wherein the latency target corresponds to a minimum delay between the basic neural network receiving an input and generating an output when the basic neural network is deployed on the target computing resources. The method of claim 1, wherein: the information specifying the target computing resources specifies one or more hardware accelerators for executing the neural network task; and deploying the scaled neural network on the target computing resources further includes executing the scaled neural network on the one or more hardware accelerators to execute the neural network task. The method of claim 1, wherein the performance threshold is a composite of a performance threshold for accuracy and a performance threshold for delay. The method of claim 1, wherein generating the scaled neural network architecture further comprises generating the scaled neural network architecture using a plurality of second scaling parameter values, the second scaling parameter values being generated based on the plurality of scaling parameter values modified by one or more complex coefficients. The method of claim 1, wherein the base neural network is a convolutional neural network and the plurality of scaling parameters include one or more of a depth of the base neural network, a width of the base neural network, or an input resolution of the base neural network. A system for neural architecture scaling includes: one or more processors, and one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for determining an architecture of a neural network, the operations comprising: receiving information specifying target computing resources for deploying the neural network to execute a neural network task; receiving data specifying an architecture of a basic neural network; identifying a plurality of scaling parameter values for scaling the basic neural network based on the information specifying the target computing resources and a plurality of scaling parameters of the basic neural network, wherein scaling the basic neural network includes adjusting at least one of the number of neural network layers or a size of one or more neural network layers of the basic neural network, wherein the identifying includes repeatedly performing the following steps: selecting a plurality of candidate scaling parameter values, determining an efficiency metric of the basic neural network scaled according to the plurality of candidate scaling parameter values, wherein the efficiency metric is determined based on a plurality of targets of an accuracy target and a latency target when operating the neural network using the target computing resources; and comparing the performance metric to a performance threshold to determine whether the performance metric meets the performance threshold, wherein the performance threshold includes an accuracy threshold and a latency threshold specific to the neural network task; generating an architecture of a scaled neural network using the architecture of the base neural network scaled according to the plurality of scaling parameter values in response to the performance metric meeting the performance threshold; and deploying the scaled neural network on the target computing resources to execute the neural network task. The system of claim 13, wherein receiving the data specifying the architecture of the basic neural network comprises: receiving training data corresponding to the neural network task, and using the training data and performing a neural architecture search on a search space based on multiple objectives to identify the architecture of the basic neural network. The system of claim 14, wherein the search space comprises candidate neural network layers, each candidate neural network layer comprising a different respective activation function and configured to perform one or more respective operations. The system of claim 14, wherein the plurality of objectives for performing the neural architecture search includes the accuracy objective and the latency objective. The system of claim 13, wherein the accuracy target corresponds to a minimum accuracy of the output of the base neural network when deployed on the target computing resources. The system of claim 13, wherein the performance metric corresponds at least in part to a delay measure between the basic neural network receiving an input and generating an output when the basic neural network is scaled according to the plurality of candidate scaling parameter values and deployed on the target computing resources. The system of claim 13, wherein the latency target corresponds to a minimum delay between the basic neural network receiving an input and generating an output when the basic neural network is deployed on the target computing resources. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for determining an architecture of a neural network, comprising: receiving information specifying target computing resources for deploying the neural network to perform a neural network task; receiving, by the one or more processors, data specifying an architecture of a basic neural network; identifying a plurality of scaling parameter values for scaling the basic neural network based on the information specifying the target computing resources and a plurality of scaling parameters of the basic neural network, wherein scaling the basic neural network includes adjusting at least one of the number of neural network layers or a size of one or more neural network layers of the basic neural network, wherein the identifying includes repeatedly performing the following steps: selecting a plurality of candidate scaling parameter values, determining an efficiency metric of the basic neural network scaled according to the plurality of candidate scaling parameter values, wherein the efficiency metric is determined based on a plurality of targets of an accuracy target and a latency target when operating the neural network using the target computing resources; and comparing the performance metric to a performance threshold to determine whether the performance metric meets the performance threshold, wherein the performance threshold includes an accuracy threshold and a latency threshold specific to the neural network task; generating an architecture of a scaled neural network using the architecture of the base neural network scaled according to the plurality of scaling parameter values in response to the performance metric meeting the performance threshold; and deploying the scaled neural network on the target computing resources to execute the neural network task.