Design World

  • Home
  • Technologies
    • 3D CAD
    • Electronics • electrical
    • Fastening & Joining
    • Factory automation
    • Linear Motion
    • Motion Control
    • Test & Measurement
    • Sensors
    • Fluid power
  • Learn
    • Ebooks / Tech Tips
    • Engineering Week
    • Future of Design Engineering
    • MC² Motion Control Classrooms
    • Podcasts
    • Videos
    • Webinars
  • LEAP AWARDS
  • Leadership
    • 2022 Voting
    • 2021 Winners
  • Design Guide Library
  • Resources
    • 3D Cad Models
      • PARTsolutions
      • TraceParts
    • Digital Issues
      • Design World
      • EE World
    • Women in Engineering
  • Supplier Listings

Nvidia expands its deep learning inference capabilities for hyperscale datacenters

By Paul Heney | April 2, 2018

Share

NvidiaNvidia has announced a series of new technologies and partnerships that expand its potential inference market to 30 million hyperscale servers worldwide, while dramatically lowering the cost of delivering deep learning-powered services.

Speaking at the opening keynote of GTC 2018, Nvidia Founder and CEO Jensen Huang described how GPU acceleration for deep learning inference is gaining traction, with new support for capabilities such as speech recognition, natural language processing, recommender systems, and image recognition—in datacenters and automotive applications, as well as in embedded devices like robots and drones.

Nvidia announced a new version of its TensorRT inference software, and the integration of TensorRT into Google’s popular TensorFlow framework. Nvidia also announced that Kaldi, the most popular framework for speech recognition, is now optimized for GPUs. Nvidia’s close collaboration with partners such as Amazon, Facebook and Microsoft make it easier for developers to take advantage of GPU acceleration using ONNX and WinML.

“GPU acceleration for production deep learning inference enables even the largest neural networks to be run in real-time and at the lowest cost,” said Ian Buck, vice president and general manager of Accelerated Computing at Nvidia. “With rapidly expanding support for more intelligent applications and frameworks, we can now improve the quality of deep learning and help reduce the cost for 30 million hyperscale servers.”

TensorRT, TensorFlow integration
Nvidia unveiled TensorRT 4 software to accelerate deep learning inference across a broad range of applications. TensorRT offers highly accurate INT8 and FP16 network execution, which can cut datacenter costs by up to 70%.

TensorRT 4 can be used to rapidly optimize, validate and deploy trained neural networks in hyperscale datacenters, embedded and automotive GPU platforms. The new software delivers up to 190X2 faster deep learning inference compared with CPUs for common applications such as computer vision, neural machine translation, automatic speech recognition, speech synthesis and recommendation systems.

To further streamline development, Nvidia and Google engineers have integrated TensorRT into TensorFlow 1.7, making it easier to run deep learning inference applications on GPUs.

Rajat Monga, Engineering Director at Google, said, “The TensorFlow team is collaborating very closely with Nvidia to bring the best performance possible on Nvidia GPUs to the deep learning community. TensorFlow’s integration with Nvidia TensorRT now delivers up to 8X higher inference throughput (compared to regular GPU execution within a low latency target) on Nvidia deep learning platforms with Volta Tensor Core technology, enabling the highest performance for GPU inference within TensorFlow.”

Nvidia has optimized the world’s leading speech framework, Kaldi, to achieve faster performance running on GPUs. GPU speech acceleration will mean more accurate and useful virtual assistants for consumers, and lower deployment costs for datacenter operators.

Broad industry support
Developers at a wide spectrum of companies around the world are using TensorRT to discover new insights from data and to deploy intelligent services to businesses and consumers.

Nvidia engineers have worked closely with Amazon, Facebook and Microsoft to ensure developers using ONNX frameworks such as Caffe 2, Chainer, CNTK, MXNet and Pytorch can now easily deploy to Nvidia deep learning platforms.

Markus Noga, Head of Machine Learning at SAP, said, “In our evaluation of TensorRT running our deep learning-based recommendation application on Nvidia Tesla V100 GPUs, we experienced a 45x increase in inference speed and throughput compared with a CPU-based platform. We believe TensorRT could dramatically improve productivity for our enterprise customers.”

Nicolas Koumchatzky, Head of Twitter Cortex, said, “Using GPUs made it possible to enable media understanding on our platform, not just by drastically reducing media deep learning models training time, but also by allowing us to derive real-time understanding of live videos at inference time.”

Microsoft also recently announced AI support for Windows 10 applications. Nvidia partnered with Microsoft to build GPU-accelerated tools to help developers incorporate more intelligent features in Windows applications.

Nvidia also announced GPU acceleration for Kubernetes, to facilitate enterprise inference deployment on multi-cloud GPU clusters. Nvidia is contributing GPU enhancements to the open-source community to support the Kubernetes ecosystem. In addition, MathWorks, makers of MATLAB software, today announced TensorRT integration with MATLAB. Engineers and scientists can now automatically generate high-performance inference engines from MATLAB for Jetson, Nvidia Drive and Tesla platforms.

Inference for the datacenter
Datacenter managers constantly balance performance and efficiency to keep their server fleets at maximum productivity. Nvidia Tesla GPU-accelerated servers can replace several racks of CPU servers for deep learning inference applications and services, freeing up precious rack space and reducing energy and cooling requirements.

Inference for Self-Driving Cars, Embedded TensorRT can also be deployed on Nvidia DRIVE autonomous vehicles and Nvidia Jetson embedded platforms. Deep neural networks on every framework can be trained on Nvidia DGX systems in the datacenter, and then deployed into all types of devices—from robots to autonomous vehicles—for real-time inferencing at the edge. With TensorRT, developers can focus on developing novel deep learning-powered applications rather than performance tuning for inference deployment. Developers can use TensorRT to deliver lightning-fast inference using INT8 or FP16 precision that significantly reduces latency, which is vital for capabilities like object detection and path planning on embedded and automotive platforms.

Members of the Nvidia Developer Program can learn more about the TensorRT 4 Release Candidate at: https://developer.nvidia.com/tensorrt.


Filed Under: ALL INDUSTRY NEWS • PROFILES • COMMENTARIES, Automotive

 

About The Author

Paul Heney

Paul J. Heney, the VP, Editorial Director for Design World magazine, has a BS in Engineering Science & Mechanics and minors in Technical Communications and Biomedical Engineering from Georgia Tech. He has written about fluid power, aerospace, robotics, medical, green engineering, and general manufacturing topics for nearly 25 years. He has won numerous regional and national awards for his writing from the American Society of Business Publication Editors.

Related Articles Read More >

PTDA Foundation now accepting nominations for 2022 leadership awards
HERMES AWARD – Jury nominates three solutions
FAULHABER-integriert-PDT_220401
FAULHABER integrates subsidiary PDT
pneuw455x255hr_LB2
EXAIR Webinar Series: Break the cycle of replacing electric shop vacuums

DESIGN GUIDE LIBRARY

“motion

Enews Sign Up

Motion Control Classroom

Design World Digital Edition

cover

Browse the most current issue of Design World and back issues in an easy to use high quality format. Clip, share and download with the leading design engineering magazine today.

EDABoard the Forum for Electronics

Top global problem solving EE forum covering Microcontrollers, DSP, Networking, Analog and Digital Design, RF, Power Electronics, PCB Routing and much more

EDABoard: Forum for electronics

Sponsored Content

  • Global supply needs drive increased manufacturing footprint development
  • How to Increase Rotational Capacity for a Retaining Ring
  • Cordis high resolution electronic proportional pressure controls
  • WAGO’s custom designed interface wiring system making industrial applications easier
  • 10 Reasons to Specify Valve Manifolds
  • Case study: How a 3D-printed tool saved thousands of hours and dollars

Design World Podcasts

April 11, 2022
Going small with 3D printing
See More >
Engineering Exchange

The Engineering Exchange is a global educational networking community for engineers.

Connect, share, and learn today »

Design World
  • Advertising
  • About us
  • Contact
  • Manage your Design World Subscription
  • Subscribe
  • Design World Digital Network
  • Engineering White Papers
  • LEAP AWARDS

Copyright © 2022 WTWH Media LLC. All Rights Reserved. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with the prior written permission of WTWH Media
Privacy Policy | Advertising | About Us

Search Design World

  • Home
  • Technologies
    • 3D CAD
    • Electronics • electrical
    • Fastening & Joining
    • Factory automation
    • Linear Motion
    • Motion Control
    • Test & Measurement
    • Sensors
    • Fluid power
  • Learn
    • Ebooks / Tech Tips
    • Engineering Week
    • Future of Design Engineering
    • MC² Motion Control Classrooms
    • Podcasts
    • Videos
    • Webinars
  • LEAP AWARDS
  • Leadership
    • 2022 Voting
    • 2021 Winners
  • Design Guide Library
  • Resources
    • 3D Cad Models
      • PARTsolutions
      • TraceParts
    • Digital Issues
      • Design World
      • EE World
    • Women in Engineering
  • Supplier Listings