Benchmark Applications
Type |
Benchmark |
Languages |
Architecture |
Description |
|---|---|---|---|---|
Webapps |
010.sleep |
Python, Node.js, C++, Java |
x64, arm64 |
Customizable sleep microbenchmark. |
Webapps |
110.dynamic-html |
Python, Node.js, Java |
x64, arm64 |
Generate dynamic HTML from a template. |
Webapps |
120.uploader |
Python, Node.js |
x64, arm64 |
Uploader file from provided URL to cloud storage. |
Webapps |
130.crud-api |
Python |
x64, arm64 |
Simple CRUD application using NoSQL to store application data. |
Multimedia |
210.thumbnailer |
Python, Node.js, C++ |
x64, arm64 |
Generate a thumbnail of an image. |
Multimedia |
220.video-processing |
Python |
x64, arm64 |
Add a watermark and generate gif of a video file. |
Utilities |
311.compression |
Python |
x64, arm64 |
Create a .zip file for a group of files in storage and return to user to download. |
Inference |
411.image-recognition |
Python, C++ |
x64 |
Image recognition with ResNet and pytorch. |
Scientific |
501.graph-pagerank |
Python, C++ |
x64, arm64 |
PageRank implementation with igraph. |
Scientific |
502.graph-mst |
Python |
x64, arm64 |
Minimum spanning tree (MST) implementation with igraph. |
Scientific |
503.graph-bfs |
Python, C++ |
x64, arm64 |
Breadth-first search (BFS) implementation with igraph. |
Scientific |
504.dna-visualisation |
Python |
x64, arm64 |
Creates a visualization data for DNA sequence. |
For more details on benchmark selection and their characterization, please refer to our papers. Detailed information about each benchmark can be found in its respective README.md file.
Note
Benchmarks whose number starts with the digit 0, such as 020.server-reply are internal microbenchmarks used by specific experiments. They are not intended to be directly invoked by users.
The only exception is benchmark 010.sleep, which is a customizable sleep microbenchmark.
Note
ARM architecture is available only for AWS Lambda. C++ benchmarks are currently not supported on the ARM architecture.
Note
While we attempt to achieve semantically the same behavior across languages in each benchmark, there are some minor differences and there is no guarantee of binary reproducibility across languages. For example, in benchmark 411.image-recognition we load weights and import the model structure from Python package, whereas the C++ version imports a serialized TorchScript model. Similarly, graph benchmarks will not produce exactly the same result, as Python and C++ interfaces to igraph library use different RNGs.
Webapps
110.dynamic-html - Dynamic HTML
Generate dynamic HTML from a template.
Type: Webapps Languages: Python, Node.js Architecture: x64, arm64
Description
The benchmark represents a dynamic generation of webpage contents through a serverless function. It generates an HTML from an existing template, with random numbers inserted to control the output. It uses the jinja2 and mustache libraries on Python and Node.js, respectively.
120.uploader - Uploader
Upload file from provided URL to cloud storage.
Type: Webapps Languages: Python, Node.js Architecture: x64, arm64
Description
The benchmark implements the common workflow of uploading user-defined data to the persistent cloud storage. It accepts a URL, downloads file contents, and uploads them to the storage. Python implementation uses the standard library requests, while the Node.js version uses the third-party requests library installed with npm.
130.crud-api - CRUD API
Simple CRUD application using NoSQL to store application data.
Type: Webapps Languages: Python Architecture: x64, arm64
Description
The benchmark implements a simple CRUD application simulating a webstore cart. It offers three basic methods: add new item (PUT), get an item (GET), and query all items in a cart. It uses the NoSQL storage, with each item stored using cart id as primary key and item id as secondary key. The Python implementation uses cloud-native libraries to access the database.
Multimedia
210.thumbnailer - Thumbnailer
Generate a thumbnail of an image.
Type: Multimedia Languages: Python, Node.js Architecture: x64, arm64
Description
This benchmark implements one of the most common serverless workloads. It downloads an image from the cloud storage, resizes it to a thumbnail size, uploads the new smaller version to the cloud storage, and returns the location to the caller, allowing them to insert the newly created thumbnail. To resize the image, it uses the Pillow and sharp libraries on Python and Node.js, respectively.
220.video-processing - Video Processing
Add a watermark and generate gif of a video file.
Type: Multimedia Languages: Python Architecture: x64, arm64
Description
The benchmark implements two operations on video files: adding a watermark and creating a gif. Both input and output media are passed through the cloud storage. To process the video, the benchmark uses ffmpeg. The benchmark installs the most recent static binary of ffmpeg provided by John van Sickle.
Utilities
311.compression - Compression
Create a .zip file for a group of files in storage and return to user to download.
Type: Utilities Languages: Python Architecture: x64, arm64
Description
The benchmark implements a common functionality of websites managing file operations - gather a set of files in cloud storage, compress them together, and return a single archive to the user. It implements the .zip file creation with the help of the shutil standard library in Python.
Inference
411.image-recognition - Image Recognition
Image recognition with ResNet and pytorch.
Type: Inference Languages: Python Architecture: x64
Description
The benchmark is inspired by MLPerf and implements image recognition with Resnet50. It downloads the input and model from the storage and uses the CPU-only pytorch library in Python.
Important Notes
Warning
This benchmark contains PyTorch which is often too large to fit into a code package. Up to Python 3.7, we can directly ship the dependencies. For Python 3.8, we use an additional zipping step that requires additional setup during the first run, making cold invocations slower. Warm invocations are not affected.
Warning
This benchmark does not work on AWS with Python 3.9 due to excessive code size. While it is possible to ship the benchmark by zipping torchvision and numpy (see benchmarks/400.inference/411.image-recognition/python/package.sh), this significantly affects cold startup. On the lowest supported memory configuration of 512 MB, the cold startup can reach 30 seconds, making HTTP trigger unusable due to 30 second timeout of API gateway. Use Docker deployments for these configurations.
Warning
This benchmark does not work on GCP with Python 3.8+ due to excessive code size. To the best of our knowledge, there is no way of circumventing that limit, as Google Cloud offers neither layers nor custom Docker images.
Scientific
501.graph-pagerank - Graph PageRank
PageRank implementation with igraph.
Type: Scientific Languages: Python Architecture: x64, arm64
Description
The benchmark represents scientific computations offloaded to serverless functions. It uses the python-igraph library to generate an input graph and process it with the PageRank algorithm.
502.graph-mst - Graph MST
Minimum spanning tree (MST) implementation with igraph.
Type: Scientific Languages: Python Architecture: x64, arm64
Description
The benchmark represents scientific computations offloaded to serverless functions. It uses the python-igraph library to generate an input graph and process it with the Minimum Spanning Tree (MST) algorithm.
503.graph-bfs - Graph BFS
Breadth-first search (BFS) implementation with igraph.
Type: Scientific Languages: Python Architecture: x64, arm64
Description
The benchmark represents scientific computations offloaded to serverless functions. It uses the python-igraph library to generate an input graph and process it with the Breadth-First Search (BFS) algorithm.
504.dna-visualisation - DNA Visualization
Creates a visualization data for DNA sequence.
Type: Scientific Languages: Python Architecture: x64, arm64
Description
This benchmark is inspired by the DNAVisualization project and it implements processing the .fasta file with the squiggle Python library.
Serverless Workflows
(WiP) Coming soon!
Applications
(WiP) Coming soon!