Benchmark Applications

Type	Benchmark	Languages	Architecture	Description
Webapps	010.sleep	Python, Node.js, C++, Java	x64, arm64	Customizable sleep microbenchmark.
Webapps	110.dynamic-html	Python, Node.js, Java	x64, arm64	Generate dynamic HTML from a template.
Webapps	120.uploader	Python, Node.js	x64, arm64	Uploader file from provided URL to cloud storage.
Webapps	130.crud-api	Python	x64, arm64	Simple CRUD application using NoSQL to store application data.
Multimedia	210.thumbnailer	Python, Node.js, C++	x64, arm64	Generate a thumbnail of an image.
Multimedia	220.video-processing	Python	x64, arm64	Add a watermark and generate gif of a video file.
Utilities	311.compression	Python	x64, arm64	Create a .zip file for a group of files in storage and return to user to download.
Inference	411.image-recognition	Python, C++	x64	Image recognition with ResNet and pytorch.
Scientific	501.graph-pagerank	Python, C++	x64, arm64	PageRank implementation with igraph.
Scientific	502.graph-mst	Python	x64, arm64	Minimum spanning tree (MST) implementation with igraph.
Scientific	503.graph-bfs	Python, C++	x64, arm64	Breadth-first search (BFS) implementation with igraph.
Scientific	504.dna-visualisation	Python	x64, arm64	Creates a visualization data for DNA sequence.

For more details on benchmark selection and their characterization, please refer to our papers. Detailed information about each benchmark can be found in its respective README.md file.

Note

Benchmarks whose number starts with the digit 0, such as 020.server-reply are internal microbenchmarks used by specific experiments. They are not intended to be directly invoked by users. The only exception is benchmark 010.sleep, which is a customizable sleep microbenchmark.

Note

ARM architecture is available only for AWS Lambda. C++ benchmarks are currently not supported on the ARM architecture.

Note

While we attempt to achieve semantically the same behavior across languages in each benchmark, there are some minor differences and there is no guarantee of binary reproducibility across languages. For example, in benchmark 411.image-recognition we load weights and import the model structure from Python package, whereas the C++ version imports a serialized TorchScript model. Similarly, graph benchmarks will not produce exactly the same result, as Python and C++ interfaces to igraph library use different RNGs.

Webapps

110.dynamic-html - Dynamic HTML

Generate dynamic HTML from a template.

Type: Webapps Languages: Python, Node.js Architecture: x64, arm64

Description

The benchmark represents a dynamic generation of webpage contents through a serverless function. It generates an HTML from an existing template, with random numbers inserted to control the output. It uses the jinja2 and mustache libraries on Python and Node.js, respectively.

120.uploader - Uploader

Upload file from provided URL to cloud storage.

Type: Webapps Languages: Python, Node.js Architecture: x64, arm64

Description

The benchmark implements the common workflow of uploading user-defined data to the persistent cloud storage. It accepts a URL, downloads file contents, and uploads them to the storage. Python implementation uses the standard library requests, while the Node.js version uses the third-party requests library installed with npm.

While 128 MB is technically sufficient for memory size, a larger memory value or a longer timeout might be required for the large input size.

130.crud-api - CRUD API

Simple CRUD application using NoSQL to store application data.

Type: Webapps Languages: Python Architecture: x64, arm64

Description

The benchmark implements a simple CRUD application simulating a webstore cart. It offers three basic methods: add new item (PUT), get an item (GET), and query all items in a cart. It uses the NoSQL storage, with each item stored using cart id as primary key and item id as secondary key. The Python implementation uses cloud-native libraries to access the database.

Multimedia

210.thumbnailer - Thumbnailer

Generate a thumbnail of an image.

Type: Multimedia Languages: Python, Node.js Architecture: x64, arm64

Description

This benchmark implements one of the most common serverless workloads. It downloads an image from the cloud storage, resizes it to a thumbnail size, uploads the new smaller version to the cloud storage, and returns the location to the caller, allowing them to insert the newly created thumbnail. To resize the image, it uses the Pillow and sharp libraries on Python and Node.js, respectively.

Inspired by AWS Lambda tutorial code.

220.video-processing - Video Processing

Add a watermark and generate gif of a video file.

Type: Multimedia Languages: Python Architecture: x64, arm64

Description

The benchmark implements two operations on video files: adding a watermark and creating a gif. Both input and output media are passed through the cloud storage. To process the video, the benchmark uses ffmpeg. The benchmark installs the most recent static binary of ffmpeg provided by John van Sickle.

While 512 MB is technically sufficient and works well for the watermark operations, the large input converts the video into a gif which is more computationally intensive. On AWS, you should expect around ~30 seconds runtime on 1024 MiB allocation.

Utilities

311.compression - Compression

Create a .zip file for a group of files in storage and return to user to download.

Type: Utilities Languages: Python Architecture: x64, arm64

Description

The benchmark implements a common functionality of websites managing file operations - gather a set of files in cloud storage, compress them together, and return a single archive to the user. It implements the .zip file creation with the help of the shutil standard library in Python.

Inference

411.image-recognition - Image Recognition

Image recognition with ResNet and pytorch.

Type: Inference Languages: Python Architecture: x64

Description

The benchmark is inspired by MLPerf and implements image recognition with Resnet50. It downloads the input and model from the storage and uses the CPU-only pytorch library in Python.

The minimal memory amount is set to 768 MiB due to GCP requirements. It works with 512 MiB on AWS.

Important Notes

Warning

This benchmark contains PyTorch which is often too large to fit into a code package. Up to Python 3.7, we can directly ship the dependencies. For Python 3.8, we use an additional zipping step that requires additional setup during the first run, making cold invocations slower. Warm invocations are not affected.

Warning

This benchmark does not work on AWS with Python 3.9 and code package due to excessive code size. While it is possible to ship the benchmark by zipping torchvision and numpy (see benchmarks/400.inference/411.image-recognition/python/package.sh), this significantly affects cold startup. On the lowest supported memory configuration of 512 MB, the cold startup can reach 30 seconds, making HTTP trigger unusable due to 30 second timeout of API gateway. Use Docker deployments for these configurations.

Warning

This benchmark does not work on GCP functions gen1 with Python 3.8+ due to excessive code size. Use container deployments on Google Cloud Run for these configurations.

Scientific

501.graph-pagerank - Graph PageRank

PageRank implementation with igraph.

Type: Scientific Languages: Python Architecture: x64, arm64

Description

The benchmark represents scientific computations offloaded to serverless functions. It uses the python-igraph library to generate an input graph and process it with the PageRank algorithm.

502.graph-mst - Graph MST

Minimum spanning tree (MST) implementation with igraph.

Type: Scientific Languages: Python Architecture: x64, arm64

Description

The benchmark represents scientific computations offloaded to serverless functions. It uses the python-igraph library to generate an input graph and process it with the Minimum Spanning Tree (MST) algorithm.

503.graph-bfs - Graph BFS

Breadth-first search (BFS) implementation with igraph.

Type: Scientific Languages: Python Architecture: x64, arm64

Description

The benchmark represents scientific computations offloaded to serverless functions. It uses the python-igraph library to generate an input graph and process it with the Breadth-First Search (BFS) algorithm.

504.dna-visualisation - DNA Visualization

Creates a visualization data for DNA sequence.

Type: Scientific Languages: Python Architecture: x64, arm64

Description

This benchmark is inspired by the DNAVisualization project and it implements processing the .fasta file with the squiggle Python library.

Serverless Workflows

(WiP) Coming soon!

Applications

(WiP) Coming soon!