SeBS: Serverless Benchmark Suite
FaaS benchmarking suite for serverless functions with automatic build, deployment, and measurements.
SeBS is a diverse suite of FaaS benchmarks that allows automatic performance analysis of commercial and open-source serverless platforms. We provide a suite of benchmark applications in Python, Node.js, Java, and C++ covering workloads from web applications to scientific computing. With automtic experiments, we test and evaluate different components of FaaS systems. SeBS provides support for automatic deployment and invocation of benchmarks on commercial and black-box platforms AWS Lambda, Azure Functions, and Google Cloud Functions. Furthermore, we support the open-source platform OpenWhisk and offer a custom, Docker-based local evaluation platform.
How can SeBS help you?
Are you looking for an experimentation platform to test and analyze the performance of serverless across cloud platforms?
Do you need a set of standardized benchmarks for your serverless experiments and research work?
Do you want a fully automated pipeline for build, deployment, and measurements, with no manual effort?
Then SeBS might just be the tool for you and your work!
See the installation instructions and SeBS tutorial below to learn how to configure SeBS to use selected commercial and open-source serverless systems. Then, take a look at our documentation to see how SeBS can automatically launch serverless functions and entire experiments in the cloud! You can also find details about SeBS design and experimental results in our peer-reviewed publications.
Getting started: how to configure cloud and serverless platforms?
Going deeper: which experiments can be launched to evaluate FaaS platforms?
Modularity: how to extend SeBS with new benchmarks, experiments, and platforms?
Do you have further questions that were not answered by our documentation? Did you encounter trouble installing and using SeBS? Or do you want to use SeBS in your work and you need new features? Join our community on Slack or open a GitHub issue.
Installation
Requirements:
Docker (at least 19)
Python 3.10+ with:
pip
venv
libcurland its headers must be available on your system to installpycurlStandard Linux tools and
zipinstalled
… and that should be all. We currently support Linux and other POSIX systems with Bash available. On Windows, we recommend using WSL.
To install the benchmarks with a support for all platforms, use:
./install.py --aws --azure --gcp --openwhisk --local
It will create a virtual environment in python-venv, and install necessary Python
dependencies and third-party dependencies. To use SeBS, you must first active the new Python
virtual environment:
. python-venv/bin/activate
Now you can deploy serverless experiments :-)
The installation of additional platforms is controlled with the --{platform} and --no-{platform}
switches. Currently, the default behavior for install.py is to install only the
local environment.
To verify the correctness of installation, you can use our regression testing.
Warning
Please do not use SeBS with sudo. There is no requirement to use any superuser permissions. Make sure that your Docker daemon is running and your user has sufficient permissions to use it (see Docker documentation on configuring your user to have non-sudo access to containers). Otherwise, you might see many “Connection refused” and “Permission denied” errors when using SeBS.
Tutorial
We provide a tutorial on basic SeBS functionality in the SeBS-Tutorial repository. You can learn there how to install SeBS, configure it, deploy OpenWhisk on your system, and launch your first experiments.
Publications
When using SeBS, please cite our published work. You can cite our software repository as well, using the citation button on the right.
SeBS has been originally released with the Middleware ‘21 paper. An extended version of our paper is available on arXiv, and you can find more details about our research work in this paper summary.
@inproceedings{copik2021sebs,
author = {Copik, Marcin and Kwasniewski, Grzegorz and Besta, Maciej and Podstawski, Michal and Hoefler, Torsten},
title = {SeBS: A Serverless Benchmark Suite for Function-as-a-Service Computing},
year = {2021},
isbn = {9781450385343},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3464298.3476133},
doi = {10.1145/3464298.3476133},
booktitle = {Proceedings of the 22nd International Middleware Conference},
pages = {64–78},
numpages = {15},
keywords = {benchmark, serverless, FaaS, function-as-a-service},
location = {Qu\'{e}bec city, Canada},
series = {Middleware '21}
}
The SeBS-Flow paper published at EuroSys’25
extends SeBS with support for serverless workflows and NoSQL database.
You can find workflow benchmarks on the feature/workflows branch (AWS, Azure, GCP).
BibTeX citation for the SeBS-Flow paper.
@inproceedings{10.1145/3689031.3717465,
author = {Schmid, Larissa and Copik, Marcin and Calotoiu, Alexandru and Brandner, Laurin and Koziolek, Anne and Hoefler, Torsten},
title = {SeBS-Flow: Benchmarking Serverless Cloud Function Workflows},
year = {2025},
isbn = {9798400711961},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3689031.3717465},
doi = {10.1145/3689031.3717465},
booktitle = {Proceedings of the Twentieth European Conference on Computer Systems},
pages = {902–920},
numpages = {19},
keywords = {benchmark, faas, function-as-a-service, orchestration, serverless, serverless DAG, workflow},
location = {Rotterdam, Netherlands},
series = {EuroSys '25}
}
The SeBS 2.0 workshop paper published at SESAME @ EuroSys’25 provides an overview of new and ongoing contributions to SeBS - benchmarks, platforms, languages.
BibTeX citation for the SeBS 2.0 paper.
@inproceedings{10.1145/3721465.3721867,
author = {Copik, Marcin and Calotoiu, Alexandru and Hoefler, Torsten},
title = {SeBS 2.0: Keeping up with the Clouds},
year = {2025},
isbn = {9798400715570},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3721465.3721867},
doi = {10.1145/3721465.3721867},
booktitle = {Proceedings of the 3rd Workshop on SErverless Systems, Applications and MEthodologies},
pages = {42–44},
numpages = {3},
keywords = {Benchmark, FaaS, Function-as-a-Service, Serverless},
location = {Rotterdam, Netherlands},
series = {SESAME' 25}
}
Development
We welcome new contributions! When extending SeBS, please check first contributor guidelines to learn the expected code style. Please feel free to get in touch with us - we are happy to provide guidance and help you to implement new features in SeBS.
Feature Branches
We provide several experimental features that have not yet been merged into the main branch. You can use them to get early access to upcoming benchmarks, platforms, and experiments.
However, they can be missing some of the features from the master branch.
feature/workflowswith serverless workflows benchmarks (AWS, Azure, GCP).feature_fissionwith support for Fission platform.oanarosca/triggerswith queue and storage triggers (AWS, Azure, GCP).