Game Server Bases

This is a framework for building game server images to run in Linux containers. You can use them with Podman, Docker, Kubernetes, or whatever you like. The idea is that the functionality you would desire on one game server (like automatic updates, or the ability to send messages into the game server when people need to log off for one of those updates), are exactly the kinds of things you would want to do on another game server. One of those servers might support RCON, the other might have a TCP socket you can throw string-like bytes into. One of those game servers might notify you of updates through an API, the other might be hosted on Steam where there's a pretty consistent way to check for updates.

So, rather than copy and paste a bunch of code, I built this generic framework on which to build game images, keeping the code required for each game image small. There are a few conventions to be aware of:

• The game server data itself should be in a volume, rather than in the image itself. This means that a game update doesn't require an image update, but that an image update may not require downloading all of the layers with the game data.
• Ideally, the save data should be in a separate volume from the server data. This helps with backup considerations.
• Any customizations beyond the code to wire up getting the server to run (and anything present in the saves or mounted-in configs) should be represented as code as well. This means that game server environments should be reproducible across environments with only the image, the save data, and the controlled environmental factors like environment variables and a single directory's configs.
• Scripts that should not be executed directly, but rather sourced, should start with an underscore.
• Since we're doing everything in containers, with almost entirely just system packages, we can abuse /usr/local and put everything in there.
• /usr/local/lib/$GAME_NAME in particular is used to house game-specific snippets, stubs, and scripts for use by the framework.
• Scripts that are executed should all source /usr/local/lib/game/common.sh and leverage the log and noisy functions, rather than echo or set -e or set -x. Sourced functions should all assume they're sourced from a script that also sources this common bit. Scripts intended to be very brief needn't source the common file, especially if they don't source anything.
  • log accepts a few arguments. -l LEVEL or --level=LEVEL will allow you to set the log level of the message (see the LOG_LEVEL environment variable below to better understand the significance). -n or --no-fail will prevent non-zero exit after logging messages of ERR or higher level (the default behavior).
  • noisy accepts a few arguments. By default, it enables set -e right before executing the command and unsets it after.
    • Passing -n or --no-fail will disable invocation with set -e. This prevents exiting the script calling the function if the command fails.
    • Passing -l $level or --level=$level will adjust the log level of the command execution notice. This works like set -x, except that the output is subject to the overall framework's LOG_LEVEL to simplify debugging in certain environments. The default level for noisy messages is INFO.
    • Passing -c "string" or --censor="string" will censor any instances of string from the echoed output. Note that this might leak your secrets if another instance of the secret is in the log line, and it doesn't modify the command output at all. It's used, for the framework itself, to censor login information for steamcmd.sh when provided.
    • Passing -o or --output will capture the stdout of the command and redirect it to the log, respecting the log level of the rest of the command.
    • Passing -q or --quiet will capture stderr and either send it to the log (if --output is also true) or send it to /dev/null.
    • Also worth highlighting that -- works as you'd expect for noisy, consuming the rest of the command line for the noisily run command rather than continuing to parse noisy options. That's probably what you want to use most of the time.
• Steam game servers can leverage the steamcmd family of base images and automatically inherit installation and update code, if they set the image's STEAM_GAME_ID environment variable to the appropriate Steam ID for that game server.
  • The steamcmd-wine and steamcmd-wine-gui images are for Steam game servers that expect to be running in Windows. The GUI image will spin up a fake 800x600 display with Xvfb and launch the game with that output set, but doesn't provide a way to interact with it.
  • Additional environment variables are provided for the steamcmd family to handle these installations, including the capability to log in with a specific username and password (if required to download the server), as well as a method for staging Steam Workshop items (potentially from a different Steam ID, as sometimes those mods are not available for the servers).
• Everything about the framework assumes that some external system will strive to keep the game running, restarting it when it exits - even successfully, as part of normal operation. That is, it doesn't stop the game server process, perform an update, and then start the game server process back up. It is preferred to gracefully shut down the game, expecting an installation or update to be performed at every container start. An external system like docker with --restart unless-stopped, a Podman systemd unit, or a Kubernetes StatefulSet controller should handle starting it up after it's stopped. Starting it should handle initial installation exactly the same as an update.
• Many systems will work with the correct scripts, stubs, and environment variables in place - but will quietly skip over if not provided. A good example is Discord webhook integration. There are many points in the scripts that will call discord.sh with some message that might be useful regarding game server state, but only if the DISCORD_WEBHOOK_URL environment variable is provided.
  • One very useful example is send.sh. This script will send a command to the game server, such as a request for a graceful save and exit of the game world. It works by looking for a game-specific implementation of _send.sh in /usr/local/lib/$GAME_NAME/ and source that. If it can't find a suitable _send.sh, it checks if both RCON_PORT and RCON_PASSWORD are set and will use rcon-cli instead. If NC_PORT is set, it will use netcat to send the arguments in a raw TCP message. This flexibility means that, regardless of how we send commands to a server, we can expect to provide just enough information to wire it up and let the rest of the framework know that send.sh provides a consistent abstraction for sending commands to the game server.

You'll find that this kind of works like the many game server images that use supervisord as their entrypoint, monitoring and running many processes at once. This system has a few distinct advantages, though:

• By delegating lifecycle events back to our external controller, environmental specific details can be delegated to the capabilities of that controller. For example, health checks can forcefully kill the server container when they fail while regular restarts can be graceful.
• We can run as a non-root user at all times.
• Health data and metrics can directly use the lifecycle controls of your orchestrator - that is, Prometheus in k8s knows how often the pod is in a Not Ready state and you don't need to wire it up to reach the supervisord metrics endpoint.

Minimal requirements for a game image

If your game server is installable from Steam, the requirements are quite low:

• Create a Containerfile based on the appropriate base in the steamcmd family, setting the following environment variables:
  • GAME_NAME should be set to the all-lowercase name of the game with no spaces - it is part of identifying what folder the game-specific files go in
  • STEAM_GAME_ID should be set to the Steam ID of the game server (you can get this from the library or store page URLs)
• Add to that image, usually via COPY as USER 0, at least a /usr/local/lib/$GAME_NAME/start.sh file. This file will be called, not sourced, from the framework start script - so you should source the common file if you're doing anything worth logging or recording. It's ideal if the last line in your script, the one that starts the game server itself, uses exec to launch the game binary (or wine).

That's it! More advanced functionality is possible, and useful for various scenarios, via environment variables and stubs/scripts.

For non-Steam game servers, the primary requirements are a little different.

• Instead of specifying the STEAM_GAME_ID environment variable, you have to implement a stub to install the game at /usr/local/lib/$GAME_NAME/_install.sh.
• Either set the GAME_UPDATE environment variable to false or implement a stub at /usr/local/lib/$GAME_NAME/_update-available.sh that exits 0 when an update for your game server is discoverable and non-zero when it's not.
  • An example of this would be the PaperMC API returning an ordered list of available versions. If we have somehow recorded (in a stateful directory, perhaps adjacent to the save) or can query the latest 1.20 version of the JAR downloaded, we could run a command like the following to compare the two:

    latest_available="$(curl -s "https://api.papermc.io/v2/projects/paper/version_group/$PAPERMC_VERSION_GROUP" | jq -r '.versions[-1]')"
    if [ -n "$latest_available" ] && [ "$latest_installed" != "$latest_available" ]; then
      exit 0
    else
      exit 1
    fi
    

game Base Details

Called Scripts

Note that all of these scripts should be executable, meaning you've run chmod +x on them.

• /usr/local/lib/$GAME_NAME/start.sh
  • This should start the game server and expect to stay running forever. You can include some cleanup tasks after invoking the command to start the server, or you could leave that to the stop event. I think it's better to have a relatively short script that uses exec to reduce the number of running shells.
• /usr/local/lib/$GAME_NAME/stop.sh
  • This should gracefully stop the server. It doesn't need to wait, if the request is asynchronous (such as sending a command via send.sh). It is always called asynchronously from /usr/local/bin/stop.sh, so it should be eager to exit. If you don't have a way to gracefully stop the server other than sending the PID a SIGTERM, don't bother implementing this - stop.sh will do it on its own.
• /usr/local/lib/$GAME_NAME/players-online.sh
  • This should determine if there are players on the server, and if there are none then it should exit with a non-zero code. If there are players, it should exit with zero. This is useful for holding off on a shutdown request, knowing when to warn players that an update is available, etc. - and not doing it blindly.

Sourced Stubs

Stubs don't have to be executable, they simply need to exist.

• /usr/local/lib/$GAME_NAME/_install.sh
  • This is used if your game has a specific installation you'd like to call. Note that it should be capable of installing from scratch as well as installing an update if it's necessary - but shouldn't install an update if it's not necessary. That is, you should find a good way to track what version you're installing.
• /usr/local/lib/$GAME_NAME/_server-healthy.sh
  • If there's any more health information that your server can emanate other than "the process is running," this is where you should put logic to make that determination. I tend to like using send.sh to send a command to the server, then parse its response for something that indicates that it's definitely running. Although this snippet is sourced, it's expected to be capable of exiting the parent script. Use zero or nothing to indicate that the server is healthy, exit with non-zero to indicate that it's unhealthy.
• /usr/local/lib/$GAME_NAME/_pre-logs.sh
  • This is useful to set environment variables of more complex types than simple strings. If your game logs to a file and you're using the logging-related environment variables to wire up log execution, you can put a script here to define some bash arrays for LOGDISCRIMINATORS and EXTRA_LOG_SED to pass command-line arguments to find and sed respectively. The logs.sh implementation uses find to look for log files to start tailing, and uses sed to prepend the log lines with the filename it found them in. Think of reasonable values for LOGDISCRIMINATORS as things like ! -name 'verbose.log' -name '*.log' to select all files ending in .log that aren't verbose.log, and values for EXTRA_LOG_SED like -e 's/some[r]egex/somereplacement/g' to perform a regex replace before dumping log output to the container's stdout.
• /usr/local/lib/$GAME_NAME/_discord.sh
  • This is useful to do some extra logic to uniquely identify the server making the Discord webhook, if it's shared between multiple server instances. I've been using it to parse things like friendly server names from configuration files. There's a function that discord.sh calls named discord_prelude that is expected to output something to put in front of all Discord messages, and here is where you would put something like "${server_name}: " for this purpose.
• /usr/local/lib/$GAME_NAME/_send.sh
  • Discussed above, this is for custom implementations to send server commands. It's given precidence over rcon-cli and netcat, even if those environment variables are defined. If your server requires special handling for netcat connections, or needs a totally unique method to send messages (like redirected stdin through a FIFO socket), then this is where you'd implement the logic to back the more generic send.sh to send those commands.
• /usr/local/lib/$GAME_NAME/_say.sh
  • Where sending a chat message to the players on the server is a special case of sending a command to the server, this snippet is useful to specify how to format the command to send a message. I tried to provide a generic way of doing this but game servers are crazy with this stuff.
• /usr/local/lib/$GAME_NAME/_update-available.sh
  • This stub is only necessary for games that you've implemented a game-specific _install.sh for, with GAME_UPDATE and STOP_IF_UPDATE_AVAILABLE both set to true. It should exit with 0 if an update is available or 1 if one isn't, or update suitability can't be determined (some API down or whatever). There is a generic implementation for steam games.

Environment Variables

• GAME_NAME
  • This is the all-lower-case, no-spaces name of the game used for internal directory organization. /game/$GAME_NAME is where the game is expected to be installed, but not necessarily the location of the actual binary. /usr/local/lib/$GAME_NAME is where snippets are expected to be. Keeping these in this alignment is a useful convention to keep track of what should be in the image vs what should be in a volume.
• GAME_UPDATE (default true)
  • Somewhat strangely, since we're using a single script for both install and update, this variable determines whether the container instance should try to install and update the game at launch. If you've side-loaded the game data itself, or don't believe in my "the game should be in a volume" approach, then set this to false.
• START_TIMEOUT_SECONDS (default 300)
  • How long, after firing off the game-specific start.sh, should the entrypoint expect server-healthy.sh to return zero. This happens after installation/update and only relates to the actual starting of the server itself. If your server takes a particularly long time to start, it makes sense to extend this higher.
• STOP_GRACE_PERIOD_SECONDS (default 60)
  • How long, after sending your game process (and all related processes) a SIGTERM should we wait for them to stop until we issue a SIGKILL. This only takes effect if your game-specific stop.sh implementation has already timed out, or is not present.
• STOP_IF_UPDATE_AVAILABLE (default true)
  • Indicates whether updates should be continuously checked for. If one is found, normal stop.sh implementation will perform a graceful termination (after players are offline)
• STOP_PLAYER_TIMEOUT_SECONDS (default 300)
  • How long, after any normal stop.sh call, to wait for players to log off before forcing them off. If you set this to 0 without a game-specific stop.sh, the logic is such that the players will simply be booted offline without much fuss. This is useful if you can tell if players are online from a query port, but can't send them a message to log on, and haven't wired up a Discord webhook to notify them out of band.
• STOP_TIMEOUT_SECONDS (default 300)
  • How long, after calling the game-specific stop.sh, to wait until the game stops running before deciding to move on and send it a SIGTERM (triggering the above timeout).
• AUTO_STOP (default: false)
  • Whether the game should stop on a cron-like schedule. Game servers with memory leaks or other performance degradation benefit from setting this to true.
• AUTO_STOP_CRON_UTC (default: 0 4 * * *)
  • The schedule on which to perform the auto-stop if enabled above. Uses a robust Cron parsing library, so you can do crafty things like 0 2,14 * * * to reboot twice per day. Always uses UTC. Time zones are hard, time zones in containers are harder.
• RELATIVE_CONFIG_DIR
  • The location, relative to /game, to copy your configuration files to, from /config where they're expected to be mounted. You can also just specify an absolute path. Note that the way this is done works gracefully with Kubernetes ConfigMap resources, despite the strange way they mount into /config that can trip some game servers up.
• WATCH_LOGS (default: false)
  • Whether to fire off the log watcher script. See _pre-logs.sh above for more information about how it works and how you can adjust behavior.
• LOGDIR
  • The directory to watch for log files. Must be set if WATCH_LOGS is true.
• RCON_PORT
  • The TCP port your game server listens on for RCON. It's not necessary to expose it outside of the container, but if you're able to issue commands with it then send.sh will respect this setting by default.
• RCON_PASSWORD
  • The password to connect to your RCON port with.
• NC_PORT
  • The port to send raw TCP commands to for send.sh, if no game-specific or RCON implementation.
• DISCORD_WEBHOOK_URL
  • The full URL to connect to for issuing an HTTP POST to Discord. If provided, will notify on various lifecycle events.
• DISCORD_USERNAME (default: Game Notification)
  • The username to publish with at the webhook URL.
• DISCORD_AT_HERE (default: false)
  • When set to true, appends @here to Discord webhook messages requesting that players log off. This is useful for games who cannot send messages in game. Only do this if the channel for your webhook is opt-in, probably.
• LOG_LEVEL (default: INFO)
  • The rsyslog keyword for the severity level at which you want the scripts themselves to emit logs to stdout. Note that this doesn't affect game server logging, only script logging. Options are EMERG, ALERT, CRIT, ERR, WARN, NOTICE, INFO, and DEBUG. Setting this to EMERG emits almost zero logs, setting it to DEBUG emits many logs.

steamcmd Base Details

This image behaves mostly like game, except that it is wired up to use steamcmd.sh to handle installation, updates, and mod configuration via the Steam Workshop. All of the above applies, and beyond that you should skip an _install.sh stub or update-available.sh script, providing at least the STEAM_GAME_ID environment variable. There are a few other things performed on this image family, partly because of the fact that the installation is already wired up for you.

Sourced Stubs

• /usr/local/lib/$GAME_NAME/_pre-inst.sh
  • This is sourced as some of the variables are being built out. Consider this your time to customize things such as runtime modifications to STEAM_WORKSHOP_ITEMS or other things. If you need to read a config file to template where to place a workshop item is the main thing I've used this for.
• /usr/local/lib/$GAME_NAME/_post-inst.sh
  • This is sourced after all game and workshop items have finished downloading and being installed/linked into place in the steamcmd installer. Because of the way the installation is called, if GAME_UPDATE is true then anything that runs here will run before configs are copied into place, before the server starts, but after the installation is completely finished. This is a good time to do side-loading of mods or the like for a given game.

Environment Variables

• STEAM_GAME_ID
  • This is critical - none of the steamcmd images will work right without it. It's used by all common steamdb parsing utilities, used by the API, used by steamcmd.sh itself to install and update the game. You can see an example here, where 2278520 is the ID we want here.
• STEAM_VALIDATE (default: true)
  • Also runs a steamcmd validation on the install at every container startup, but only if GAME_UPDATE is true.
• STEAM_ACCOUNT and STEAM_PASSWORD
  • Overrides anonymous login when used together. Be careful how you supply these to the server 🙂
• STEAM_WORKSHOP_ITEMS
  • A whitespace-separated list of itemid:relativepath pairs. That is, if you wanted to use the workshop item of Reforged Eden, its ID is 2550354956. If we wanted it in a place where our game, installed in /game/$GAME_NAME could reach it for an installed scenario, we might want to provide it a relative path like this (since we're not really starting the game with Steam): 2550354956:Content/Scenarios/RE_1_11. Now our configuration, for this particular game, can be set to load the scenario named RE_1_11 to run Reforged Eden for version 1.11. Note that this doesn't do the wiring for you, it just gives you a way to reference the location reproducibly.
• STEAM_WORKSHOP_GAME_ID
  • In the case of the example above, the "Game Scenario" we want the server to run is distributed as a Steam Workshop item. The catch is that it's in the workshop for the game, not the server. So, when using steamcmd to install the server and the workshop item, it can be useful to download a workshop item for a different game. In this particular game's case, the STEAM_GAME_ID we set was 530870 for the dedicated server, while the STEAM_WORKSHOP_GAME_ID is 383120 - the Steam app ID for the game client.
• EXTRA_STEAMCMD_ARGS
  • A string - which will regrettably be unable to include proper escaping of spaces - which is inserted immediately after steamcmd.sh in the steamcmd family's _install.sh snippet. This means you can provide direct args to steamcmd.sh before beginning to parse its +-formatted download-related arguments. Note that this is already used by the steamcmd-wine image to force Windows platform installation from Linux.
• FINAL_STEAMCMD_ARGS
  • Another troublesome string that will be split on whitespace, but this time added to the end of the steamcmd.sh command line, right before +quit, to let you inject some extra pizzazz into the call.
• STEAM_BRANCH (default: public)
  • This is the branch of the game that you're installing, likely via FINAL_STEAMCMD_ARGS="-beta public-test" or some such. This is the endpoint queried at the API for the latest Build ID to determine if a Steam game update is necessary. These are unique per game, but the default for most of them is public at the Steam API. If you provide a -beta argument to steamcmd.sh through FINAL_STEAMCMD_ARGS, this will need to be aligned to the correct branch for that beta. For example, this command will show you the branches available in the Valheim Dedicated Server depot:

    curl https://api.steamcmd.net/v1/info/896660 | jq -r '.data."896660".depots.branches | keys[]'
    

And for the steamcmd family of images, that's pretty much it. The steamcmd-wine image adds Wine and uses a few other things that might prove useful to you, but doesn't expect any more of you. Ensure you use wine in the exec line at the end of your game-specific start.sh, if you want to use it at least. steamcmd-wine-gui just bolts on that Xvfb server and exports DISPLAY before going on to do the same thing as the steamcmd-wine image does.

backups Details

There's one other image in this bases repo that's not quite like the others. backups is an image I use myself, because I happen to deploy all of these images inside Kubernetes. Specifically, I use OKD - and I run my storage with Rook. This gives me really nice zero-copy snapshot behavior, including full image-layering, copy-on-write behavior. So, instead of a normal "rsync this directory a billion times, totally destroying the write endurance of my SSD cluster," I decided to implement a backup mechanism that took advantage of this setup. If this doesn't work for you, you'll have to write your own backup process - though admittedly, I personally think it's better if this is done outside your image (using Velero on Kubernetes, or just a simple Cron job that runs a Docker or Podman backup).

Ultimately, because I'm doing backups this way, it may not be as useful to you - but I'll go ahead and document it anyways. This image expects to run inside the cluster, with a ServiceAccount that has create, get, delete, and list on volumesnapshots in the snapshot.storage.k8s.io API Group. From there, you just set the appropriate environment variables for it to do its thing.

Environment Variables

• SAVE_PVC
  • The name of the PVC you want to back up.
• MAX_HOURLY_BACKUPS (default: 3)
  • The number of hourly backups you want to keep. It will create a backup ever hour, so this is about how old you want them to be before being pruned off.
• MAX_DAILY_BACKUPS (default: 2)
  • The number of daily backups you want to keep.
• MAX_STARTUP_BACKUPS (default: 3)
  • The number of startup-specific backups you want to keep. If the backup container is in its own pod, this isn't much help. If you run it as an initContainer, perhaps using the new Sidecar Containers behavior, then when your Pod gets schwacked by a lifecycle policy that determined that the main container in the pod died (if you, hypothetically, kill the container for normal things like scheduled shutdowns and updates), this will take a snapshot just before your server starts. That's pretty handy if you ask me.
• BACKUP_THRESHOLD (default: 30m)
  • A string in the form of #h#m#s that is parsed to enforce a cooldown period between backups, mostly for preventing backups from being aged out if a lot of stars align and many come at once.
• START_BACKUP (default: true)
  • Determines whether a backup will be run as the script initializes. Skipped if set to a non-truthy string.
• SCHEDULED_BACKUPS (default: true)
  • Determines whether the script will gracefully exit after performing its startup backup (if enabled), or wait for the next hourly/daily backup time period. This should be set to a non-truthy string if you're using it as a regular initContainer, instead of a proper sidecar.
• CRON_JITTER_SECONDS (default: 900)
  • How much to jitter the backup time by, in seconds. This is useful if you're hypothetically running a bunch of these servers in a single Kubernetes cluster and don't want to bombard your snapshot controller all at once.
• SNAPSHOT_CLASS (default: csi-rbdplugin-snapclass)
  • The name of the VolumeSnapshotClass to be used when creating the backups.

At one point I had environment variables for providing the Cron strings for backups but I decided that was too messy and easy to mess up, so I just figured I'd say "this takes hourly, daily, and startup backups - and can prune them based on a simple integer." If you want to rework the script to do your own bidding, have at.

Contributors

So far just me! If you, now that there's documentation, want to modify these you're welcome to. If you have a really cool improvement that I'd appreciate, I'm happy to hear from you. My email address is the root domain of this repository at gmail. If you want to address some use case I don't have, you're welcome to fork this. Do whatever, I wrote this code for me. (see also)