tech, life & music.

I'm blocking private IPv4 addresses on my servers by default from now on

2025-11-22T00:00:00Z

Yesterday, I was setting up a new Raspberry Pi Zero 2W in my home network. I'm going to use it to revive an old Brother printer I have lying around and install CUPS on it, so that I can use said printer as a network printer on the cheap.

I just had the Pi flashed with a headless Ubuntu, and preconfigured it to connect to my home network via WiFi. I even gave it an example.local hostname so that I can talk to it as soon as I plug it in. But the hostname didn't seem to be working when I turned it on; I couldn't SSH into the Pi from another computer in my network. That's not a big problem; I didn't know it's resolved IPv4 address, but I can always just run a network scan for it.

So I jumped onto a terminal and did just that:

nmap -p22 192.168.1.0/24

Oops.

What I didn't realize was that that terminal was currently live, tunneled into one of my remote servers via ssh. This particular one was a baremetal I was renting off Hetzner. Never mind that I had a live ssh terminal left open, shame on my cow!

That must've rung alarm bells pretty swiftly over on Hetzner, because I got my server locked and isolated as a result.

The email I got from Hetzner

Protecting myself from myself

As recommended by the support team I talked to, it's probably a good idea to block outgoing network packets from my box to private address spaces, (defined as RFC1918) especially if I don't expect my machine to be talking to other servers inside the same space anyway. This particular one wasn't, so I didn't have a problem with it.

You can do this by setting an outgoing block in your iptables:

iptables -A OUTPUT -d 10.0.0.0/8 -j DROP
iptables -A OUTPUT -d 172.16.0.0/12 -j DROP
iptables -A OUTPUT -d 192.168.0.0/16 -j DROP

Or, if you're like me and prefer using ufw:

ufw deny out to 10.0.0.0/8
ufw deny out to 172.16.0.0/12
ufw deny out to 192.168.0.0/16

I'm likely adding those commands into my initialization playbook, which I follow whenever I'm setting up baremetals from scratch.

Visiting localhost from a physical Android device

2025-04-07T00:00:00Z

I recently had to test a web app I was developing on an real mobile phone, so I had to figure out how to (1) run the development server on my computer, and (2) access it from the Android phone I had in my hand.

Most of the how-tos I found online were either outdated, or felt like they were a bit too complicated for what I was trying to achieve. I did not want to have to switch my computer network every time I needed to test something on my phone; just thinking about having to switch back and forth between WiFi and a mobile hotspot tires me out.

I'd rather have a solution that I set up once, and then just use it from then on out.

So how then?

You might already have adb installed on your computer, but if you don't, you can install it from the Android SDK Platform Tools site.

Next, you need to enable USB debugging on your Android device. This lets you communicate to your phone from your computer over USB. The steps to do that can be found here.

After all that is done, you're ready to go. Connect your phone to your computer via USB --- you can optionally check if your phone is visible to your computer by running the following command:

adb devices

If you see your device listed, you're good to go.

Finally, whenever you want to access your localhost server from your phone, you just need to run the following command:

adb reverse tcp:8080 tcp:8080

This assumes that your development server is running on port 8080. Now, as long as your phone is connected to your computer via USB, you can use any browser on your phone to access http://localhost:8080 and it will be forwarded to your computer's localhost server.

Patching Berkeley Mono v2 with Nerd Font glyphs

2025-02-22T00:00:00Z

I love me a great font, and considering that I look at a text editor at least half of the time I'm in front of a computer every day (which is a lot!), I think it's worthwhile making sure I like what I'm looking at. Since the start of 2025, that font's been Berkeley Mono, which has recently just been updated to version 2.

Unfortunately, this one doesn't come with Nerd Font glyphs (like with other fonts that come with paid licenses), so I need to patch it myself. Not a problem, since there's a handy dandy Nerd Font patcher tool available. I've used it before, and having it available as a container makes it even easier to use.

There's a problem with the .OTF version of Berkeley Mono v2 that results in a botched font after patching. You'll want to use the .TTF version if you're trying this yourself.

If you're trying it yourself, this is the command I used to patch it. It should put in all icons available via Nerd Fonts. (Note that I did this with the semi-condensed version, which I kinda like best.)

docker run --rm \
  -v ./original:/in \
  -v ./patched:/out \
  nerdfonts/patcher \
  --complete \
  --single-width-glyphs \
  --adjust-line-height

This assumes my Berkeley Mono v2 font files are in the ./original directory, of course. Patched fonts will be put in the ./patched directory as well. Adjust these to your needs.

Configuring Paperless-ngx to monitor emails from a Proton Mail account

2024-01-18T00:00:00Z

I've been setting up a Paperless-ngx instance on my homelab as part of my 2024 resolution to immerse myself more into managing a baremetal server. One of the things I want it to do is to monitor my email inbox for documents I receive regularly (think bills, receipts, etc.) so that they are automatically fetched into the digital archive.

However, I use Proton Mail, and as part of their security measures, they don't provide direct access to their IMAP or SMTP servers like how you'd expect with most email providers. I do appreciate this, but it does make it difficult to hook up Paperless-ngx to monitor my mail.

How I got it to work

Proton does provide a bridge that connects to their servers on your behalf, and exposes an IMAP and SMTP server that you can connect to. Simply put, this acts like a middleman between your email client and Proton's servers.

However, the bridge has a hard requirement to only accept connections from 127.0.0.1 (or localhost). This is a problem since I run my Paperless-ngx stack in a Docker container as part of a Compose stack, so network packets from the container will look like they're coming from somewhere external to my server, even though they're actually running in the same machine. This is complicated even more by a requirement I have to maintain a separate network for my containers, so the bridge and my apps aren't even going to be running in the same network.

An interesting project I came across is the ProtonMail IMAP/SMTP Bridge Docker Container, which wraps the bridge in a Docker container. But the real magic it does is that it uses socat to forward network packets the container receives to the actual ports the bridge is listening on. This will make it appear to the bridge that the packets are coming from localhost.

So making the two containers work together can be as simple as:

networks:
  my-network:
    # ...

services:
  paperless:
    image: ghcr.io/paperless-ngx/paperless-ngx:latest
    restart: unless-stopped
    networks:
      - my-network

  proton-bridge:
    image: shenxn/protonmail-bridge-docker:latest
    restart: unless-stopped
    networks:
      - my-network

The bridge will be visible to the Paperless-ngx container via host name proton-bridge, and will be listening on the default ports 25 and 143 (not 1025 and 1143).

I have to admit it took me too long to finally get this working

Caveats

One very obvious consideration you have to make is that you're potentially exposing your Proton Mail inbox to more consumers than what it was originally designed for. There's tons of discussion on this topic online, so I'm not going to go into that here. What you should know is that by doing this, any other entity in the my-network network can potentially read your emails.

Another thing I had to do was patch the bridge container so that the base build image uses golang:1.20 instead of what it's currently using. I haven't dived too deeply into it, but the current master on project does not build, since the Proton bridge codebase now seems to require at least Go 1.20.

Finally, when registering the IMAP server to Paperless-ngx through the web UI, I couldn't get it to connect successfully with either SSL or STARTTLS. You might have noticed that the connection above had No encryption set.

If you go through the steps to register your Proton Mail account to the bridge, you'll confirm that the bridge does protect the IMAP server using STARTTLS. I should investigate this further, but it may be that this is a side effect of using socat to route network packets to the bridge.

I'm trying out Nobara Linux for my gaming PC

2023-12-22T00:00:00Z

I've been very vocal about wanting to try it out before, but I never got around to it. Yesterday, I finally did: I reformatted my gaming PC and installed a Linux distro.

Path of Exile is first on my must-install list

Nobara?

I went with Nobara Linux 38. As much as I'd like to say that I'm comfortable with getting my hands dirty with all the techno mumbo-jumbo that notoriously comes with working with Linux, I wouldn't trust myself with the stuff that make gaming actually work. Let's leave that to the experts.

Nobara is a modified Fedora distro that is geared towards gaming. It has a lot of customizations and hand-picked tweaks that are supposed to make your games run better on it. It was created by Thomas Crider aka GloriousEggroll. Among other things, he's also the creator of Proton-GE, a custom version of Valve's Proton with an emphasis on being tech-forward.

It also comes with a welcome wizard that was very helpful in setting up the stuff I needed, like OBS Studio.

How did the install go?

It definitely wasn't without problems, but that's not as bad as it sounds.

I was coming from a Windows 10 machine, and I used Rufus to create a bootable USB drive containing Nobara. I used to use Etcher for this, but it seems to be creating corrupted boot disks with Nobara, so I had to drop that. If you want to try it out yourself, you can grab a Nobara ISO you can use from here.

I really wanted to install the KDE/Plasma version because of the customization options I keep hearing about, but for some reason it just won't run correctly. Booting up Nobara 38 KDE, I'd always end up with a frozen screen a few minutes into a session.

So I went with the GNOME version instead. This installed without a hitch, and is the desktop environment in the image I have above. This is working pretty well so far.

I did try to install the KDE version from the GNOME installation, following the steps on this page, but I ended up with a borked system after all of it. Instead of frozen screens, I'd end up with black screens after a few minutes into a session. Following the troubleshooting steps on the site didn't really lead me anywhere, because the commands didn't quite work. dnf wouldn't let me run the following commands from the page because of protections in place:

# This doesn't work for me.
sudo dnf remove *nvidia*
sudo dnf update nobara-login --refresh

In any case, I've decided to stick with the GNOME version for now. I'll try to switch over to KDE/Plasma once Nobara 39 is out, which chatter seems to suggest is going to be soon.

How's the gaming?

I've only given it a quick whirl so far, but I'm somewhat happy with the results.

Part of the reason why I wanted to try this out with Path of Exile first is because:

it doesn't have a native Linux client: I have to use Proton to run it,
I normally play PoE with a bunch of other tools alongside it, like Path of Building and Awakened PoE Trade, so I needed to make sure these also work in-game, and
it's not Crysis, but PoE can be somewhat demanding with lots of particles, effects, and bloom.

I gave that a whirl last night, and I have to say: I kinda feel like this is actually playing better than when I was running it on Windows 10.

I'm giving it a few weeks with various other games though, just to make sure it's not placebo or anything.

Copying files to a volume on Fly.io

2023-01-19T00:00:00Z

Today, I finally did something I've been meaning to do since last year: I migrated my Vaultwarden installation from Amazon ECS to Fly.io.

The migration really was very simple and mostly painless to do — the only part that slowed me down was figuring out how to copy files up to a (disk) volume I've set up on the Fly.io platform. Vaultwarden uses SQLite, so all of its data was on the file system. For a complete migration, I will need to copy the files from my old installation to the new one.

Given a Fly.io app:

app = "my-sample-app"

[build]
  dockerfile = "Dockerfile"

[mount]
  source = "my_volume"
  destination = "/data"

We have a single volume my_volume mounted to our app at /data. We might be able to scp into the app but Fly.io apps now don't automatically come with a static IPv4 address. You still can get an IPv4 address, but this will cost USD 2.00/month.

I wanted to look for a way to do this with just default resources. Unfortunately, it also gets a bit complicated.

So, how?

First, we grab a Wireguard config file:

$  fly wireguard create

This creates a config file on your machine which you can then import into Wireguard. Doing so gives you network visibility to your app via a private VPN connection. While connected to the VPN, your app will be available at <app-name>.internal. (For example, the app above will be at my-sample-app.internal).

Next, we create a (temporary) SSH key:

$  fly ssh issue

This creates a pair of SSH keys on your machine that you can use to connect to your app. For example, my-app-key and my-app-key-cert.pub. We're only interested in your private key here, not the one with a .pub extension.

To test connectivity, just run ssh root@<app-name>.internal -i <your-key-name>. For example:

$  ssh root@my-sample-app.internal -i my-app-key

If everything is OK, you should now be in a tunneled SSH session inside your app. Exit the session by typing exit.

Since we can now ssh into our app normally (i.e. without using fly ssh console), we can now also use the same credentials to run scp to copy files to/fro our app.

For example, to copy foo.txt from my machine's current directory to /data/foo.txt in the app (and therefore, into the volume):

$  scp foo.txt root@my-sample-app.internal:/data/foo.txt

Caveats

Something that you will have to make sure of is that the container that your app is running actually has scp available to it. Otherwise, the operation will fail.

If that's not the case, you can generally install it using something similar to this in your Dockerfile:

RUN apt-get update && \
    apt-get install -y openssh-client

I used ChatGPT instead of Stack Overflow today

2022-12-05T00:00:00Z

There's been no shortage of people harping about the amazing things you can do with OpenAI's ChatGPT. I admit to losing exorbitant amounts of sleep over the weekend because I was trying to "break" it --- I kept asking it practical but somewhat obscure technical questions to see what it can say.

Here's an example:

ChatGPT seems to know how to use Rust to read state from a Raspberry Pi's GPIO pins

This is amazing. I personally don't know how to do that myself, but even if this response from ChatGPT has inaccuracies:

it's pointed me to the rppal crate which looks to be completely within context of what we're trying to do (crate documentation here)
it's specified that the example it's given is for version 0.12.0, so I can work towards grokking this in comparison to the latest version 0.14.1 at the time of this writing
it's shown how to get a handle on a specific pin, and even configured it to (seemingly) behave as an input GPIO pin, which is exactly what we want,
and shows I can poll a pin with a specified interval duration

This is more than enough information to get me going if I was really doing something like this now. I don't expect it to be 100% accurate and up-to-date, but it's more than enough to point me in the right direction of where to focus studying more about this topic.

Using it to work through today's Advent of Code problem

So I thought, why not try to use ChatGPT as a learning assistant? I could use it instead of something like Stack Overflow, and instead of sifting through pages and pages of search results of varying quality and mismatched contexts, ChatGPT should be able to give me responses that are more direct to the point, within context, and (hopefully) much more up-to-date.

I've been using Advent of Code this year to learn how to code in Ruby. While I do have the advantage of generally knowing what I have to do to solve problems, I currently don't know exactly how those are achieved in Ruby. So instead of asking ChatGPT right out to solve a problem for me, I'll instead ask it how something can be done in Ruby as I go along.

For reference, here is today's Advent of Code problem (2022 day 5).

Splitting arrays

The input data for today comes in two parts: (1) an initial crate configuration, and (2) a sequential list of instructions for moving crates across the system.

My idea was to read off the entire input file per line into an array, and split that into the two constituents on the empty line.

That's implemented here.

Splitting a string in groups of N characters

Here is the sample crate arrangement input:

    [D]
[N] [C]
[Z] [M] [P]
 1   2   3

Because of the nature of the formatting, I can take each line of that input, collect the characters in groups of 4, and just drop everything but the 2nd character (which is the crate ID itself).

That's implemented here.

Parsing instructions using regex

Here is the sample instructions input:

move 1 from 2 to 1
move 3 from 1 to 3
move 2 from 2 to 1
move 1 from 1 to 2

I wanted to use regular expressions to capture the three values in each instruction: (1) how many crates to move, (2) from which column, (3) to which column.

Here I realized that I wanted to use named capture groups instead of just plain ones.

This is implemented here.

But then I wondered if it's possible to destructure the result of that into its constituent variables (instead of doing captures["count"], etc.).

This is implemented here.

Concatenating arrays

I realized I didn't know how to actually append elements of one array into another array short of doing something like:

array_1 = (1..5).to_a
array_2 = (6..10).to_a

array_2.map { |item| array_1 << item }

So I thought to ask anyway:

And sure enough, there was an easier way. This is implemented here.

How did that feel?

Honestly, really well. Most of the time when I'd be searching for how something is done in a specific language / framework / platform, it's a huge bulk of sifting through various search results, evaluating blog posts, or adapting a working solution for a slightly different problem than I was going through. When I was using ChatGPT, it felt like I had the rough equivalent of a mentor / paired programmer that was working on the same thing beside me. The responses were on point most of the time, and I am very appreciative of the time that saves for me.

When I was still a Solutions Architect for AWS, one of the things we've really wanted to build was a theoretical system we called an "SA-in-a-box". It was going to be an interface where you could ask it some of the most common questions SAs get asked by customers all the time, and it would spit out valid answers, ideally within context.

I tried that with ChatGPT today too:

I wouldn't just follow any of those suggestions blindly, but heck, I'd be damned if they weren't a good start to work off of.

I'm building my own hitbox-style fightstick controller

2022-10-06T00:00:00Z

I am absolutely hyped up for Street Fighter VI.

When I was still in high school, a huge amount of my free time went to visiting the video game arcades. The fighting games were my go-to then. I got hooked with Capcom's Marvel Super Heroes back in middle school, but it was definitely high school when I got totally hooked. Me and my friends would easily gobble up whatever the fighting game of the season was then, and just spend the afternoons after school (and our lunch money) plonking coins into arcade cabinets.

I'd eventually move on to spending even more exorbitant amounts of time in rhythm games (my favorite was Konami's DrumMania series), but fighting games never really left. To this day, I'd argue that Street Fighter III: 3rd Strike is one of the absolute best fighting games of all time.

That, and that the small amount of teaser content that's been released for the upcoming Street Fighter VI seems to suggest that Capcom's really steering this next one into great gaming waters, made me decide that, hey, I want to have a fightstick of my own to play with — why not build my own?

Yeah, totally logical

The very first thing I did was look for prior art.

I very quickly came across the GP2040-CE project, an open source firmware for fightsticks that ran on an RP2040 MCU. It seemed great — it has support for a good amount of consoles and machines, allowed for 8+2 action buttons out of the box (P1-P4, K1-K4, L3, R3), had what looked to be a convenient customization interface, and (most importantly) the option for RGB LEDs.

Yeah, this looked like a stellar starting point.

Looking for projects based on GP2040-CE, I think one of the better ones I found was Flatbox (specifically, the Flatbox rev 4).

To be honest, Flatbox looked to be almost perfect to me, but I really wanted to put in a few changes (improvements?) for myself:

1. Slightly bigger buttons

I have very chonky hands, and the Flatbox uses 24mm diameter buttons. As much as I really want a small form factor fightstick, I feel like this size might feel a bit too cramped for me. I'm thinking a more standard 28 / 30 / 32mm size would be better.

2. Slightly angled button positioning

I still want to keep the fightstick as small as possible so that it's portable, just like the Flatbox. But that also means that, unlike a traditional arcade panel, a player's arms will have to come into the fightstick at a more constricted angle, just like with a traditional keyboard.

And just like with keyboards, we want to really want to keep the wrists as straight and parallel with the arms as possible. To make the board a bit more ergonomic, I want to tilt the buttons towards the inside of the board, so that they receive the player's hands at a more natural wrist angle.

3. Optional RGB LEDs

The Flatbox PCB does not have routing for any LEDs. I'm not totally sure yet if I'd want LEDs on my fightstick, but I'd at least want to have the option to do so if I wanted.

Maybe at the very least, have support for underglow. Even better if per-key LEDs can be fixed onto the board.

4. Acrylic case

For some reason, I really want this in an acrylic case. In my head, a clear acrylic case will let me (or anybody else that uses this) customize the board much more easily than a 3D printed one.

Or that frosted / translucent acrylic can better scatter LED glow. I don't know.

A first draft

My first layout idea for a fightstick

I first came up with a squarish layout idea. I was thinking that the left hand index finger can quite easily position between the right hand index and middle fingers on buttons, so the right hand can bring itself more into the inside of the board.

After a bit of testing and consulting with other folks, I shelved that idea for now. I still want to maybe pursue it someday, but everyone I asked felt like it would feel unnecessarily cramped.

One thing I like about this though is that the K1 button is in a very good position in relation to the UP button at the bottom. Theoretically this would make it easier to use one's right thumb to press either of those. K1 is already very commonly hit with one's thumb (e.g. with P1+K1 and P1 > K1 > P2); having the UP button within small hitting distance gives you two options for a jump.

A second draft

For my second try at a design, I started with a more standard layout first, and adjusted accordingly.

My second layout idea looked much saner.

This layout is actually based off of the button layout generalized from Vewlix arcade cabinets. I've adjusted that a bit by:

Tilting the buttons into the center by about 12 degrees,
Removed the stick, of course,
Mirrored the P1, P2, and P3 buttons on the other side, also giving it a slant,
Placed the auxiliary buttons on the center of the board — we don't expect to use them much, but it'd nice if we do have to hit L3 / R3 to just right be there,
And finally, eyeballed the positions of the UP button in relation with the positions of the two hand clusters.

I haven't done any serious prototyping with this layout yet. I have printed it on paper 1:1 and the buttons seem to be in good positions for my hands, so I'm hopeful. The UP button is still accessible to both thumbs, and K1 is positioned well for the right thumb too.

The PCB is a bit bigger than I was initially hoping for at 236 x 147 mm, but it's still compact enough for travel.

Now if only we can ensure that the entire build won't be too thick as well. I'm going to try to keep it under 2 cm thick — ideally around 1.0 to 1.5 would be great.

The next steps would largely be designing the PCB.

If you're interested in following along the build, I'm keeping track of progress in this Github repository.

Build log: Vortex Pok3r hotswap mod

2022-09-17T00:00:00Z

My first mechanical keyboard is a Vortex Pok3r, which I've had since 2015. It's a 60% keyboard — so it doesn't have arrow, function, navigation and numpad keys. It was a very difficult transition at the time, having only worked with standard keyboards ever since, but I've grown very accustomed to it, and now work pretty darn fast with it. Point is, it's been something close to my heart, I suppose.

I've since tried many more keyboards, having also built a Lily58 Pro and a Corne v3, and one thing I've really come to realize is that the MX clear switches on my Pok3r were really heavy. I thought it'd be really nice if my Pok3r was hotswappable, so I can just switch out new, um, switches whenever I wanted to try something new.

Disassembly

Taking the PCB out of the case is a piece of cake: after removing all the keycaps, there will be six screws that secure the PCB down onto the metal case. Removing them lets you take it out easily.

The locations of the screws on the Pok3r.

The PCB assembly is made up of two major components: the (black) metal plate where the switches secure themselves onto, and the (red) PCB underneath where the switches are soldered onto. The two pieces are only secured to each other by the switches themselves being soldered onto the PCB — so to separate them, we'll have to desolder all of the switches one by one. That would be 61 switches to desolder.

Desoldering

Switches are soldered on the underside of the PCB.

The underside of the Pok3r PCB.

Each switch will have two solder points that will need to be desoldered. I didn't have an RGB Pok3r (so my switches didn't have LEDs), but an RGB Pok3r would also have solder points for the LEDs that will need to be removed. Look for each switch's bottom housing foot (the stem that is under each switch), and the switch contacts will be easy to identify.

Note that even on a non-RGB Pok3r like mine, the Caps Lock switch will have an LED mounted anyway, so that will also have to be removed.

Desolder each switch contact (and LED contact on an RGB Pok3r).

I used a desoldering pump to get most of the solder out of the PCB, and desoldering wick to clean up what the pump couldn't suck out. The wick is really great for keeping the contact points as clean as possible for later as well.

Once a switch has been cleanly desoldered, you should be able to easily pull it out using a switch puller.

The bottom row of switches desoldered.

Once all the switches have been removed, the plate and PCB will separate. This would be a good time to clean the components. Mine had a lot of dirt and dead skin that had accumulated over the last seven years!

Hotswap sockets

These days, custom keyboards can be fitted with hotswap sockets that are simple to install on a PCB. The Pok3r PCB is not particularly made to accommodate these kinds of hotswap sockets however, so we can't use them unfortunately.

There's a few options that we can use — I personally used holtites so that I didn't have to solder them onto the board anymore. Another common alternative is mill-max sockets, which would need soldering.

A single holtite socket.

Holtites are really small and very easy to lose if mishandled. I used pointy-tip tweezers to put them into the right PCB sockets by "poking" a tip into a holtite, then using that to position into the hole. Occasionally, I'd use a pair of long nose pliers to hold the holtite in place inside the socket as I removed it from the tweezers.

Holtites placed in PCB sockets.

To secure holtites into place, you'll need to heat them up with a soldering iron as you push them down into the hole. The heat makes them more malleable and will let them find a better fit in the socket. I used a blunt tip for my soldering iron to do this so that I can more evenly provide force into the push.

You don't need to solder the holtites into place at all.

Once secure in the holes, the holtites will protrude from the PCB by a small amount on the underside. On the top side, however, they should lie flush with the surface as much as possible. They should also feel fixed in place, and should not give when you try to push them out of the sockets.

Holtites protrude a bit on the underside of the PCB when installed.

PE foam mod

One modification I did to my Corne v3 that I ended up really liking was to put some PE foam on the PCB for the switches to rest on. They're meant to provide a more snug fit for the switches once they're put in place, and helps minimize clatter when you type on them. In my case, they also seem to help give me a more thocky sound even with tactile switches, which I really like.

To make this easier for myself, I use pre-cut PE foam "stickers" that I can just stick onto the PCB directly, right where the switches would be. I've got enough left over from my Corne v3 build, so this works out nicely.

These PE foam stickers are very convenient!

This might just be the most enjoyable part of this build.

Re-assembly

Now that the holtites are in place (and we've got PE foam installed as well), we can put the keyboard back together.

I started by placing switches in the corners of the plate, along with a couple in the very center. I then aligned this with the PCB and just placed the switches snug into place. With the holtites installed, you'll need to give each switch a firm push so that the contact pins fit into the sockets. The holtites should hold the switches in place firmly.

I'm using C3 Kiwis here, except for the bottom right ones, where I'm using U4S Bobas.

Holtites will grip the switch contact pins.

And, really, you just keep on going from there by putting your switch of choice back into the plate and PCB. Once you have all the switches in place, the plate and PCB will hold together on their own.

Just take care not to bend the switch contact pins are you push them into the sockets. If you do, take the switch out again and use a pair of pliers to straighten the pin out and try again.

Before you screw the PCB back into the case, you should test the keyboard first and make sure that all the keys are registering correctly.

Et voila

And there you have it. Just screw the PCB back into the case (again, there are six screw points!), and place your choice of keycaps, and the Pok3r is good as new. It's now been given a breath of fresh air, and is now much easier to swap switches around to try new stuff out in the future!

Phew!

Validation checks in AWS CDK constructs

2022-08-03T00:00:00Z

Working with quite a lot of AWS CDK for our prototypes, it's very important to be able to ensure our Stacks and Constructs are in a valid state at all times. When your infrastructure is layers upon layers of nested components, it's nice to have that guarantee at the back of your mind that your system is within expectations, and that none of our components are entering invalid configurations.

Yes we can write up unit and integration tests using something like Jest, but full-fledged tests can be a bit heavy-handed at times. Sometimes we just want to make sure our system is in an acceptable state right before deployment or synthesis. That's where validation checks can help best.

CDK constructs and `.validate()`

I surprisingly found very little on this topic from official documentation. It appears that all CDK constructs, when mapped in the construct tree, has a .validate() function, which the CDK uses to determine whether or not its in a valid state. This returns string[] which is a list of validation error messages --- if it's an empty list, then the construct is valid.

The .validate() function seems to be called by the CDK during stack synthesis (so, as part of a cdk synth operation). This also has the neat side effect that validation also occurs right before deployment actually happens on cdk deploy.

This lets us create some nice guardrails around our systems that prevents deployment if something is not as we'd expect, for example.

Adding validation checks

Of course, that's only useful if we can add validation checks in the first place. Turns out all CDK constructs have access to an .addValidation() hook through their .node accessor. .addValidation accepts an argument with the shape { validate: () => string[] }. The object's .validate() function is just a validation function that, again, returns a list of validation error messages. In a gist, construct.node.validate() calls the .validate() function of all validators added via .addValidation(), and collects the validation error messages. If there is at least one such error message, then the construct is in an invalid state (and synthesis ends with an error).

An example instead?

Let's create a construct that will trigger an event on a specified hour every day, just like a cron job. We'll use Amazon EventBridge to create a rule the fires on the specified hour --- we can then use this to, for example, invoke a Lambda function, or start a downstream process (like retraining an ML model).

We'll start with a simple scaffold:

import * as cdk from 'aws-cdk-lib'
import * as events from 'aws-cdk-lib/aws-events'
import { Construct } from 'constructs'

// :: ---

class ScheduleConstruct extends Construct {
  constructor(scope: Construct, id: string) {
    super(scope, id)

    // :: This creates our daily rule.
    //    For now, this will trigger at 8:00 AM daily.
    const rule = new events.Rule(this, 'my-daily-rule', {
      schedule: events.Schedule.cron({
        hour: '8',
        minute: '0',
      }),
    })
  }
}

We could let this construct be configurable through props passed into it, so let's try that:

// ...

type ScheduleConstructProps {
  hour: string
}

class ScheduleConstruct extends Construct {
  constructor(scope: Construct, id: string) {
  constructor(scope: Construct, id: string, props: ScheduleConstructProps) {
    super(scope, id)

    // :: This creates our daily rule.
    //    For now, this will trigger at 8:00 AM daily.
    const rule = new events.Rule(this, 'my-daily-rule', {
      schedule: events.Schedule.cron({
        hour: '8',
        hour: props.hour,
        minute: '0'
      }),
    })
  }
}

Now, putting aside the possibility that events.Schedule.cron() does validation of its own (I haven't checked, to be honest), we'd want to ensure whatever we're passing into our construct is within acceptable bounds. Specifically, we want to make sure that props.hour is always gonna be an integer >= 0 and < 24, but in a string type. Otherwise, we'll have a construct trying to set a scheduled trigger at an unheard of time like 37:00 pm or something.

Let's add in validations

We can guard our construct from those cases by adding in validation checks:

// ...

class ScheduleConstruct extends Construct {
  constructor(scope: Construct, id: string, props: ScheduleConstructProps) {
    super(scope, id)

    // ...

    // :: Add our validation checks
    this.node.addValidation({
      validate: () => {
        const messages: string[] = []
        const hourInteger = Number.parseInt(props.hour)

        if (Number.isNaN(hourInteger)) messages.push('Hour provided is not a valid integer.')
        if (hourInteger < 0) messages.push('Hour must be non-negative.')
        if (hourInteger >= 24) messages.push('Hour must be less than 24.')

        return messages
      },
    })
  }
}

Now that our validation checks are in place, any time you try to run cdk synth or cdk deploy with an invalid provided hour value, you'll get an error message, and the CDK will refuse to progress.

For example, if I try:

new ScheduleConstruct(this, 'my-schedule', {
  hour: '30',
})

λ  cdk synth
# ...
Error: Validation failed with the following errors:
  [CdkTestStack/my-schedule] Hour must be less than 24.

And my mind is more at ease as a result.

Zip files from Amazon S3 without saving them locally

2022-06-01T00:00:00Z

I recently had an engineering requirement where I had to build a Lambda function that would grab an arbitrary number of files stored in S3, compress them into a single ZIP archive, then upload the archive into another S3 bucket. This was eventually going to be given to a user using a presigned URL to access.

It would have been simple enough to implement if we could just download all the files into the Lambda function, zip them up, then upload. However, the number of files (and total size of all of it) was unknown and completely arbitrary. Lambda functions only come with 512 MB of ephemeral storage. You can extend this up to 10 GB at a time, but the storage costs can add up. Besides, there's still no guarantee we can keep under 10 GB.

Let's not download the files then

It turns out that we can use a stream to upload content to Amazon S3 when using the AWS SDK. This is true at least with the v2 of the AWS SDK.

This can be very useful. Using something like archiver, we can directly stream files into an archive. If we can then pipe that stream directly into the S3 upload call, then we're technically uploading a ZIP file, without having to compress the contents locally.

First, let's create the archival stream. This is simple enough to do:

import archiver from 'archiver'

// :: ---

const archiveStream = archiver('zip')

archiveStream.on('error', (error) => {
  console.error('Archival encountered an error:', error)
  throw new Error(error)
})

We'll then set up a passthrough stream into an S3 upload. The idea here is that whatever data is put into the stream will be passed through directly to the output --- in this case, an object in an S3 bucket.

Before anything else though, we have to make sure that our S3 client keeps connections open long enough for us to finish uploading. Remember that we're opening up what's essentially a pipe into an S3 object file, and will be dumping data into it continuously until we're ready to finalize / close the stream.

import * as AWS from 'aws-sdk'

// :: ---

const s3 = new AWS.S3({
  httpOptions: {
    timeout: 60 * 10 * 1000, // :: 10 minutes
  },
})

import stream from 'node:stream'

// :: ---

declare const TARGET_BUCKET_NAME: string
declare const ARCHIVE_KEY: string

// :: ---

const passthrough = new stream.PassThrough()

// :: We wrap this in a promise so we have something to await.
const uploadTask = new Promise((resolve) => {
  s3.upload(
    {
      Bucket: TARGET_BUCKET_NAME,
      Key: ARCHIVE_KEY,

      Body: passthrough,
      ContentType: 'application/zip',
    },

    // :: This callback fires when the stream is closed.
    //    As you can see, we're just resolving the promise here,
    //    so whatever awaits this is notified.
    () => {
      console.log('Zip uploaded.')
      resolve([TARGET_BUCKET_NAME, ARCHIVE_KEY])
    }
  )
})

Then finally, we just pipe the archive stream into the upload stream. Anything we put into the archive stream, eventually finds itself passed through the upload stream.

archiveStream.pipe(passthrough)

With that out of the way, all we need to do now is to actually put the files into the stream. Let's talk about that now.

Throw the files in

If we already know which files stored in S3 we want to include in the archive, then we can just use the AWS SDK to grab the file contents, then put them in the stream. Thankfully, the getObject operation using the AWS SDK gives us a Buffer of the object. We can use this Buffer directly with our archive stream.

import path from 'node:path'

// :: ---

declare const SOURCE_BUCKET_NAME: string
declare const OBJECT_KEYS: string[]

// :: ---

for (const key of OBJECT_KEYS) {
  const params = { Bucket: SOURCE_BUCKET_NAME, Key: key }
  const response = await s3.getObject(params).promise()

  // :: `response.Body` is a Buffer
  archiveStream.append(response.Body, path.basename(key))
}

// :: When all the files have been added, then we can
//    finalize the archive stream. This eventually closes
//    the stream, and subsequently closes the passthrough
//    stream we created in the upload task.
archiveStream.finalize()

To finish things off, we can await our uploadTask earlier somewhere in the application, and get the resulting bucket name + object key when it resolves.

// :: Remember that this task resolves only when the
//    passthrough stream closes, and the passthrough
//    stream closes only when the archive stream
//    (that is piping into it) closes too.
//
//    When we finalize the archive stream, pretty much
//    everything else collapses, and this resolves.
const [BUCKET_NAME, OBJECT_KEY] = await uploadTask

// :: Now we can do whatever we want with this.
//    How about we generate a presigned URL?
const params = {
  Bucket: BUCKET_NAME,
  Key: OBJECT_KEY,
  Expires: 60 * 60 * 24, // :: 24 hours
}
const url = await s3.getSignedUrlPromise('getObject', params)
console.log(url)

And there you have it. We just archived files stored in an S3 bucket, and stored that into another S3 bucket without having to save the files locally first.

Tracing through your distributed applications using AWS X-Ray

2020-07-11T00:00:00Z

On my last livestream session with the AWS DevAx Connect team, I talked about how you can build observability into distributed applications on AWS Lambda (among other things), particularly using AWS X-Ray.

With all the positive buzz around the whole idea of microservices and immensely distributed applications, one of the very real problems it creates is that --- now that most everything in your application is a Lego piece in a great, big whole --- how do you keep an eye on all of those moving pieces? When one of your users complains that your service is responding very slowly, which of the hundreds (or even thousands) of pieces a single API call even lightly interacts with could be the culprit? When an exception or error breaks out, where exactly did it come from?

A sample figure caption.

This is a question that's been asked of me time and time again, and something that I'm always a bit surprised how it's not given more thought by more people --- at least not until much, much later in the engineering process. Oftentimes, by then there would already be a fire being put out, and everyone would then be wishing that they had focused on a solution much, much sooner.

Setting the stage

In the livestream, I shared that I've prepared a demonstration serverless project using the Serverless Framework. Once deployed, it exposes a single API endpoint that generates a random string you can then use to name one of your new projects. Snazzy!

When an HTTP request is received through Amazon API Gateway, an entrypoint AWS Lambda function handles it. This handler then selects, at random, one Lambda handler from each of five different services: nouns, adjectives, verbs, colors, and descriptors. Each service, by default, has ten Lambda functions up for grabs.

Now, it has to be said that this definitely isn't how something like this should be built, and the setup is, as the README file says, somewhat convoluted. But while it's not quite how a real-life application would be built (at least I hope not!), it does mimic how a bunch of small services can talk to one another to address an end-user request.

Enabling tracing through AWS X-Ray

Let's start putting in the tracing functionality.

Enabling tracing on your Lambda functions is pretty easy — you just have to tick a checkbox on the Lambda function's dashboard.

However, when you're managing multiple Lambda functions in your application (as you most probably are going to be), toggling it this way, going through your functions one by one, is not gonna be fun.

If you're using the Serverless Framework, it fortunately has an easy way to enable tracing across your entire workload. Just add the following to your serverless.yml manifest, under the provider section:

provider:
  name: aws
  runtime: nodejs12.x
  stage: dev
  region: ap-southeast-1

  tracing:
    lambda: true
    apiGateway: true

  iamRoleStatements:
    - Effect: Allow
      Action:
        - lambda:InvokeFunction
        - xray:PutTraceSegments
        - xray:PutTelemetryRecords

We're adding in a couple of IAM permissions at the bottom just to be sure --- when you're toggling tracing on the Lambda console, these will be automatically added to your Lambda function's execution role if they're not there yet. But, of course, if you know you need it, it's better to be explicit about it, so let's just add those in.

These changes make it so that whenever requests go into the API Gateway, and invoke any of the Lambda functions on this application, X-Ray will know about it, and will be able to map your traces accordingly.

What this doesn't do is trace whatever else your Lambda functions may be calling --- if your functions talk to other AWS services (e.g. it writes into an Amazon DynamoDB table, or it fetches a file from an Amazon S3 bucket), we won't be able to trace those yet.

So let's add that functionality in.

Going the full mile with tracing

Extending the traces to whatever else we're calling from our Lambda functions is really simple to do as well --- and it works with anything that you're using the AWS SDK in, not just Lambda functions.

We'll need to add in the AWS X-Ray SDK

npm install --save aws-xray-sdk

I'm using the AWS X-Ray SDK for Node.js here, but there's a handful of other options you can use depending on your preferred language. Check out the AWS Developer Guide for more info.

Afterwards, all you have to do is to wrap the AWS SDK with the AWS X-Ray SDK everywhere in your code that you're using it. So, for example, in src/handlers/entrypoint.js in the demo project, you can do it like this:

// :: bring in the X-Ray SDK
const AWSXRay = require('aws-xray-sdk-core')

// :: ... then you can either wrap your AWS SDK like this:
const AWS = AWSXRay.captureAWS(require('aws-sdk'))

// :: ... or maybe like this:
const { Lambda } = AWSXRay.captureAWS(require('aws-sdk'))

Now, most any other AWS service we call from our Lambda function will automatically have tracing information embedded in them, so when we view your service map later on AWS X-Ray, you'll see exactly where and when your requests are jumping to.

At this point, you're already done.

Your application will start tracing all the requests going through it. If you want to check out how it looks in action, however, read on. We'll set up an installation that will send a lot of requests into it so you can see how the tracing is done.

You're curious to see how it looks

Of course you are.

If everything was set up correctly, all requests going through the application (through the API Gateway) will now have tracing information that's followed by X-Ray across all your AWS resources.

Setting up a load testing platform

We'll generate a huge load of requests, just like how you'd most probably do it if you were load testing an application. (Which you really should be doing, by the way.)

I like to use Nordstrom's serverless-artillery for this --- which coincidentally also uses the Serverless Framework, so setting it up is going to be pretty familiar.

npm install -g serverless-artillery

mkdir loadtesting     # :: or wherever you want to store your load testing manifest
cd loadtesting

#  :: this will generate a load testing starter manifest
slsart script

This will generate a script.yml file containing the information that serverless-artillery will need to run your load tests. Try switching out the contents of that file with the following (feel free to tweak, if you'd like):

config:
  # :: Change this to the URL of your API Gateway entrypoint
  target: 'https://xxx.execute-api.yyy.amazonaws.com/dev/generate'

  # :: These are your testing phases.
  #    This configuration sets up the load test so that:
  #      1/ For the first 15 seconds,
  #         start with 10 requests / sec,
  #         and steadily increase that to 25 requests / sec,
  #      2/ Then for the next 60 seconds,
  #         maintain a constant 25 requests / sec rate
  phases:
    - duration: 15
      arrivalRate: 10
      rampTo: 25
    - duration: 60
      arrivalRate: 25

# :: This section allows you to define more complicated scenarios,
#    but we won't use this for now.
#    What this setting currently does is just send the HTTP GET requests to the root
#    of the target URL as specified above.
scenarios:
  - flow:
      - get:
          url: '/'

Then we need to deploy that to your AWS account as well:

slsart deploy --stage my-test-setup

It'll take a few minutes, but when the deploy is done, we can then instruct it to run the load testing scenario above:

slsart invoke --stage my-test-setup

Inspecting the traces

It'll take a few seconds for the tracing information from the requests to be available at AWS X-Ray, but go ahead and go to your X-Ray Console anyway.

Navigate to your Service Map, and you should get a pretty cool graph of nodes in your application, as well as the connections between them:

Ooh, circles and lines.

Awesome, but aside from a few nice latency and transaction rate numbers we can pat ourselves on the back with, there isn't much else this is telling us. So let's try introducing some errors into the stack, and see how that will affect our map.

Add in some artificial defects

I'll change some of the Lambda handlers so that:

one of them takes a much longer time to respond, and
one of them generates an HTTP 4xx error half of the time

We have a total of 50 handler functions here, and just having to fish out two faulty ones without tracing tooling is a pretty tedious effort --- imagine if we had hundreds of these services up and running.

I'll add the following code blocks to the handlers:

// :: src/handlers/adjective.js

// :: mock a handler that takes an abnormally long time to resolve
export const faultyHandler = async () => {
  return new Promise((resolve) => {
    const response = handler()
    setTimeout(() => resolve(response), 2000) // :: resolve after 2 sec
  })
}

// :: src/handlers/verb.js

// :: throw an error half of the time
export const faultyHandler = async () => {
  if (Math.random() > 0.5) return handler()
  // :: ---
  throw new Error('DEFECT')
}

Then we'll adjust src/handlers/functions.js to take the faulty handlers into account:

// ...

const createHandler = (type, i) => {
  const isFaulty =
    (type === 'adjective' && i === 4) ||
    (type === 'verb' && i === 6)

  const handler = isFaulty ?
    `src/handlers/${type}.faultyHandler` ||
    `src/handlers/${type}.handler`

  return {
    [name]: { handler }
  }
}

// ...

Finally, we'll deploy the faulty handlers:

sls deploy -f adjective-4
sls deploy -f verb-9

Then run the load tests again.

This time, when you look at your service map, you'll easily see where in your application you're experiencing degraded performance:

Let's see if you can find exactly where our application is hitting errors.

You can select any node in the map to look more into it. When you select a node, you can isolate the map to just that part of the application, and see the resulting distribution of performance results.

You can also click through the Analyze Traces button to view the traces in detail. Let's do that to investigate our response time behavior.

Let's say we wanted to investigate all the requests in our application that are taking longer than 1.5 seconds to complete --- we can do that by searching for responsetime > 1.5 on the search input bar. (You can click on the help button to see how the search syntax works.)

This will give you a list of traces in your application that logged a response time of, expectedly, longer than 1.5 seconds. Click on one of the traces, and you'll see a dashboard with a trace map of all the services involved in that trace, as well as a timeline of events making up that entire trace:

It's clear to see which particular service was at fault for the slow overall response time with this request.

There's definitely more to this

Having active tracing available for your applications make it so much simpler to perform investigations for debugging, as well as for just keeping an eye on all your services, and making sure they're behaving as expected.

This is definitely just scratching the surface of it, and if you want to know more about what else you can do with AWS X-Ray, then don't miss checking out the AWS Developer Guide for it.

Working with the Amazon Chime SDK

2020-06-19T00:00:00Z

I was one of those people that were pleasantly surprised, waking up one morning earlier this month, to the news of Slack announcing a partnership with AWS. While the decision to adopt or strengthen their AWS posture isn't exactly something new for a lot of companies, the real standout for me was that Slack will also migrate their voice and video call functionality to use the Amazon Chime service moving forward.

Now, I use Slack very significantly in my day-to-day, and Slack already has voice and video calls available, so actively migrating that functionality over to a different platform feels like a really major move.

So, in the spirit of discovery, wonder, and that itch to build, I spent a bit of time working with the Amazon Chime service to get a better feel of what it has to offer.

The skinny

The Amazon Chime SDK allows you to create managed online meetings where attendees can do simultaneous audio and video streaming, as well as share content across all attendees (e.g. a screen share). It does a great job of abstracting a lot of the details away, so you (as the developer) won't have to deal too much about the intricacies of working with websockets, WebRTC, and media handling.

Even better, you don't need to create servers and storage to support your users --- the technology resources required to run your meetings are automatically managed for you. This means that you don't have to think about how much infrastructure to prepare to potentially support so and so amount of users; all of it just automatically adjust and scale for you.

At the time of this writing, the pricing for the Amazon Chime SDK is also very cost-effective: it amounts to just USD 0.0017 / minute connected for every attendee. Easy.

To successfully create and run an online meeting on Chime essentially requires two steps:

Create the Chime meeting
Connect attendees to the Chime meeting

Creating an Amazon Chime meeting

Creating the actual Chime meeting is done using the AWS SDK. This is the general SDK that's used for most of the stuff you can do on the AWS platform. (Yes, the Amazon Chime SDK is different --- we'll get to that in a bit.)

import { Chime } from 'aws-sdk'

const chime = new Chime({ region: 'us-east-1' }) // :: has to be us-east-1 for now

/** @returns {ChimeMeetingInstance} */
async function createChimeMeeting() {
  const params = {
    ClientRequestToken: 'some-random-string-of-your-choice',
    MediaRegion: 'ap-southeast-1', // :: where your call is actually hosted
  }

  return await chime.createMeeting(params).promise()
}

The instance of the AWS.Chime object currently has to be in us-east-1.

This is not where your Chime meeting will be hosted; you can specify that using the MediaRegion field on the call to Chime.createMeeting(...).

Attendees that you want to bring into the meeting will also require a sort of join ticket. Later, back at the actual client application, users that want to join a Chime meeting will need to present attendee information that was explicitly generated for them.

/**
 * @param {ChimeMeetingInstance} meetingInstance
 * @returns {ChimeMeetingAttendee}
 */
async function createChimeAttendee(meetingInstance) {
  const { MeetingId } = meetingInstance.meeting
  const params = {
    MeetingId,
    ExternalUserId: 'unique-user-identifier',
  }

  return await chime.createAttendee(params).promise()
}

After that, you just have to give the client application both the meeting and attendee information, and they can use that to actually connect themselves into the meeting.

For example, if you were using AWS Lambda behind an API Gateway, you might be doing it like this:

async function handler() {
  const meeting = await createChimeMeeting()
  const attendee = await createChimeAttendee(meeting)

  const payload = { meeting, attendee }

  return {
    statusCode: 200,
    body: JSON.stringify(payload),
  }
}

Connecting to an Amazon Chime meeting

Back on your client application, you will need both the meeting and attendee information we generated above to complete the connection to Amazon Chime.

Also as mentioned above, the Amazon Chime SDK is different from the AWS SDK --- this is probably because client applications (like a web app, or mobile app) don't need all the functionality behind the main SDK, and are just interested in connecting to the meeting.

There are also Amazon Chime SDKs for iOS and Android, if you need them.

npm install --save amazon-chime-sdk-js

Connecting to a Chime meeting is very straightforward. Once you have both a meeting instance (which can be shared across all attendees), and an attendee ticket (which is unique to every attendee), you can then use the Chime SDK to complete the connection.

import {
  ConsoleLogger,
  DefaultDeviceController,
  DefaultMeetingSession,
  LogLevel,
  MeetingSessionConfiguration,
} from 'amazon-chime-sdk-js'

const logger = new ConsoleLogger('Chime Logs', LogLevel.INFO)
const deviceController = new DefaultDeviceController(logger)

// ...

/**
 * @param {ChimeMeetingInstance} meeting
 * @param {ChimeMeetingAttendee} attendee
 * @returns {ChimeMeetingSession}
 */
function connectToChimeMeeting(meeting, attendee) {
  const meetingConfig = new MeetingSessionConfiguration(meeting, attendee)
  const meetingSession = new DefaultMeetingSession(
    meetingConfig,
    logger,
    deviceController
  )

  // TODO --- configure other stuff for the meeting

  console.log('Starting the Chime meeting!')
  meetingSession.audioVideo.start()

  return meetingSession
}

The Chime SDK will take care of abstracting out the details of juggling your websockets, attendees syncing, content and media shares, etc. Pretty convenient!

Connecting audio

So, doing everything above gets you in the Chime meeting, which gives you ready access to everything that's being thrown around in there --- but unless you bind those things to actual elements in your client application, you won't see a lot of use here.

One thing you will want to do is configure audio --- both sending out the audio coming from your machine, and playing the audio that you are receiving (coming from everyone else on the call). The Chime SDK does a great job of simplifying this, but you still need to push it a bit more.

To specify what audio device to use as an input (most of the time, this will be your microphone), you can do the following. The Chime SDK will automatically bind the audio coming in through the selected device, and broadcast that across all attendees in the meeting.

/** @param {ChimeMeetingSession} session */
async function setMeetingAudioInputDevice(session) {
  // :: This will select the default audio input device on your machine, generally.
  //
  // :: You will probably want to let the user select
  //    which device they specifically want to use.
  const availableAudioInputDevices = await session.audioVideo.listAudioInputDevices()
  const deviceId = availableAudioInputDevices[0].deviceId

  await session.audioVideo.chooseAudioInputDevice(deviceId)
}

To actually play the audio that you receive in the meeting (e.g. someone speaking), you need to bind an <audio> element on your page using the Chime SDK.

<!-- we just need this somewhere on the page -->
<audio id="my-audio-element"></audio>

/** @param {ChimeMeetingSession} session */
function bindMeetingAudioOutput(session) {
  // :: or however you prefer to get a reference
  //    to the <audio> element above
  const audioElement = document.getElementById('my-audio-element')
  session.audioVideo.bindAudioElement(audioElement)
}

Connecting video

Connecting and displaying video and content is done pretty similarly to audio above.

To share video out into the Chime meeting (like, for example, your webcam output), you need a MediaStream object you can pass through the Chime SDK.

If you're already using a third-party library to capture and display what your camera sees onto the page (e.g. react-webcam for React), there's a good chance that it also exposes a MediaStream object you can readily use.

To broadcast out a video stream:

/**
 * @param {ChimeMeetingSession} session
 * @param {MediaStream} videoStream
 */
async function broadcastVideo(session, videoStream) {
  await session.audioVideo.startContentShare(videoStream)
}

If you just want to share what's on your screen though, the Chime SDK has a convenience method that does this for you automatically:

/** @param {ChimeMeetingSession} session */
async function shareScreen(session) {
  await session.audioVideo.startContentShareFromScreenCapture()
}

To display incoming video you're receiving through the Chime meeting, you will need to bind to a <video> element just like with <audio> above.

However, unlike with audio, you will need to determine when video content is actually being shared. My guess here is that this is because, unlike with audio where it's trivial to combine multiple audio signals together into one, with multiple video streams, you will need to handle them one by one. So you will actually need to listen in on how what video streams are being shared (e.g. multiple webcams, screen shares, etc), and bind them to individual <video> components as you see fit.

To do this, you will need an observer object that has a .videoTileDidUpdate(...) method implementation, and register that into the meeting session. This observer will be notified whenever video content is started or stopped in the meeting, so you can adjust accordingly.

<!-- Again, you will need video elements on the page.
     Unlike with audio elements though, you may want to have to create these dynamically,
     since you may need more than one element at a time.
-->
<video id="my-video-element"></video>

/** @param {ChimeMeetingSession} session */
function displaySharedVideoContent(session) {
  const observer = {
    // :: a tile represents a single instance of shared video content
    videoTileDidUpdate: (tile) => {
      console.log('Received content with ID:', tile.tileId)

      // :: TODO: get a video element specifically for this tile
      const videoElement = document.getElementById('my-video-element')
      session.audioVideo.bindVideoElement(tile.tileId, videoElement)
    },
  }

  session.audioVideo.addObserver(observer)
}

That should be it!

You'll probably also want to know how to pause or stop content shares, mute your audio, see who the attendees are in your meeting --- all of those are in the documentation for the Amazon Chime SDK.

As a sort of cherry on top, here's me making a fool of myself testing the Chime meeting video latency. The Chime meeting here is hosted in the AWS Singapore region (ap-southeast-1), with me being in the Philippines.

Benchmarking the AWS Graviton2 processors

2020-06-16T00:00:00Z

There's been a lot of fanfare about the new AWS Graviton2 processors, and the performance and cost benefits of using them compared to their alternatives in the cloud.

These ARM-based processors power the new M6g, C6g, and R6g family of EC2 instance types, with the previous (first) generation powering the A1 family of EC2 instances. AWS promises that the Graviton2 offers significant improvements over the current 5th gen x86 EC2 mainstays. (20% lower cost and 40% higher performance, based on AWS' own internal testing, according to the AWS Graviton2 landing page.)

So, I wanted to have a bit of fun and do some light benchmarking myself.

Methodology

I wanted to gather as many data points as I could without spending too much effort on it, so I created a bash script that's meant to be run as an EC2 instance's user data. I've always thought that the naming was pretty off --- EC2 user data scripts execute on launch.

You can take a look at the script here. It will essentially do the following:

Gather some information from the instance metadata,
Install sysbench and the AWS CLI,
Set the instance to terminate when it's shut down (instead of just stopping),
In sets of powers of 2, for the available cores in the machine:
- Run benchmarking tests,
- Format and append the results into a CSV file
Upload the results to an Amazon S3 bucket,
Shutdown the instance (effectively terminating it)

To automate the entire process, I created an EC2 Launch Template that uses this user data script, and created an EC2 Autoscaling Group from it. The EC2 instances will run the benchmark tests and terminate themselves, and the autoscaling groups will happily create new instances to replace them. Left those running for a day or so, just collecting results on S3.

Once the data points are ready, accessing them is simply just a matter of using AWS Glue and Amazon Athena to run some SQL queries directly on the CSV files to crunch the numbers.

Results

After a weekend of running the benchmarks, I managed to get the sample pool below.

Count of data points for each instance type.

I wanted to get benchmarking results for when all the cores in the instance are used together, as well as when constraining the testing to just a single core. All the instance types I used have 2 cores, with varying levels of memory. If you use the user data script above for your own testing, however, it should also test for other core combinations (e.g. 1, 2, 4, 8, etc).

These are the summarized results from the single-thread benchmark tests:

Results when benchmarking was constrained to a single thread.

And these are the summary from the multi-thread benchmark tests (all of which effectively have 2 threads):

Results when benchmarking using both cores on each instance type.

I had a few minutes to spare, so I went ahead and looked at how much improvement each Graviton2 instance gave over each other non-Graviton2 option:

Percent increase of average total events of Graviton2 instances.

For really quick, shallow numbers, I think these are definitely quite telling. Granted, the real litmus test of any new tech is on their application to real scenarios (and not just tests like these that lack context), but I'd say that the numbers as they are are indicative that the Graviton2 processors definitely offer pronounced improvements over the other options.

It'd be nice to find some time to be able to run an applied benchmarking test, like how KeyDB did theirs.

How to: Create a Remote Gaming Server using Parsec on AWS

2020-04-26T00:00:00Z

As is with many people, I'm stuck at home because of the COVID-19 lockdowns most everywhere. Unfortunately, even with the increased time at home that I can use to game, I'm also stuck with a old-ish gaming rig that's wedged with a video card that is showing it's age. As much as I'd like to order new equipment, I can't do that with the restrictions on deliveries.

I've heard of Parsec some years back, but never really paid it a lot of attention. Being stuck as I was, I decided to give it a shot (also due to the coaxing of some friends), and, boy, was I surprised at the results. The whole setup of having a remote gaming server running Parsec while another (local) machine played the games on it works just as well as described.

So here we are. I thought I'd share how I did my setup on AWS, as well as throw in some recommendations on how to maximize your fun, while minimizing your costs as you do it.

If you wanna skip all the auxiliary details and go straight to the build, just click any of these links to jump straight to it.

Parsec?

Parsec is a bit of software that works very similarly to how Remote Desktop works for something like a Windows machine. The idea is that:

You have a (beefy) machine where your games are installed (called the host),
You have another (less powerful) machine where you want to play your games (called the client),
Parsec handles the connection between the two so that you can run your games on the stronger machine, while you're controlling your game on the weaker one. Parsec uses networking magic to reduce the lag by a significant amount, while maintaining great graphics quality.

How Parsec works

Costs

I opted for a Windows-based g4dn.xlarge EC2 instance in the AWS Hong Kong region. As a spot instance, it's currently costing me USD 0.427 per hour it's turned on, which is about a 55% discount on the on-demand price. Since this is only turned on when I'm actually playing, that's not a bad deal at all.

Your Parsec server will also need storage --- I opted to stick with the General Purpose SSDs. This will cost you even when the server is turned off, so keep this in mind. You will need enough storage to install the games you want to play, but you can always install / uninstall games over time to minimize this. In the AWS HK region, I'm getting this at about USD 0.132 per GB per month.

There are also networking costs involved (since Parsec is streaming your game video out), but I haven't metered and done the math there yet.

So assuming I'm playing an hour every day for a month, with a 100 GB machine, my playtime is coming up to around USD 0.867 for every hour of play.

Not bad.

But will it run Crysis?

Of course, as is with any gaming rig, it has to run Crysis, right? No way was I not gonna try that out.

This Parsec host was running from the AWS Hong Kong region, and was being streamed to a local laptop here in the Philippines, with a reported ping of around 60-80 ms. Assuming we're hitting 60 FPS, that comes to around being 4-5 frames behind (probably double that, since our control input has to go back round to the host server too). Even at that level of lag, though, the game was pretty much playable, and the graphics, while not the best, was extremely better than I expected.

Let's get started

While Parsec will need to be installed on both the host and client machines, the host will need to be a Windows machine to function correctly.

You can also review the hardware requirements for the client machine here.

Create a Windows-based EC2 instance

Unlike what the Parsec documentation says, we'll use the new g4dn.xlarge instance type on Amazon EC2. This is the newest generation of the G-type of EC2 instances, and comes with 4 vcpus, 16 GB of RAM, 16 GB of VRAM, and a Nvidia T4 Tensor Core video card. It's also signficantly cheaper than the older generations, even though it looks to be much more powerful. We'll also use Spot Instances to lower our costs a huge deal.

Spot Instances?

AWS lets you create spot instances on Amazon EC2 instead of the normal on-demand type. I like to think of this as like an auction, and is a very effective way to get EC2 instances on the cheap.

When launching spot instances, you specify how much you're willing to pay for your EC2 instances (called the bid price). AWS pretty much always has extra machines in their data centers, so this excess is auctioned out at discounted prices as spot instances. The price of these extra machines changes in real time, based on availability and demand (called the spot price).

When launching spot instances, you get your EC2 instances if:

there is excess capacity, and
your bid price is higher than the current spot price.

No matter what happens, you will only ever pay the current spot price for your instance (not your bid price).

The catch here is, if ever the spot price goes above your bid price, or if for some reason there isn't enough extra capacity anymore, your EC2 instance will be terminated and reclaimed. If that sounds like something you'd prefer to avoid, then you might want to use normal on-demand instances instead.

Go to your Amazon EC2 dashboard, and launch a new instance. Make sure you're in the AWS region that makes most sense from where you'll be playing. (I'm in the Philippines right now, so I opted for the Hong Kong region.)

Step 1:

Select Windows Server 2019 Base.

Step 2:

Find and select g4dn.xlarge. This would be under the GPU instances category.

Step 3:

This will be the most involved step. Hang in there.

Purchasing options: opt for Spot Instances. Put in your bid price (I normally set bid price to be the normal on-demand price of the instance).
Persistent request: you'll want this checked, because it allows you to configure ...
Interruption behavior: set this to Stop. This allows you to stop your Parsec server whenever you're not actually playing, saving you more money.

If you don't do this, you can only terminate your server if you want to turn it off, and that's probably not something you want.

Those are really the most important settings here, but feel free to change stuff up if you want to (like, say, the subnets wherein the instance launches).

Step 4:

This is just about storage --- you'll want to put in as much storage as you think you'll need here. Keep in mind though that storage will continue to cost you even when your machine is stopped, so give this a good thought.

You'll also probably want to leave this at the General Purpose (gp2) setting.

You also can set up a small gp2 SSD as your root volume, then go nuts with a second storage disk set to Throughput Optimized (st1) if you plan on using it as the default install location for your games. (For example, Steam allows you to do this.) Setting it up this way will lead to cheaper storage costs (st1 costs about half of gp2).

Step 5:

You don't have to change anything here, but it's always a good idea to at least put in a Name tag for your server, just so you can easily find it later.

Step 6:

This step is crucial, because this is where you define what networking connections are allowed to and fro your machine.

At the very least, you'll want to open the following inbound rules:

(RDP) TCP/3389 from Anywhere
UDP/8000-8200 from Anywhere
UDP/9000-9200 from Anywhere
TCP/5900 from Anywhere — only if you want to access your machine via VNC

If you know the IP addresses where you'll be connecting from, you'll want to change those here too.

Step 7:

Confirm your settings, here and proceed with launching.

When you launch, you will be prompted for a key pair. Opt to Create a new key pair, download it, and keep it somewhere safe.

Your machine will take a couple of minutes to launch, but we don't have to wait for it to finish to do the next step.

Allocate an Elastic IP

Your Parsec host server will need to have a public IPv4 address that doesn't change. To do this, we'll allocate an Elastic IP address on our AWS account, and assign it to the instance we just launched.

Still on your EC2 dashboard, go to Elastic IPs on the left sidebar, and click Allocate Elastic IP address on the top of the screen. Just accept the defaults, and confirm.

Once you're back on the dashboard, select the Elastic IP you just created, then select Actions > Associate Elastic IP address. On the form that pops up, select the instance you just created, and select its private IP address as well (there should only be one anyway). Confirm that, and you're done.

Create an IAM user for the Parsec installation

During the installation process for Parsec, it will need an IAM user to download GRID drivers to augment the video card. Let's prepare this now.

Go to your IAM dashboard, select Users, and then select Add user. Give your user a name (something like parsec-host-server works great), and check Programmatic Access.

On the next screen, opt to Attach existing policies directly, then attach the AdministratorAccess policy. Complete the user creation.

Once the user is created, you'll be given the user's Access Key and Secret Access Key. Store this somewhere safe for now --- we'll use it in a bit.

Prepare your EC2 instance

By now the EC2 instance you created should have finished launching already. You will then need to connect to it via Remote Desktop.

Select your EC2 instance, then click Connect at the top. Follow the instructions for a Remote Desktop connection.

Once you're connected, you'll want to do a few things before you actually start installing Parsec on the machine.

Change the Administrator password

You'll probably want to change the password to something saner. Totally optional, but I prefer changing the password to something else so that the key pair we generated for the machine effectively becomes useless too.

Disable CTRL+ALT+DELETE to unlock

By default, Windows is set up to require you to press CTRL+ALT+DELETE whenever you unlock the machine --- once Parsec is set up (and we're not using RDP anymore), we won't be able to do that, so we'll need to disable this (so that Parsec can also autologin for us).

From your Start Menu, search for Local Security Policy. Navigate to Local Policies > Security Options, and change the Interactive Logon: Do not require CTRL + ALT + DEL option to Enable.

Install Parsec

Here we go. Open up Powershell from your Start Menu, then paste in the following commands. (Alternatively, go to the script's Github repo, and grab the one there.)

[Net.ServicePointManager]::SecurityProtocol = "tls12, tls11, tls"
(New-Object System.Net.WebClient).DownloadFile("https://github.com/jamesstringerparsec/Parsec-Cloud-Preparation-Tool/archive/master.zip","$ENV:UserProfile\Downloads\Parsec-Cloud-Preparation-Tool.zip")
New-Item -Path $ENV:UserProfile\Downloads\Parsec-Cloud-Preparation-Tool -ItemType Directory
Expand-Archive $ENV:UserProfile\Downloads\Parsec-Cloud-Preparation-Tool.Zip -DestinationPath $ENV:UserProfile\Downloads\Parsec-Cloud-Preparation-Tool
CD $ENV:UserProfile\Downloads\Parsec-Cloud-Preparation-Tool\Parsec-Cloud-Preparation-Tool-master\
Powershell.exe -File $ENV:UserProfile\Downloads\Parsec-Cloud-Preparation-Tool\Parsec-Cloud-Preparation-Tool-master\Loader.ps1

[Net.ServicePointManager]::SecurityProtocol = "tls12, tls11, tls"
(New-Object System.Net.WebClient).DownloadFile("https://github.com/jamesstringerparsec/Parsec-Cloud-Preparation-Tool/archive/master.zip","$ENV:UserProfile\Downloads\Parsec-Cloud-Preparation-Tool.zip")
New-Item -Path $ENV:UserProfile\Downloads\Parsec-Cloud-Preparation-Tool -ItemType Directory
Expand-Archive $ENV:UserProfile\Downloads\Parsec-Cloud-Preparation-Tool.Zip -DestinationPath $ENV:UserProfile\Downloads\Parsec-Cloud-Preparation-Tool
CD $ENV:UserProfile\Downloads\Parsec-Cloud-Preparation-Tool\Parsec-Cloud-Preparation-Tool-master\
Powershell.exe -File $ENV:UserProfile\Downloads\Parsec-Cloud-Preparation-Tool\Parsec-Cloud-Preparation-Tool-master\Loader.ps1

This script will download the Parsec preparation script by James Stringer, and execute that in the Powershell context. It'll basically prepare most everything for you, and you'll just need to watch it go to town.

It will ask you a few things from time to time, so don't leave it for too long.

Once Parsec has been installed, make sure you log into your Parsec account in it.

Parsec configuration changes

I had to make a few config tweaks on Parsec to make it work completely:

Host
- Hosting Enabled: Enabled (of course)
- Host name: Change this to something that makes more sense for you
Network
- Client Port: 9000
- Host Start Port: 9100

The install script above should have taken care of most everything else.

Other stuff to take care of

The install script should've taken care of installing Xbox software (for gamepad compatibility), and Razer software (for audio). To complete the setup, we'll need to do a few more things.

Set up Auto Logon

On your desktop should be an Auto Logon folder. Open it and run the script inside. Type in your user name as Administrator, and the computer password (you should've changed this by now). Confirm that auto logon has been completed successfully.

Run GPU Auto-updater

Also on your desktop is a shortcut for the GPU Auto-updater script. This will install any other missing drivers / software required for your video card.

Once you run this, it will prompt you for a few things --- just say yes when it asks if it's OK to reboot. This will also ask for IAM user credentials --- put in the IAM user access key and secret access key we prepared earlier here.

Your machine will need to reboot once or twice to complete this step. When the machine reboots, just connect to it again via RDP, and run the GPU Auto-updater tool again. It will tell you if the installation is complete when you do.

Set up auto-shutdown

Also installed by the script earlier is a script that sets up automatic shutdown for the machine when it has been idle for a while. Consider running this so that the EC2 instance stops when you haven't been using it (and saves you money).

I've personally set mine to auto-shutdown after 20 minutes of inactivity.

Confirm that you can connect via Parsec

Make sure your remote desktop session is disconnected at this point.

From another machine, install Parsec (not using the install script this time). Log into your Parsec account, and just confirm that you can connect to your Parsec host server.

Once you're connected, Parsec will function just like Remote Desktop earlier, but it will generally feel more fluid and faster.

At this point, you can then install your games (e.g. Steam), and your gaming apps (e.g. Discord), and use Parsec to play remotely.

Cleanup

There's a bunch of things you'll want to do once you've confirmed that you can connect via Parsec.

Close down your RDP and VNC ports in your security group

Remember the security group we created earlier for our EC2 instance?

Once you've set up your Parsec connection, you won't need to remote desktop and/or use VNC anymore (and even if you do, you can always just open the ports again).

So you'll want to modify the security group to remove the RDP and VNC ports to make your machine a bit more secure.

Remove the access credential from your IAM user

Also at this point, your EC2 instance won't need the IAM user you created earlier anymore.

To ensure it can never be used again (remember: this has Administrator permissions!), you can either delete the IAM user itself, or revoke the access credentials you've given it in IAM.

Outcome

Your latency will greatly depend on which AWS region you set up your machine, and where you're playing from. I set up Parsec hosts in the Singapore, Tokyo, and Hong Kong regions in AWS, with best results streaming to the Philippines from Hong Kong.

It also seems like the lag differs from game to game as well, even if the network latency is the same. I found that the gameplay experience of, for example, Crysis 2 and Witcher 3 were really good, but Borderlands 3 was very difficult to play --- even if the latency across all three games were the same.

Here are some more sample videos:

Dota 2

Witcher 3

Borderlands 3

Subnautica

Setting up Microphone Audio Filters in OBS using Reaper plugins

2020-04-13T00:00:00Z

In this time of global pandemic, not only are there more people jumping onto remote online meetings, but there seems to be more people trying out livestreaming and recording videos for the benefit of others online.

However, unlike most any other element you're including in your recording, audio from a microphone will tend to be very finicky. There's a lot of factors that go into the quality of spoken voice --- the background noises from where you do your recordings, the quality of your actual microphone, and even the nuances of your own voice.

If you're using Open Broadcaster Software (OBS) to record, I'd like to share a handful of things you can do beforehand to help ensure your voice sounds cleaner, crisper, and much more professional.

OBS and Reaper

OBS is free and open-source software for video recording and livestreaming. It's very powerful, and while it's not the simplest tool if you just want to get set up and running quickly, it's relatively easy to use considering what you can get out of it.

Reaper is an impressive software suite for working with digital audio. The suite costs USD 60.00 for a personal license (which a lot of my audiophile colleagues say is worth the investment). It's traditionally set up so that the audio from your microphone flows into Reaper, wherein you can set up a pipeline of audio filters, which eventually is then output into wherever it needs to go (normally, your recording). That's not what we're doing today (although it's an interesting setup to learn about as well).

Fortunately, a good amount of the audio filters that Reaper uses is available as a set of VST plugins for absolutely free. Even better, OBS also allows you to bring in third-party VST plugins to supplement and extend what you already get out of the box. So what we'll eventually end up building is a pipeline of Reaper audio filters that clean up your microphone audio before it goes into your stream or recording, all within the OBS software interface. Spiffy.

The only caveat is that the Reaper VST plugins are only available on Windows platforms, so you'll only be able to do this on a Windows machine.

Building the setup

You'll need to have OBS already installed on your machine --- we're assuming you already have that.

Download ReaPlugs

The Reaper VST plugins are collectively called ReaPlugs, and you can download the installer for your system here. Make sure you're downloading the correct executable for your computer (which is probably going to be the 64-bit version).

Conveniently, when you run the installer, it should automatically register itself into your machine so that OBS can discover it for the next steps.

Set our microphone input to a comfortable pickup level

If you haven't configured your mic yet to a default working setup yet, now is a good time to do so. This is important because each physical microphone model behaves and picks up sound differently, and all the next steps will build on top of how your specific microphone performs.

If your microphone has a physical volume knob, it's oftentimes best to set the mic to 100% volume in Windows, and only adjust the physical volume knob whenever you need to. This minimizes the potential distortion that your computer can introduce to the sound.

If your microphone does not have a physical volume knob, open your mic's settings on your computer, and initially set it at 100% with max microphone boost. Continuously speak into your microphone in about as loud a voice as you'd expect yourself to be doing while recording (louder than normal), and adjust the levels so that the audio meter shows the sound almost going to the very top on average, but not all the way. Whenever the audio meter goes to the max, there is a real possibility of the audio signal being clipped, and that's not something we want.

Once you've set your initial microphone pickup levels, you generally won't want to touch those again.

Set up the audio filters in OBS

In OBS, you'll ideally have your microphone already set up as an input audio device. If you right click on your microphone source, and select Filters, you can add in audio filters that OBS will run your signal through during a recording.

Since the filters are evaluated in order (from top to bottom), I'll talk about these in order as well.

Remove ambient / background noise

Unless you're working in an effectively soundproofed environment, you're bound to have some background noise where you're recording. Could be the ongoing hum of your computer, or your room's fan / airconditioning, or the general ambient wind noise from outside. What we want to do before everything else is to try to cancel out all that background noise. Thankfully, that's easy to do with the Reafir filter, and also one of the easiest filters to set up.

Add a VST plugin into your filters, and select the Reafir plugin on the right side of the OBS panel. Afterwards, go into the plugin settings.

In the settings panel, set the mode to Subtract (since we're essentially removing sound), then click on Automatically build noise profile --- leave it on for 3-5 seconds, then turn it back off. Don't talk or make any noise during this time.

During that time, the plugin will listen on your audio signal, and automatically determine the audio frequencies (and at what levels) it will need to adjust output audio signal to. Since it's set to subtract mode, we're essentially telling the filter to remove the audio it heard during those 3-5 seconds from any audio signal that goes through it in the future.

My Reafir plugin configuration after building my personal noise profile

Remove everything else aside from your voice (ideally)

The above will work wonderfully for static noise, but for things like keyboard clacking, random noises, accidentally clicking your tongue, etc. it's not going to do a lot. Since ideally we'd only want our own voice to carry through to the recording, we also need to proactively remove most other arbitrary sounds. This is what a noise gate is normally for.

A noise gate works by continuously listening in on a specific volume threshold --- if the audio is within this threshold continuously for a set amount of time (generally what happens when we talk), then the noise gate will open, and let the audio signal through. Otherwise, the noise gate prevents any audio signal from getting through at all. It's a bit of an all or nothing.

So add another VST plugin into your filters, and set it to a Reaper noise gate. Again, just jump into its configuration.

There's a bunch of stuff here that we'll need to tweak. So here goes:

To the left is a vertical volume threshold meter. Speak into your microphone continuously in your most normal cadence, volume, and tone --- how you'll be speaking most of the time during your recording.

The green meter will bob up and down as you speak, so you'll want to bring the slider down so that it's just below of the general peak area of the green meter when you're speaking normally.

You're essentially telling the noise gate to consider only audio that go above that volume threshold level and discard anything lower.
Pre-open is how long the audio has to be (continuously) above the threshold level before the noise gate allows audio through.
Attack is how quickly the gate opens when it determines that it needs to allow audio through.
Hold is the opposite of pre-open --- once the noise gate is open, this determines how long the audio has to be (continuously) below the threshold before it closes back again.
Release is the opposite of attack --- how quickly does the gate close when it needs to.
Hysteresis turns your threshold from (1) above into a range instead of just a single number. To vastly oversimplify things, if the noise gate only used a single threshold number for both deciding when to open and when to close, the audio tends to have a lot of chattering.

Your hysteresis setting tells your noise gate what your closing threshold is (as you want this lower than your opening threshold). If you have a threshold of -20 dB, and a hysteresis of -5 dB, then your noise gate uses -20 dB to determine when to open, and -25 dB to determine when to close.

I've read that a value of -6 to -8 dB is a good starting value for this.
Lowpass and highpass tells the noise gate what range of audio frequencies to consider for deciding when to open and close. As you can imagine, this works hand-in-hand with your threshold setting above.

I've read that 85-255 Hz (that is, a lowpass of 255, and a highpass of 85) is a really good average for male and female voices, but you'll definitely want to tweak this to better match your own vocal tone.

Setting this correctly is going to very significantly affect how much benefit you get from your noise gate, because it'll be better at pinpointing your voice from other noises (instead of just relying on how loud your voice is).

Your noise gate config will vary based on your own vocal nuances.

Equalizing

I personally think an equalizer is the hardest thing to tweak in this whole set, largely because there's so many factors that go into it, and that it's the filter that is most dependent on the nature of your own voice.

I've received the following recommendations that I've used for my own setup, which only use 2 bands:

Band (1) is a High Shelf filter, meant to give a bit more oomph to the pops in your own speech (like consonants or stops). Set this to around 3000 Hz first with a bandwidth of 3 octaves, and a gain of around 10 dB, then adjust based on how your own voice sounds in recordings.
Band (2) is a High Pass filter, meant to cut some of the bass in your voice. Set this to around 60-80 Hz, and play around with it to match your preference.

Obviously, you can do more with this, but I honestly don't know enough about it to tell you more.

Your equalizer settings are going to heavily depend on your own voice.

Cut out sudden loud shouts

Most especially if you're livestreaming video games like I do, you're bound to find yourself in situations that just make you suddenly scream loudly, or shout at your teammates, or other similar things. While this is all fun and exhilirating for the player, anyone watching wouldn't appreciate being suddenly shot an earful of maxed-out vocals without a volume warning.

This is what an audio compressor is for. A compressor will weaken the volume of loud sounds, as well as amplify the volume of quiet sounds, so you kind of hit two birds with one stone --- it'll cut out the volume whenever you shout, but it will also make your own whispers more pronounced and intelligible. Perfect when you're sneaking around a game!

Throw in another VST plugin, and set to a compressor (not the compressorx!), and jump into the settings.

Attack and release work in pretty much the same ways as the ones for your noise gate above. I've been recommended 5 and 50 ms for these respectively as a starting point.
Ratio is an indication of how much the filter compresses the audio signal --- a 6.0:1 ratio should be a good start.
You'll probably want to set lowpass and highpass to the same settings you've used for your noise gate too.
Threshold will need a bit more TLC. You'll want to speak continuously at a low volume --- around as soft as you'd speak when you're trying to be quiet in your recordings --- and set the threshold a little below that. If you're seeing that your voice is not registering at the threshold meter when you're speaking softly, that's probably a good indication that you need to tweak your noise gate a bit more.

The goal here is to tweak the settings above so that when you're speaking --- all across soft, normal, and loud voices --- your output audio is registering just below peak on average (probably around -6 to -12 dB is good). You definitely don't want to have it go into the positive, as this indicates audio clipping.

Make sure your audio output at the right does not go into the positive across your expected vocal range.

Plugging overly loud sounds

Your compressor will do pretty much everything to keep your audio output within a certain acceptable volume --- unless you happen to speak so overly loud that it just overwhelms the compressor.

You can optionally put in a master limiter to your audio filter chain to prevent those (supposedly) very rare occassions.

Again, throw in another VST plugin into your chain. There won't be a standalone limiter filter in the choices, so select ReaJS, and in the configuration settings, search for masterLimiter in the loser category.

You'll pretty much only want to tweak two things here:

set threshold to around -3 dB, and
set limit to around 0.1 dB

This will effectively stop your audio signal from ever going into the positive.

Maybe a gain?

If at the end of this all, your microphone audio seems to be lower in volume than you initially expected (which shouldn't really happen noticeably), you may want to put in a gain filter to help boost the mic level back up. Totally optional.

A final look

So my final audio filter chain looks like this after all that:

You should have an ultimately cleaner, crisper, and oomphier microphone audio after correctly setting up yours.

tech, life & music.

I'm blocking private IPv4 addresses on my servers by default from now on

Oops.

Protecting myself from myself

Visiting localhost from a physical Android device

So how then?

Patching Berkeley Mono v2 with Nerd Font glyphs

Configuring Paperless-ngx to monitor emails from a Proton Mail account

How I got it to work

Caveats

I'm trying out Nobara Linux for my gaming PC

Nobara?

How did the install go?

How's the gaming?

Copying files to a volume on Fly.io

So, how?

Caveats

I used ChatGPT instead of Stack Overflow today

Using it to work through today's Advent of Code problem

Splitting arrays

Splitting a string in groups of N characters

Parsing instructions using regex

Concatenating arrays

How did that feel?

I'm building my own hitbox-style fightstick controller

Yeah, totally logical

1. Slightly bigger buttons

2. Slightly angled button positioning

3. Optional RGB LEDs

4. Acrylic case

A first draft

A second draft

Build log: Vortex Pok3r hotswap mod

Disassembly

Desoldering

Hotswap sockets

PE foam mod

Re-assembly

Et voila

Validation checks in AWS CDK constructs

CDK constructs and .validate()

Adding validation checks

An example instead?

Let's add in validations

Zip files from Amazon S3 without saving them locally

Let's not download the files then

Throw the files in

Tracing through your distributed applications using AWS X-Ray

Setting the stage

Enabling tracing through AWS X-Ray

Going the full mile with tracing

At this point, you're already done.

You're curious to see how it looks

Setting up a load testing platform

Inspecting the traces

Add in some artificial defects

There's definitely more to this

Working with the Amazon Chime SDK

The skinny

Creating an Amazon Chime meeting

Connecting to an Amazon Chime meeting

Connecting audio

Connecting video

That should be it!

Benchmarking the AWS Graviton2 processors

Methodology

Results

How to: Create a Remote Gaming Server using Parsec on AWS

Parsec?

Costs

But will it run Crysis?

Let's get started

Create a Windows-based EC2 instance

Step 1:

Step 2:

Step 3:

Step 4:

Step 5:

Step 6:

Step 7:

CDK constructs and `.validate()`