ITGmania project is an effort to modernize the StepMania game, a Dance Dance Revolution arcade game simulator. It’s a fork that receives regular updates, both security and new features.

In January I proposed to replace an old vendored libjpeg library with a git submodule¹ of a popular drop-in replacement, libjpeg-turbo. It uses SIMD instructions to accelerate compression and decompression operations, so we could also get some performance gains for free. The crucial part of the whole story is that libjpeg-turbo uses CMake as its build system.

That’s great news because ITGmania also uses it, so the old glue CMake module for building libjpeg could be replaced with an ExternalProject. I wrote a short module to specify the library, how to build it, and the target for linking in the main project.

After some tweaking related to the Ninja generator that’s by default used on Windows in Visual Studio, all the obvious problems have been resolved and the automatic builds in the CI pipeline in GitHub Actions were green. The game worked with the new library.

The PR was merged a week ago, so the “real-world integration” could begin ;) First, there was an issue reported about linking the project on Arch Linux. It turned out that the output lib directory was using a different scheme – lib64 instead of lib that was expected by the linker. I quickly fixed that and submitted a PR.

Yesterday, teejusb, the project maintainer, contacted me about linking issues on macOS. He got an Undefined symbols for architecture x86_64 message related to some symbols from the libjpeg library, so I assumed he was building on an x86_64 macOS. That’s weird because we perform that build in the pipeline on every commit and it passed.

I don’t have a Mac, so I created a fresh KVM VM with macOS in order to reproduce the issue. I had some problems with getting the GPU passthrough to work, but I managed to build and start the game on a virtual GPU. It compiled and linked – as expected.

What he didn’t tell me was that he was trying to cross-compile the x86_64 target from his Apple Silicon macOS (some aarch64). I traced the issue to the two bugs in these 3 lines of the original PR:

if(MACOSX)
  set(ARCH_FLAGS "-DCMAKE_OSX_DEPLOYMENT_TARGET=${CMAKE_OSX_DEPLOYMENT_TARGET} -DCMAKE_OSX_ARCHITECTURES=${CMAKE_OSX_ARCHITECTURES}")
endif()

The first one being the wrong variable to check for macOS – it should be APPLE, not MACOSX. The other one was more intriguing, it was related to how ARCH_FLAGS were used later in the ExternalProject_Add call within CMAKE_ARGS.

include(ExternalProject)
ExternalProject_Add(
  libjpeg_turbo_project

  SOURCE_DIR ${SM_EXTERN_DIR}/libjpeg-turbo
  INSTALL_DIR ${INSTALL_DIR}
  CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR}
             -DCMAKE_INSTALL_LIBDIR=${LIB_DIR}
             -DENABLE_SHARED=OFF
             -DENABLE_STATIC=ON
             ${ARCH_FLAGS}
  (...)

After changing ARCH_FLAGS to a list, I ended up with:

if(APPLE)
  LIST(APPEND ARCH_FLAGS
    -DCMAKE_OSX_DEPLOYMENT_TARGET=${CMAKE_OSX_DEPLOYMENT_TARGET}
    -DCMAKE_OSX_ARCHITECTURES=${CMAKE_OSX_ARCHITECTURES}
  )
endif()

This now properly passes the target and architecture to the libjpeg-turbo build. I’ve submitted it as a Pull Request this morning.

The old code never executed the if branch in the first place, and even when I changed the condition to APPLE, the variable didn’t work as intended. So, why did it seem to work? Why was it merged? It’s simply because the builds in the pipeline never exercised the cross-compilation path – they used separate hardware for each macOS build.

I also started an issue to propose new automatic builds to cover more actual use cases, not just the basic ones. They will hopefully prevent similar issues from happening in the future.

Background

Let’s step back for a moment to introduce a little background. I’m a Tech volunteer at GSPS Foundation which hosts the Polish bi-yearly (as-in happening twice a year) charity speedrunning marathon based on the formula popularized by Games Done Quick (GDQ). GSPS stands for Gramy Szybko, Pomagamy Skutecznie, which roughly translates to “Gaming Swiftly, Providing Support”¹ when we try to keep the acronym.

The next GSPS event will be GSPS Dzieciom 2025. It’s scheduled for March 26-30. You can buy a ticket and visit us in Kraków in person or watch the live transmissions on Twitch and YouTube.

The Foundation owns a digital audio mixer, a Behringer XR18, which offers great mixing capabilities for its price. We currently use all but one of the input channels during the events, so we’re dangerously close to the limit.

GDQ has an interesting way of indicating who is currently speaking on the stream.

GDQ’s audio input indicators.

It mimics the classic UV meter bars that meet at the center of the speaker’s label. I wondered if it would be possible to implement something similar in our stream layouts because I knew that they use an older brother of our mixer, an X32.

Oh, There’s An API

Behringer’s X-AIR-Edit app is the official way to control the mixer. It somehow manages to display the current signal input level for each channel in nearly real-time. I just needed to know how, so I could use that information in our code.

X-AIR-Edit channel input indicators – the green bars next to the faders.

Onyx, an old friend of mine, has already researched the topic of the API offered by the whole family of Behringer’s digital mixers and wrote a python library for interacting with them. It turns out that they use a standard OSC (Open Sound Control) protocol over UDP.

It’s a simple protocol. Each message consists of an address and a types tag, followed by some values for the fields specified in the aforementioned tag. The protocol requires 4-byte alignment using null bytes. Luckily, there are libraries to handle the protocol, so we don’t have to (yet ;).

I contacted Onyx because I couldn’t find anything in his library that would allow me to query the input levels. He directed me to the official docs, which are allegedly suitable for the firmware version “1.11 or higher” (our mixer runs on the newest, 1.22 firmware). I found what I was looking for. The /meters command was supposed to be an answer to all my needs.

Enter The Docs

The input levels that I was interested in are returned as a “blob” of bytes. For example /meters/2 yields an i32 (the size of the payload) and 36 i16s (16 analog, 2 aux, and 18 USB inputs) that we have to decode manually because the OSC protocol doesn’t specify i16 type, so the OSC libraries will just return the raw bytes. The i16 values’ resolution is specified to be 1/256 dB, that’s pretty simple and more than enough for our needs. To convert them to dB we just need to divide the raw value by 256.

So far so good, I quickly started hacking some PoC to connect to the mixer and request the /meters/2. Everything worked as expected, I could read and decode the values and match them with what I was seeing in the official app.

I was happy that it worked. However, that hasn’t lasted very long. Every time I started my script, it only worked for roughly 10 seconds before freezing. The docs for /meters/ clearly only stated that:

Results in regular updates to batch of meter values as a single binary blob (…)

Debugging

I checked that there were always 200 responses from the mixer every time I started the app and subscribed to the meters. One every 50 ms; that gives the 10 seconds I observed. Perhaps that’s an intended behavior, not a bug.

Technically, I could just re-request the /meters/2 every 200 messages or 10 seconds, but considering that this runs over UDP, there are no guarantees for deliveries or timings.

I decided to sniff the traffic of the official X-AIR-Edit app with Wireshark to see what they do. I knew the mixer’s address and that we’re using UDP, so the filter was a pretty straightforward one, like: udp && ip.src == 192.168.11.167. There was a lot of traffic exchanged between the mixer and the app.

I further refined the filter to look just for the meters: && data contains "meters/1" (they used different combinations of meters to display the same info; /2 is what I ultimately needed, but that’s not important here). That’s when I realized that every two seconds there’s a periodic message from the app to the mixer, that looks like this:

Periodic message sent from the X-AIR-Edit app to the mixer captured in Wireshark.

Huh? What’s that /renew OSC command? I haven’t seen anything about it in the docs.

Renew The Subscription

It turns out that there’s an undocumented feature that refreshes the subscription for the meters. That makes a lot of sense, as it lets the mixer stop sending the data in case of network issues or app failures, we’re using UDP after all.

The syntax is pretty obvious (dots for null bytes):

/renew..,s..meters/1....

,s means that there will be a single null-terminated string in the payload.

I quickly added a periodic /renew call to my code and observed that the mixer now sends me the updates without any interruptions.

Final Integration

GSPS stream layouts are based on the NodeCG framework that supports building interactive graphics and overlays for live broadcasts. I wrapped everything together into a NodeCG extension and added some glue code for passing the state between the extension and the rendered overlays. Here’s a preview of the final effect:

I didn’t really like the aggressive animation on the GDQ layouts, so I opted for the subtle background highlights.

The implementation has been submitted to the public repository in this Pull Request.

Closing Words

If you’re offering a public API, please document it properly and keep it updated – it saves developers time and frustration. The Wireshark part in this story could have been entirely avoided if the manufacturer updated the docs.

I’m not entirely sure what’s the case here – was the /renew not available in the 1.11 firmware? Is it an omission and the docs will eventually be updated? Was it a deliberate decision not to share that endpoint, hence rendering the entire /meters feature inconvenient to use?

I usually prefer to read everything in one place myself. I understand it’s tempting for some portals to monetize through ads or deals when people visit them directly. It also allows them to track traffic, user behavior, etc. to focus on creating content that sells better.

However, I don’t have any ads or tracking, and I don’t plan to introduce them in the foreseeable future. And I’m all in for letting people consume the content in the way that’s most comfortable for them. I believe that serving the whole posts over RSS also makes them more accessible for people with specific needs, who require bigger fonts, higher contrast, or text-to-speech because they can customize everything in one place.

My blog’s RSS used to include only the description tag with the first paragraph, but now you can read everything if your reader supports <content:encoded> tag. The blog is powered by Hugo so it was only a matter of changing image rendering to use absolute URLs and including the <content:encoded> tag for each post in the theme.

I also updated the theme to use new Hugo configuration options instead of the deprecated ones to silence the build warnings that were soon to be errors even though I don’t use them. I found out that the standard authorship tags were also missing from the rendered XML, so I updated them to match the variables defined in my site’s config.

Numbers

Let’s talk numbers. My main disk, a modest 500 GB SSD by today’s standards, can fill up pretty quickly.

$ df -h /
Filesystem                  Size Used Avail Use% Mounted on
/dev/mapper/vgxubuntu-root  467G 467G     0 100% /

A lot of data has crossed this disk since the system installation at the beginning of 2021:

# smartctl -a /dev/nvme0
(...)
Data Units Read:                    12,215,530 [6.25 TB]
Data Units Written:                 25,077,401 [12.8 TB]

I started the cleanup by removing some undoubtedly optional files like ~/Downloads/ and migrating data files to the second drive. I managed to salvage a decent bit:

$ df -h /
/dev/mapper/vgxubuntu-root      467G  421G   23G  95% /

So where is the rest of the junk? I was astonished when I found out that 25% of my disk space was occupied by various caches!

The Root of All Evil

I’ve identified the three most painful storage eaters:

• Docker
• Poetry
• pip

That’s not a huge surprise given that I mainly focus on Python development.

Docker

I use Docker for various stuff, such as running legacy projects that require ancient system deps, quick testing in limited prod-like environments, or just experimenting with unsafe code.

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          44        9         80.36GB   78.29GB (97%)
Containers      11        1         2.618GB   2.618GB (100%)
Local Volumes   47        5         1.219GB   983.4MB (80%)
Build Cache     0         0         0B        0B

You can see the full summary of your docker daemon with docker system df -v. There’s a mostly safe command for removing all stopped containers, unused networks, dangling images, and dangling build cache:

$ docker system prune
(...)
Total reclaimed space: 8.371GB

There’s also a more aggressive option -a, that will also remove all images without at least one associated container and the whole build cache:

$ docker system prune -a
(...)
Total reclaimed space: 73.91GB

I decided to go with -a right now, to have a fresh start. 80 GB reclaimed.

Poetry

In 2023 we started using Poetry to manage our Python environments at work. Check it out, it’s finally production-ready after maturing for a couple of years (the early performance issues are now gone!).

I couldn’t find a built-in command to list Poetry’s cache size. There’s poetry cache list, but it only shows cached package registry names:

$ poetry cache list
PyPI
_default_cache
secret-registry-1
secret-registry-2
(...)

By default Poetry keeps the cache in ~/.cache/pypoetry, but it can be configured with the POETRY_CACHE_DIR env variable:

$ du -sh .cache/pypoetry/* | sort -h
11G	.cache/pypoetry/artifacts  # packages (wheels, tarballs)
12G	.cache/pypoetry/cache  # package metadata used to build `poetry.lock` file
99M	.cache/pypoetry/src  # this one stores source repositories; for example, when deps are specified with a git URI
21G	.cache/pypoetry/virtualenvs  # this one stores actual virtualenvs ;)

You can use poetry cache clear --all <cache name> to delete all cache entries from a given repository. I find it overly complicated to use:

$ poetry cache clear --all _default_cache
Delete 1208 entries? (yes/no) [yes]

There’s an undocumented option that lets you pass . as a cache name to clear all caches:

$ poetry cache clear --all .
Delete 4426 entries? (yes/no) [yes]

Unfortunately, none of those will clean the artifacts directory, so we’re left with:

$ rm -r ~/.cache/pypoetry/artifacts/*

That’s an extra 23 GB.

Legacy pip

I don’t use pip directly anymore since the migration to poetry. Some leftovers were lurking in the shadows:

$ pip cache info
Package index page cache location (pip v23.3+): /home/florczak/.cache/pip/http-v2
Package index page cache location (older pips): /home/florczak/.cache/pip/http
Package index page cache size: 5398.9 MB
Number of HTTP files: 2218
Locally built wheels location: /home/florczak/.cache/pip/wheels
Locally built wheels size: 15.6 MB
Number of locally built wheels: 41

Yay, another 5 GB reclaimed.

Closing Words

The cache is crucial for optimizing performance, but an unmanaged cache leads to resource drainage. When you design an application that heavily relies on cache let your users know the consequences. Create cache invalidation/cleanup policies that make the most sense in your use cases.

I suspect that at least half of the 108 GB cache space I freed was stale and will never be populated again. Only time will tell.

Ethical Disclaimer

This post contains information that can be used to cheat (and avoid being caught red-handed) in Call of Duty 4 (CoD4) multiplayer servers that require latest official version of the game. This also applies to servers with the default anticheat software enabled at both ends of communication. Without a need to modify any of the game files.

I decided to finally publish this post because the game it’s mostly related to has been released in 2007 and the latest official patch (version 1.7) has been released in June 2008, 14 years ago. All the information presented here can be found on the internet anyway.

Intro

Most new games nowadays don’t have option to let people host and manage (and often modify) their own dedicated servers. However, a decade ago the situation was a complete opposite. Before the age of automated matchmaking, you had to manually find and join a server to play. Usually there were a lot of active servers but not all of them were given a top-notch care from the administrators in all matters.

Some issues were related to keeping the chat civilized. Others revolved around abusing some gameplay mechanisms. However, the worst problems were always related to detecting and removing people who cheated. It’s a complex topic thus I see why it’s usually ignored or delegated to the game’s developers and calling it a day.

All the information in this post is based on my knowledge gathered over the years of hosting various game servers between 2009 and 2016. The most interesting years were definitely the final ones under the Morning Star Gaming¹ brand, where we had access to high-end hardware that allowed us to deliver lag-free experience for thousands of unique players over the years.

In order to visualize some of the topics better, I fetched old backups so that I could use real-world data. The logs I will use come from the CoD4 servers that were used for competitive gameplay between 2014-07-01 and 2015-07-01.

PunkBuster

PunkBuster is anti-cheat software suite created by Even Balance back in 2000. It was first used in Return to Castle Wolfenstein and since then in over 30 other titles including games from Battlefield, Call of Duty and Far Cry series. CoD4 has an optional (as in possible to be forced by each game server to let people in) PunkBuster integration.

PunkBuster works in client-server architecture to allow real-time memory and file scans. It also allows to stream globally issued bans from the master servers² to the participating servers.

PunkBuster was rarely updated, even at the peak of its popularity, therefore only the most obvious and popular cheats were possible to detect this way. The fact that PunkBuster worked in userland didn’t help in case of more sophisticated hacks.

But wait, there’s more!

Built-in Cheat Detection

There’s an undisclosed database of known programs used for cheating that PunkBuster looks for. They can target either the game or the anti-cheat itself.

You can find some examples from my logs (sorry for the MM.DD.YYYY date format but all excerpts are copied as-is) below:

[08.15.2014 17:59:58] VIOLATION (GAMEHACK) #89182: sSs*R|neon (slot #1) Violation (GAMEHACK) #89182 [6fcfb7e796d093e7b781a94e81f10015(VALID) 89.12.208.250:28960]
[08.28.2014 12:18:15] VIOLATION (GAMEHACK) #89182: DK'NEON (slot #8) Violation (GAMEHACK) #89182 [1c8d0554b041a5bf4b81ba7657dcc329(VALID) 89.12.241.2:28960]
[09.02.2014 17:31:45] VIOLATION (PB HACK) #139015: |G4E|SaschaDE (slot #1) Violation (PB HACK) #139015 [24cede36831db30128d586a3387319d8(VALID) 188.195.155.156:28960]

First two entries show a generic GAMEHACK entries, the last one indicates a cheat that interfered with the PunkBuster (PB HACK). All violation log lines are pretty much similar. They consist of a timestamp, violation type, nick and game slot of a player in question, violation details (they are the same as violation type in case of these built-in detections), GUID (a unique identifier of a player derived from the game’s CD-KEY) and client’s IP.

Screenshots

PunkBuster offers a somewhat limited but still sometimes useful functionality of taking screenshots of player’s screen and sending it back to the server. It can be used, but not limited, to spot (wallhacks)[https://en.wikipedia.org/wiki/Cheating_in_online_games#World-hacking] (a class of cheats designed to reveal enemies in some ways, not necessarily making the walls invisible ;). Unfortunately, as with other aspects of PunkBuster, this feature was designed when commonly used screen resolutions were much smaller. As a result, we end up with a small fraction of the screen available or a decimated picture that convey no meaning.

Example of a screenshot containing a wallhack.

There were also hacks to disable some or all of the screenshot functionalities. Beside the hacks there were also bugs in game, DirectX or PunkBuster itself that prevented taking proper screenshots.

Black screen. Either a bug in game, PunkBuster or a deliberate doing.

Screenshot of my foobar2000 instead of the game.

Example of a screenshot blocker that hides player info.

Examples of other cheats found with screenshots:

Aimhack.

Crosshair hack, probably just an all-in-one hack because the line on the right is most likely a part of a bounding box around enemy.

Custom textures are indicator of modified game files. There’s also an ESP (Extra Sensory Perception) hack that shows info about enemies other than just their position.

Hack config.

Radar hack. Positions of all enemies are always shown on the radar, not only when scanned by in-game UAV or when shooting without a silencer. Achievable with setting read-only g_CompassShowEnemies to 1.

An overlay of a Mombot hack.

CVAR Violations

Now we can move on to the less known features of PunkBuster. As an administrator of a server you can also define a list of game variables (CVARs) to scan periodically. They can be set in couple different modes: value range (both in and out) as well as string inclusion/exclusion.

Here’s a general definition from the docs and an excerpt from the config file we used back in the days (; denotes a comment in PunkBuster config syntax):

;PB_SV_Cvar [Cvar_name] [IN/OUT/INCLUDE/EXCLUDE] [Param1] [optional_Param2]

;melee checks
pb_sv_cvar aim_automelee_lerp IN 40
pb_sv_cvar aim_automelee_region_height IN 240
pb_sv_cvar aim_automelee_region_width IN 320
pb_sv_cvar aim_automelee_enabled IN 1
pb_sv_cvar aim_automelee_range IN 128
pb_sv_cvar aim_automelee_debug IN 0
pb_sv_cvar player_meleerange IN 64

;disabled anti-aliasing (more on that later!)
pb_sv_cvar r_aasamples IN 1

;misc
pb_sv_cvar developer IN 0
pb_sv_cvar sv_cheats IN 0
pb_sv_cvar g_compassshowenemies IN 0
pb_sv_cvar r_fullbright IN 0
pb_sv_cvar com_maxfps IN 40 333

This method has worked great for some time. It allowed us to find cheaters that used cvar unlocking hacks – programs that basically removed the read-only status from some variables which values, when changed, could give someone an unfair advantage.

And guess what? CoD4 has a built-in method of changing the read-only variables! It’s possible through a side-effect of a quite complicated scripting function for loading translations from files to variables. Namely, SelectStringTableEntryInDvar foo foo sv_cheats 1 command would fail to find a foo in foo so it would just forcibly set sv_cheats to 1 despite the variable being set as read-only. This bug hasn’t been officially patched. And as far as I can tell, it even made to the next installment of the game!

Below you can find some examples of variables violations and stats from the period covered by the logs:

[10.16.2014 20:15:04] VIOLATION (CVAR) #9001: testing stuff (slot #1) Cvar sv_cheats = 1 [db4a323740c24b9a31915c3e636e8fdb(VALID) 217.99.224.187:28960]
[07.15.2014 18:48:58] VIOLATION (CVAR) #9001: mM#neon (slot #2) Cvar developer = 1 [c108d2381c62d8e0aadc17ffa941bfc2(VALID) 89.13.148.61:28960]
[09.22.2014 18:41:58] VIOLATION (CVAR) #9001: Dougmau5 (slot #7) Cvar r_fullbright = 1 [9ed7ac09b6b429a2f04357d3419a658d(VALID) 81.131.93.97:28960]
[09.02.2014 15:36:00] VIOLATION (CVAR) #9001: Fire (slot #3) Cvar aim_automelee_range = 255 [babaa765e4257430197cce9b0c8a008a(VALID) 31.183.36.255:55195]
[09.18.2014 13:36:49] VIOLATION (CVAR) #9001: spec (slot #1) Cvar cl_yawspeed = 22500 [de20c946db20b3f64758ee24b48c4c23(VALID) 89.14.116.144:28960]
[08.12.2014 12:57:59] VIOLATION (CVAR) #9001: LS|Wojtix. (slot #8) Cvar j_lodScaleRigid = 4 [cf108bcb2a4ca15408e5b78e79d5a62e(VALID) 89.230.231.36:28960]
[09.27.2014 21:34:08] VIOLATION (CVAR) #9001: Mokova (slot #5) Cvar fx_enable = 0 [01174c7e83fd6cc9a52db198899ac0c9(VALID) 96.22.12.28:28960]
[08.27.2014 13:02:21] VIOLATION (CVAR) #9001: sSs*|Asqwe:3 (slot #1) Cvar cg_hudDamageIconHeight = 0 [ffa81a6a32c10f23b68a6e65f87512d8(VALID) 86.111.104.134:28960]
[08.29.2014 18:57:15] VIOLATION (CVAR) #9001: RAGE HorNed (slot #5) Cvar phys_bulletSpinScale = -1 [2bfe80aa39ec04249a2897370f7301c5(VALID) 95.52.34.145:28960]
[09.16.2014 00:22:20] VIOLATION (CVAR) #9001: ^3[^0ms^3]^0nor (slot #1) Cvar cl_forceavidemo = 1 [9126e989c56bfbc92307082716646ab8(VALID) 86.16.229.77:28960]
[08.21.2014 22:00:25] VIOLATION (CVAR) #9001: =sQ=TEST CFG GG (slot #7) Cvar cg_nomip = 128 [4e7dbd87cf2bb93d7359334939aa463d(VALID) 193.106.161.132:26456]
[08.12.2014 15:15:13] VIOLATION (CVAR) #9001: neon (slot #8) Cvar phys_bulletUpBias = 0 [6fcfb7e796d093e7b781a94e81f10015(VALID) 89.12.93.110:28960]
[07.08.2014 13:02:06] VIOLATION (CVAR) #9001: M][K*knifer (slot #5) Cvar cl_punkbuster = 0 [89377397ef9633e2b30c1f0f7ad02e3d(VALID) 81.241.85.246:28960]

• Total CVAR violations: 994
• Unique CVAR violators: 296

Top offenders:

     62 8b2fcad7b931f60ebfff1631ab97562a
     29 6fcfb7e796d093e7b781a94e81f10015
     26 5f58a8d513865b56df363d380d72f71d
     21 ffa81a6a32c10f23b68a6e65f87512d8
     20 3d689aa282bf6ade29d809f4e8ec7c1b

r_aasamples equal to 1 means that the anti-aliasing feature in the game’s renderer is disabled. It became a problem in newer Windows versions when it came to screenshots taken by PunkBuster. All of them came back pink when anti-aliasing was enabled so we decided to disallow it altogether.

Pink screen caused by enabled anti-aliasing on Windows 8.

There’s also an on-demand version of this functionality that allows you to just peek at the values without issuing kicks or bans. It was useful early on, to collect as much data about cheaters as possible without alerting them.

[01.03.2015 18:29:23] Malfunctioning PB Client (slot 12) - Ignoring Cvar Queries

[07.29.2014 10:53:58] 5 Cvar Queries Sent
[07.29.2014 10:53:58] [From #1 b7e8(VALID) {RR}andr] ai_meleerange = "64"
[07.29.2014 10:53:58] [From #7 143b(VALID) =sQ= Savie] ai_meleerange = "99999999"
[07.29.2014 10:53:58] [From #3 54b4(VALID) *LEU*Romeo] ai_meleerange = "64"
[07.29.2014 10:53:59] [From #4 2c81(VALID) [VS-UK]psychOl] ai_meleerange = "99999999"
[07.29.2014 10:53:59] [From #2 c664(VALID) =sQ= SKiLLzZ] ai_meleerange = "64"

Do you remember when I said that this method worked great for some time? It was fine until cheaters learned that they could modify strings in game’s executable to use different names for the variables we were enforcing values for. For example, the variable aim_automelee_ranga should not exist at all because it’s real name is supposed to be aim_automelee_range:

[07.06.2014 19:26:01] VIOLATION (CVAR) #9001: god#D1veR (slot #1) Cvar aim_automelee_ranga = 155 [f289fb8c7131e39cafc4fe6c8552d040(VALID) 81.4.194.85:28960]

But how did we even realize this was the case?

Bind Scans

Lo and behold, here come the bind scans. This little functionality allowed us to poll players’ configs for known game actions assigned (or “bound” in gamers jargon) to keys. On some occasion they also revealed illegal macros used to perform actions in sequence that were impossible otherwise.

Cheaters sometimes assigned extra stuff to the normal movement or attack keys. That way we could reveal some of the custom variable names that their modified game clients actually used.

I’m sorry but I couldn’t find full logs, only partial messages from admins communication because bind scans were performed on-demand and weren’t logged to the main log file. Here’s what I managed to salvage for the sake of this post:

[07.01.2015 18:04:14] [From #4 9ee6(VALID) rOSTA|FFF] [Q = weapnext; wait 2; weapprev; wait 5; +melee; com_maxfps 80; -melee;wait 33; com_maxfps 333; wait 18; +activate; wait 2; -activate][1 key binding match found]
[07.01.2015 18:04:53] [From #4 9ee6(VALID) rOSTA|FFF] [3 = zzz_popokills_mamba 129][W = +forward;z_loh 0;zz_kokokillsmamba 0;zz_tutu 0;zz_lalala 0;sv_clientsidebullets 0;r_znear 30;r_znear_depthhack 4;bg_bobmax 0;fx_drawclouds 0][2 key binding matches found]
[07.24.2014 12:24:29] [From #10 426f(VALID) dUb crizis] [E = +melee][KEY_F3 = aim_automelee_ranga 165][KEY_F4 = aim_automelee_ranga 0][3 key binding matches found]
[07.07.2014 16:10:36] [From #7 2543(VALID) BT#XtronS] [F = +activate][V = weapnext; wait 2; weapprev; wait 5; +melee; com_maxfps 125; -melee; wait 40; com_maxfps 333; wait 18; +activate; wait 2; -activate][2 key binding matches found]
Z = "SelectStringTableEntryInDvar FPS FPS r_fullbright 1"

I can assure that neither zzz_popokills_mamba nor zz_lalala or even zz_kokokillsmamba are built-in variables ;) They are, however, matching the lengths of their original names because they were patched directly in game’s binary without changing any of the program’s logic.

So how do we deal with those modified clients?

MD5Tool

The last tool in our arsenal is a mechanism that allows to run periodic checksum scans on files in player’s game directory. Unfortunately, the limits imposed on the frequency and size of scans were the major PITA. You can check at most 2048 bytes in a single scan, every 30 seconds. Just to scan a relatively small – 3330048 bytes – game executable you’d need a player to stay on the server for 12 hours³. There are also other game files that could use some scanning, for example asset packs, to eliminate alteration of textures (that could lead, for instance, to walls or grass transparency).

In order to overcome the limitations I first generated a sequence of the checksums on a vanilla game. Then I modified the game executable to alter the variable names I deemed the most important for us and generated another set of checksums. Final lists were then diffed and I came up with a final list of 15 offsets that could be scanned in 7 minutes:

;generated by andr (2014-10-10 18:41)
;iw3mp.exe v1.6

pb_sv_md5toolempty
pb_sv_md5toolfreq 30

pb_sv_md5tool a "" v iw3mp.exe SZ3330048 AT2738176 LEN2048 928C325DC37E804F3C77B6D2E46FF294
pb_sv_md5tool a "" v iw3mp.exe SZ3330048 AT2926592 LEN2048 B2C3332CBA37E374C867883AB179F68D
pb_sv_md5tool a "" v iw3mp.exe SZ3330048 AT2914304 LEN2048 4E833D47539C77F6E6597467745DE83E
pb_sv_md5tool a "" v iw3mp.exe SZ3330048 AT2959360 LEN2048 7B5555B4E8ECE6D9ECAFD4D93CF88B10
pb_sv_md5tool a "" v iw3mp.exe SZ3330048 AT2934784 LEN2048 70CD05CA9BE840A07B3E8024A2CBC5B8
pb_sv_md5tool a "" v iw3mp.exe SZ3330048 AT2947072 LEN2048 FC8D1E1A568BE2C639E03BDC7C179061
pb_sv_md5tool a "" v iw3mp.exe SZ3330048 AT2951168 LEN2048 F8D39DF25CB8ECF3752162A6BCCD6F97
pb_sv_md5tool a "" v iw3mp.exe SZ3330048 AT2953216 LEN2048 BE4B6F1A3E6CF5F395D7C703BA373D29
pb_sv_md5tool a "" v iw3mp.exe SZ3330048 AT2992128 LEN2048 CD114E133DB909032914F846A0D041BA
pb_sv_md5tool a "" v iw3mp.exe SZ3330048 AT3020800 LEN2048 ED047C9BFD0EE6068B3FA48E8BB17611
pb_sv_md5tool a "" v iw3mp.exe SZ3330048 AT3026944 LEN2048 5063B2EE918E3DF1CE90CF51830A43EF
pb_sv_md5tool a "" v iw3mp.exe SZ3330048 AT3061760 LEN2048 FD246F355BCD68D85126833A1D93F670
pb_sv_md5tool a "" v iw3mp.exe SZ3330048 AT3115008 LEN2048 90160CB306D7F13D2C8F08F272ABD2A9
pb_sv_md5tool a "" v iw3mp.exe SZ3330048 AT3168256 LEN2048 34D42E2241D52143CE8707CA9A95E0DE
pb_sv_md5tool a "" v iw3mp.exe SZ3330048 AT3162112 LEN2048 FEE4B1D001F2DFEA1C3A0C7F394D0E26

An excerpt from logs that indicates MD5Tool violations:

[10.11.2014 16:41:47] VIOLATION (MD5TOOL) #9002: cg_lan 1 (slot #1) MD5Tool Mismatch: iw3mp.exe (len=2048) [2b9ab1fbc7cdc8ee11a2a26249b4413e(VALID) 195.242.125.179:28960]
[10.11.2014 20:50:00] VIOLATION (MD5TOOL) #9002: BASEDGOD (slot #4) MD5Tool Mismatch: iw3mp.exe (len=2048) [6cfb244345650db8b2837bf151ff8742(VALID) 107.209.190.143:28965]
[07.05.2014 15:18:21] VIOLATION (MD5TOOL) #9002: [ms] (slot #1) MD5Tool Mismatch: main/iw_13.iwd (len=2048) [af3d2e82f247c3be99a17f9ffc873c30(VALID) 83.85.114.231:28960]
[07.05.2014 15:21:24] VIOLATION (MD5TOOL) #9002: [ms] (slot #1) MD5Tool Mismatch: main/iw_05.iwd (len=2048) [af3d2e82f247c3be99a17f9ffc873c30(VALID) 83.85.114.231:28960]

• Total MD5Tool violations: 294
• Unique MD5Tool violators: 49

Top offenders:

     26 1f97d04c79e3cc150232cb8d196843bb
     22 e4afe5a995f9701a6e29dace7d109348
     15 68a1304d02b53d6a62003dc7337639c4
     15 26956b1198b880bd6001eece83dea4a0
     14 9126e989c56bfbc92307082716646ab8

Unfortunately, as with countless other anti-cheat measures, this one is particuarly easy to overcome. The message that is shown in logs was also present in a message on the automatically issued “kick” when the violation was detected. As you probably figured out by now, running the game from foo.exe, while keeping unmodified iw3mp.exe at the same time, would let you fly under the radar.

Closing Words

It was a fun couple of years I spent doing work that’s barely visible but much needed by the community. However, I still don’t get it why would anyone want to cheat in a game. What’s to gain?

Dealing with those people, especially after they’ve been caught red-handed, was draining a lot of energy. Constant “ban appeals”⁴ and comebacks with new game copies and names, just to do same shit over and over again…

Downloading 70GB would take roughly 3 days at 300KB/s. I knew I could do better than that. But I didn’t want to circumvent any mechanisms they put in place to protect their infrastructure from people like me.

I was sceptic at first but I started my qBittorrent and loaded the torrent from the website. To my surprise, there were two HTTP mirrors instead of one. That’s twice the original speed already! There were no other peers at the time.

My theory was that everything would speed up substantially if more people were to join because the HTTP mirrors are artificially slowed down. So I invited some friends to the experiment. After an hour there were 7 of us downloading different parts at “full” speed from the Archive mirrors and sharing with the rest.

Some time later one of the upstream hosts went down but I managed to download the whole thing in under 8 hours. One of the friends keeps seeding it on his server and reached the ratio of 9.7 over last month. So there are other people interested in this package after all.

Shout-outs to aquaz, dexterw, Halamix2, Kaadzik, pitpo and PokerFacowaty for help.

I wanted to share some markdown (.md) files but my configuration forced download (or at least didn’t request in-lining) even though it’s just a text file that could be rendered in the browser. I learned that this behavior depends on Content-Disposition HTTP header. Apache has a module mod_headers that allows setting arbitrary headers, along with FilesMatch I came up with the following solution:

<IfModule mod_headers.c>
  <FilesMatch ".+\.md$">
    Header set Content-Disposition inline
    Header set Content-Type text/plain
  </FilesMatch>
</IfModule>

Today I wanted to add a static files hosting to a VirtualHost that was serving as a reverse proxy until now. I knew about Alias directive but it didn’t work out of the box — the requests for aliased path were still proxied to an application that didn’t know how to handle them. It turned out that the priority of ProxyPass is higher than Alias so I had to explicitly write an exclusion for the aliased directory like so:

(...)
ProxyPass /static !
Alias /static /path/to/my/static
ProxyPass / http://127.0.0.1:1337/
ProxyPassReverse / http://127.0.0.1:1337/
(...)

Long story short — we wanted to test whether Linux was a problem or not¹ so we installed Windows 10. Installation took couple minutes, new system booted from the NVMe drive, so far so good. There was one little issue, the network didn’t work. The on-board LAN (some popular Intel chip) was detected but it was unable to negotiate any link. Some generic Microsoft-provided driver was loaded…

It’s 2022 and the system is dysfunctional after the fresh installation. What if someone didn’t have a second computer to fetch and transfer the driver via the USB stick? Less and less PC cases include 5.25″ bays² so the provided driver CDs are only a piling trash.

Second story is also going to be about a network adapter driver but on my private computer. Recently I changed my ISP to the one that offered a FTTH so I could upgrade from the old unstable 150/15 Mbps cable³. The technician did his thing in 1.5h and left. It was around 11:30 so I was still working but I “risked” switching to the new connection. The speeds were oscillating around the expected values (943/301 Mbps averaged over three speedtests) on Linux. After work I switched to Windows to, you know, maintain the mythical work-life balance⁴.

First thing I did was another speedtest, just in case. I observed the pointer of the speedometer going rapidly up. 200, 300, 500, 700, 800. BAM. Both my screens went black for a second. Huh. Strange. Can’t be related to the new internet, can it? I fired up another speedtest. 300, 700, 800. BAM. Black screens again but also recovered quickly. When I checked the Event Viewer, I found out that the black screens were related to the Nvidia driver crash.

I started yet another speedtest but this time I also opened Task Manager to take a look at resource usage. To my surprise both CPU and GPU were going crazy high on some system daemon when the test was accelerating. I didn’t have high hopes but I tried to update my Nvidia driver… and the system crashed while downloading it. This time the screens didn’t come back without a reboot. Then it suddenly dawned on me that it must be related to the crappy MS-provided drivers for the network adapter. I manually overrode the link speed to 100 Mbps and downloaded a driver from the mobo manufacturer.

Et voilà, all my problems solved. This time the generic driver “worked” and I didn’t realize that it was not offloading to the chip for almost a year thanks to my old internet connection. I’d probably found out about it sooner if I had less powerful components in my PC. But hey, at least now we know that a 10th gen i9 and an RTX2070S are not capable of calculating checksums at gigabit speeds using the generic driver. This time I wished it didn’t work at all.

Third story is going to revolve around a HDD. In January I bought a new 8TB HDD⁵, formatted it under Windows to use NTFS and Bitlocker so I could use it from Linux as well. The last time I checked, Windows still didn’t support ext4 and LUKS out of the box :(. Everything looked good so far. This unfortunately changed when I started to copy files onto the new disk. Write speeds were never exceeding ~30 MBps flat on Windows with huge files. I couldn’t resist and checked how it behaves on Linux. Despite a weird filesystem and Bitlocker, the drive could maintain a stable 110+ MBps writes there.

I did some googling but nothing useful or even remotely connected came up so I gave up. It was just Windows doing windowsy things all over again. The experiment showed that the drive clearly wasn’t an issue here. I could live with those speeds because it’s going to be used only for archiving purpose. However, this weekend while I browsed the Device Manager I found out that the SATA controller was listed as Standard SATA AHCI Controller and used the MS-provided driver… So, I hope you know the drill by now.

The thing that pains me the most is that it’s 2022 already and Microsoft still doesn’t ship proper drivers for common hardware. And that their automatic updates fail to update the drivers to match the devices properly. Ok, ok. Most people probably wouldn’t notice the 3rd issue I presented but it’s still there.

To complete the game in 100% one has to finish the following side missions²: Paramedic (reach level 12 in one go), Vigilante (kill 20 criminals on each island), Firefighter (extinguish 20 fires on each island). They have one thing in common – all require player to spend a substantial amount of time in a vehicle without normal radio stations.

This summer my friends from the Polish speedrunning community are going to play the entire GTA 3D trilogy 100% in a relay for a third time. Previous attempts were completed in 55h in 2020 and 49h in 2021. One of them found this plugin by ThirteenAG that enables radio stations in all vehicles but unfortunately it doesn’t work with the Steam version of the game. It also has a (later discovered to be deliberate) side effect of allowing all vehicles to use the Pay ’n’ Spray service (it fixes the car and cancels police wanted level) which changes the core gameplay. That last thing motivated me to figure out how the plugin worked in order to create a version that: 1. will work with Steam version; 2. will not make the game easier.

For some reason, the original plugin is no longer available on the GTAGarage page. Luckily, it has been indexed and archived in 2015 by the Wayback Machine so I was able to retrieve it for analysis. It turned out to be a regular DLL file with extension changed to .asi. GTA III loads .asi files placed inside mss directory in the installation directory on startup – that’s great beacause no extra injectors are needed to make mods work.

After a quick analysis it appeared that all the plugin did was:

• create a thread with delayed execution (never figured out why)
• detect the version of the game by checking some magic values in memory
• set other magic values in memory depending on the detected verison. Cool, now I could figure out which offsets in memory did what.

Cheat Engine values table when running a Steam version of the game.

Once I realized which branch should I take for each version of the game I started to play with the values in Cheat Engine debugger. That’s when it turned out that the aforementioned Pay ’n’ Spray “issue” was an intended feature! At this point I could just recreate the mod (or patch the current one) without this “feature” but I wanted to fully understand why it worked. Remember the magic values the mod was setting? I’m not proficient at reverse engineering nor x86 assembly but thanks to pitpo I managed to figure out that the 0xEB in place of 0x77 was in fact turning a conditional jump into an unconditional jump.

I was sceptical about loading the game binary into Ghidra but I eventually did it only to learn that I could easily map the related code to re3. Couple years ago a group of fans made an effort to rewrite the original game in order to fix bugs and port the game to new platforms. It was called re3. I wrote “was” because the project has received a takedown from the publisher of GTA, which was later countered, and then Take-Two took it to court… Thanks to re3 I could confirm that the change I’m making won’t affect any other aspect of the game.

re3 on the left, ghidra output on the right. Spot 10 differences. Whole car list that includes IDs is available here.

Now the only thing left was to somehow make it work with a binary shipped by Steam. That version is supposedly corresponding to the retail version 1.1 however there’s also a layer of DRM sprinkled on top. Once again pitpo’s help was invaluable as he pointed me in directrion of Steamless. That tool can take out some variants of the SteamStub DRM. To my surprise, it just worked. With the unpacked binary, I was able to trace the required code which turned out to be only slightly offsetted from the 1.1 version.

The final outcome of this journey is available on my GitHub. The pun from the post’s title is now apparent ;)

2014 GTA 100% Trilogy Race layout. Source: @AdamKuczynski

I wanted to pay homage to them so I did my best in the little time I had before the stream.

My layout for GTA IV Blind Race)

I was satisfied with the result but something was missing. It lacked the dynamism and after a moment of studying you already knew everything what it had to offer. So after the race I thought I could make it more unpredictable by increasing the number of images. And showing them in random order. And adding random entry animations.

Example run of the generator.

It took some time and money to get all the assets but the final effect was definitely worth it. I wrote everything in pure javascript embedded on a static website so I could easily use it as a background in my OBS via Browser Source. Some of the APIs for drawing work only in Chromium based browsers at the time of writing this.

You can find project’s repository on my github account. The live demo is available on my website (the first load will be slow because all the images have to be downloaded).

Screenshot taken at 800x600. Scene rendering doesn’t work properly because I was running the game with Wine on Linux but the UI behaves just like it does on Windows.

I first had a look at the files of the game. It turned out that it stores graphics as DDS and PNG files directly in the game tree without extra intermediate formats or archives. Just for the sake of it, I upscaled all assets by a factor of 2. As you might have already guessed, it didn’t work. The game still worked but the UI was the same size as before but only the top-left quarter of each asset was used. Radar didn’t work and all the text was off; the gauge on the speedometer was broken.

That’s actually something I anticipated from the very beginning because there ought to be some logic to drawing. I found references in two types of files: .mis and .dso. .mis files are just “mission”, i.e. level descriptions. Each map in game had a hardcoded parameters for a displayed minimap. Luckily it was enough to just sed these files with new, twice as big, values. The only side-effect of the change was a need to generate new shader cache for each level when loading it for the first time but the game takes care of it by itself.

.dso files, on the other hand, are a compiled game script files that maintain some internal structures and offsets. The same trick as before didn’t work here and the game crashed when the size of texture changed from 3 to 4 digits. After doing some research, I found a source code of the newer version of the engine that I could use to write a basic version of parser that was compatible with the game because the basic concepts were still there. The remaining opcodes and structures were easy to figure out from the trial and error using a hex editor. Equipped with the parser, I could update all the strings¹ in the readable form. The last step was to “compile” the scripts back to dso files.

Screenshot taken at 1920x1080 before and after upscaling the UI by a factor of 2. You can only imagine how ridiculously small it is in 4K. Minimap and speedometer are still functional.

After some debugging sessions I had everything I needed to create a PoC of an upscaler for the game. The library I wrote for parsing and patching dso files and the PoC itself (in examples dir) are available on my github.

FUSE is an interface for unprivileged programs running in userpace to implement virtual filesystems. Examples of popular and maintained filesystems built on top of FUSE include: sshfs (for mounting remote filesystems over SSH), ntfs-3g (for mounting NTFS partitions), s3fs (for mounting Amazon AWS S3 buckets).

After an evening of prototyping I had a first “working” version of SlackFS, a virtual filesystem that allowed opening and uploading files from a Slack workspace. The code is available on my GitHub. It uses Python FUSE bindings to implement the minimum required set of operations for the whole thing to at least not crash programs that try to access the files.

There are many limitations I’m aware of listed in the README. For example, files from channels are fetched only once, on mount. Files, however, are fetched lazily when they are accessed for the first time and cached for later use. Uploading new works but it’s not intuitive because I need to change the names of the uploaded files because Slack allows mutliple files with the same name to be sent to a channel. There are couple problems related to the APIs that I didn’t bother resolving because the current version is “good enough” for a demo and my use case.

To sum up, it was a fun couple evenings. I ended up with a “polished prototype”. Later I gave a short talk at work about FUSE where we live-coded a driver for accessing employees’ leaves database. It served two purposes, first one – to show FUSE, second – to show a problem with authorization in the corpo’s in-house solution for managing leaves. Yep, it was possible to access ALL the details (like reasons and types of leaves) for ALL employees in Poland¹.

At work I have to track and request matlab licenses for my coworkers in the meta-team of over 100 people that changes quite dynamically lately. However, since it’s a big corpo, there are already some LDAP groups implemented at the company. All I had to automate was to fetch and compare the changes in the top-level group that contained all the teams.

comm made this super easy once I had the previous and current lists. Using the -13 and -23 options I could generate lists of people to add and remove. The data, however, had to be mailed to managers to get all the necessary approvals and blessings… But that was a trivial task since the mail body could be composed using a HEREDOC and piped directly to mail because the server where the cronjob was installed had a proper mail configuration.

Mathworks, the company behind Matlab, provides a decentralized on-prem licensing services to track and authorize use of their products. Surprise, surprise, it’s called Network License Manger. The only validation it performs is through the shared licensing key and a username taken from the OS running Matlab. While the corpo-managed Windows don’t allow employees to change their usernames, Linux users can use whatever username they want. Here’s the issue: people who decided to use a custom, not-company-assigned username on their computers couldn’t use Matlab.

There were couple options to trick Matlab to start. Starting from the trivial “just use the blessed username”, through suing into another user just to execute Matlab, modifying network traffic on the fly, and ending on creating a wrapper that faked the username. We aimed for the last one.

After some straceing it turned out that Matlab first fetches the uid of the user that started it and later performs a lookup in /etc/passwd to get the username. emdej suggested that we use bubblewrap to fake the username associated with current user. The wrapping script looked more or less like this:

1. fetch current uid
2. take the blessed-corpo-username from an env variable or a config file
3. use the two from above to create a copy of /etc/passwd with an altered entry
4. exec bwrap with a bind for a real /etc/passwd and rest of the filesystem as-is.

The lesson learned from this story is that the Network License Manager’s authentication sucks (or at least it is configured poorly at our company). Theoretically we could fake all usernames to a single one as well as the reported hostnames (the combo was used to deduplicate licenses taken from the pool – one user could spin as many instances of matlab on one host as they wanted¹).

However, there’s a neat tool that I saw my colleague emdej use. Meet fzf, a command-line fuzzy finder.

Well, it’s much more than the tag-line says. The fuzziness can be used in many contexts and integrated anywhere you want (for example for matching hosts from the ssh config). fzf comes with default bindings for recursively searching files and commands from history. The latter, together with:

HISTSIZE=
HISTFILESIZE=
export HISTCONTROL=ignoredups:erasedups
shopt -s histappend
export PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND$'\n'}history -a; history -c; history -r"

for making the history infinite and reloading live², makes my life much easier.

I found GNU Stow that fits my needs perfectly. It allows maintaining symbolic links in a natural matter, just like if you were installing a package. This way I can just update the repo with configs and there’s no need for me to manually keep track of any changes. The only time I have to do something more is when I add and want to use a config for a new program.

Stowing configs is simple:

$ stow some_dir/ other_dir/ yet_another_one/

By default it installs the files one directory above the current one, hence it’s easier to use when you keep the source config directories in a folder directly in your $HOME.