Timothy Baldridge

Weekly Update - Upgrading the build server

The Wabbajack server-side code consists of two main parts, the CDN (essentially a FTP+HTTP server, with a 1Gig pipe) and a backend server that does list validation, Nexus caching, list healing and a host of other small tasks (but doesn't require a fat upload pipe). In addition, this same server also runs a  VM which handles our CI test runs from Github.

For the past year or so that backend server has run off a consumer Windows 10 box. Due to how it's built we have a limited amount...

Weekly #4 - Fixing the root drive bug

I just released version 2.4.3.2, it's a small release but fixes a rather big bug that was causing WJ to crash on startup if run from a Root drive. It ended up being related to a bug that's existed in the code for some time. We normalize paths in the Wabbajack libraries, so `c:\foo\` becomes `c:\foo`. This is fine, except when the path is `c:\` in which case we'd normalize it to `c:` and then try to write a log to `c:foo` instead of `c:\foo`. That path `c:foo` is invalid and that caused a cras...

Weekly Update - #3

It's time for a new update, and this time the post is also synchronized with a somewhat minor but foundational update.

I spent most of the past 7 days doing research into the best way to update our UI to be a bit more consistent. As it stands, parts of Wabbajack were developed about 1.5 years ago with the 1.0 release, then thanks to Noggog and Erri, my rather poor excuse for a UI was given a much needed face-lift. In the time since the original UI designs the code has gone through sev...

Weekly Update - #2

Little over a week, since the last update, but a lot's been going on.

Almost right after my last post we had the fantastic experience of having our CDN go down.  Out of the blue parts of the files we hosted on BunnyCDN decided to not sync across regions. I contacted support and while they eventually fixed the problem, the fact is we don't really need a full blown CDN, and I like the tighter control I can get from running a server on my own.

So I went and rented a VPS serv...

Long Overdue Update

Each week that goes by reminds me that I *still* haven't updated all our patreon supporters on the state of Wabbajack. I often sit down in the evening, think I should make a patreon post, then go do some coding and forget to actually write the article. At any rate, my plan now is to try an write a post once a week to update you all on the progress of the project.

Going back several months, life first got interesting around the new year with a massive outage from Lovers Lab, and a mont...

Looking forward to 2021 and looking back at 2020

So another year with WJ is in the books! Statistics are always fun to look at, so let's start with those, in the past 12 months, 48,879 users have completed installs 191,690 times. Over that time the most popular lists have been:

* Living Skyrim: with 22,550 installs

* Ultimate Skyrim: with 11,722 isntalls
* The Phoenix Flavor: with 10715 installs

* Total Skyrim Overhaul: with 8091 installs

While Skyrim still seems to be on the top of the downloads list, we have s...

Bugfixing and Stability

It's been way too long since I've given an update on Wabbajack. TL:DR of this post will be "things are fine, I've been fixing bugs". Kicking off the past two months was the wonderful experience of contracting <human malware>. Wasn't a bad case, but it's pretty brutal none-the-less. That removed almost all desire I had to code, so development took a hit in that time. Thankfully Wabbajack's errors and rough edges are mostly well known at this point, so the mods and admins on the WJ server...

Improving file extraction in Wabbajack

Over the past week I've spent quite a few hours working through our rather messy file extraction code. There's quite a few problems with file extraction in general, and finding the correct combinations of features and limitations has been quite hard. As an example, here's a list of some of the challenges with extracting mod files:

* Extracting a file requires both reading and writing data, this can be problematic for HDDs that prefer to read data in fairly large chunks in order to redu...

A bit of a victim of success

At the start of the month we bragged that on Aug 1st we had the most users of Wabbajack in a single day. At that time we would have never guessed that that record would be broken by 2-3x in the next week. Shortly after that post, PC Gamer ran an article about Wabbajack, and since then our user base exploded. That's fantastic, but it does come with a few drawbacks.

The modlists for Wabbajack are hosted on a CDN that's reliable, simple and yet also the cheapest in the business. But even...

This one weird trick can save you dozens of GB

I couldn't resist the troll title on this one. A few weeks ago a developer in our Discord (thanks Unnoen!) pointed me at a new feature in recent versions of Windows 10. It seems they finally came up with a mostly decent version of disk compression that works quite well. Disk compression has been in windows since the Win 98 days, but it's always been pretty crappy. The compression algorithm, was slow, and it had to re-compress files whenever they were modified.

This new compression syst...

Wabbajack 2.1.0.0 improvements

Yesterday I released Wabbajack 2.1.0.0 which includes several small, but fairly important improvements:

First of all, Wabbajack now use game files as a "download source" both during compilation and installation. This seems strange at first, until you consider how something like cleaned game ESMs could be implemented. In the past Wabbajack had specialized code for recognizing game ESMs and then would activate specialized code to patch these files. With this new release this code is much...

20,000 users served

This is just a quick update post to say thank you to all our patrons, and supporters in the modding community. This past weekend we surpassed 20,000 users on our discord! In addition I have a few more fun statistics:

Since January 1st, 2020, We've seen 43,903 completed installs of modlists using Wabbajack. If we estimate about 10 hours to install or update a list by hand, and an average of 40GB of downloads per lists (that's extremely conservative, but some of these were updates of exi...

Refactorings and Rewrites

It's been a busy past few weeks, and I haven't had time to update y'all on what's been going on, but I finally have some breathing room.

2.0 was released recently and we've already had several point releases after that fixing bugs and the like. More recently, a new developer has started providing UI updates (thanks  Khamûl), and their work can be seen in the new filters and infos in the modlist gallery. Erri has been hard at work on the website and has made several improvement...

Wabbajack 2.0 in the works

In the aftermath of the Wabbajack 1.0 launch I took some time off for a bit of a rest. Coming back to the project a week or so later, I realized there was a lot of cruft in the project I would like to fix. Issues that didn't impact the user experience, but resulted in confusing code. So I'm about two weeks into a major refactoring of the Wabbajack code. This rewrite accomplishes a few things:

Firstly, the code currently in use abuses strings quite a bit, we'll have code that looks like...

Wabbajack 1.0 is now released

After 7 months of heavy development Wabbajack 1.0 is now live. Thank you so much to our moderators, mod list authors, mod authors, discord members and supporters for making this effort possible.

Unlike many "1.0" releases, this one is considered feature complete. We've spent the last two months testing, refining, and improving the app to the point where it contains every major feature I've wanted to see in the app since the start of development. So if you haven't yet, stop by 2020-02-29 19:36:36 +0000 UTC View Post

The Last Missing Downloaders

There's a saying in computer programming, that the last 10% of a project takes 90% of the work. At this point it certainly feels like we've been "almost ready to release 1.0" for a few months. The good news is that the last few bits of major programming effort are completed, and I'm ready to spend the next few weeks in bug-fixing mode.

Something that's bugged me in the past was how bad the user experience in Wabbajack was for non-Premium users. As you can see in the image above, that ...

And this week Wabbajack exploded...

That phrase could be seen as good or bad, and frankly it was a little of both this time around.

About two weeks ago Total released the Total Skyrim Overhaul, a Requiem based SSE list focused on role-playing and a bit more "hard core" mechanics set than our more casual lists. We saw a huge influx of users of Wabbajack due to this list and we hit around 100 unique finished installs per day since the list has been out, so that's fantastic. Then a few days ago Lexy updated her guide to sup...

Road to 1.0 - Generic Downloading Framework

Although the Nexus is the defacto-standard source for downloading mods, there are a surprising amount of mods that exist on other sites: Lovers Lab, ModDB, tesall.ru to name a few. Unfortunately many of these sites have custom login systems and captchas. Instead of attempting to circumvent these systems, we took a different approach with Wabbajack: we embedded a browser in the app.

As you probably saw in the latest tutorial video, we now ship Wabbajack with a copy of Cef, a stripped-d...

The Road to 1.0 - Curated Mod Lists

The end goal of Wabbajack has always been to install mod lists with as few user interactions as possible while still respecting the copyrights of the mod authors. One of the hiccups in this quick install process has always been the two steps of downloading Wabbajack, and then sending the user back to the internet to find a mod list to install. A simple solution to this two step process is to integrate a mod list gallery into Wabbajack. And as you can see from our latest video, we've done exac...

The Road to 1.0 - New Tutorial Video

Our previous tutorial video was woefully out of date, so I took some time today to record a new one. You can see several of the improvements we've made over the past few months. A new GUI, new download sites, and a curated gallery of mod lists. In the next few days I'll be writing a series of posts going over these features, in preparation for our official 1.0 release.

In the mean time, enjoy this video, and please give the latest betas a whirl!

Polishing Wabbajack

Things have been quiet on the news front lately, but that's not to say work isn't still continuing at a rapid pace on Wabbajack. At this point the app is feature complete, but we're spending the next few months in polishing and improving the rough edges of the application. During big bursts of development, most projects acquire what's known as "technical debt": those things the programmers intentionally left out or hacked around in order to get a working application out as quickly as possible...

Update on Content Management Permissions

As you may know, if you follow Skyrim modding drama at all, Arthmoor and I have been in a bit of a debate over the past few months over what actions Wabbajack should be allowed to perform during installation. Before I continue, I do want to make it clear that the debate is not about what *can* be done by someone on their own machine, but what actions should be enabled or *encouraged* by a tool. Wabbajack has the potential to broker some of the first interactions a gamer has with the modding c...

Update on status of the project and our roadmap

It's been exactly two weeks since I last posted an update, so it's about time I explain some recent developments.

First I want to start by addressing a bit of the drama happening in the Bethesda modding community, caused mostly by my actions creating Wabbajack. I'm happy with what I put on Reddit earlier today, so here's a quick re-post:

"My goal is to provide as painless of a installation process for users who want to experience a heavily modded game. It is not a go...

It's been an insane week!

About 4 days ago from today the Wabbajack community exploded. Mostly due to JuiceHead's video on Wabbajack ( https://www.youtube.com/watch?v=57WSI36bc4U ) we saw our discord numbers go from 200 users to over 2800. That was enough to have the news picked up by a few gaming websites, which prompted more than a few Reddit posts, and all that has added up to an insane few days.

Thankfully this hasn't stopp...

Almost ready to release 0.9

It's been awhile since I've updated this patreon, but things have been trucking along in the mean time. The changelog shows the complete changes ( https://github.com/halgari/wabbajack/blob/master/CHANGELOG.md ) but as you can see the issues getting fixed are becoming more and more trivial. In short we're pretty close to being able to cut a release.

At this point I think Wabbajack is pr...

0.7 Release

I just cut a release of Wabbajack 0.7. Mostly internal changes, but the biggest features are that Fallout New Vegas is now supported, and compilation times are much reduced. The new VFS code stores all hashes in a single location. This means that the initial hashing takes much longer, but once all the files have been analyzed, subsequent compilations of modlists will run much faster.

I plan on creating some video tutorials in the next few days. Then doing a bit of a press release to a...

Time for some performance tuning

So I got a rather large modlist installed and while testing the compilation of the modlist with Wabbajack, I realized we have a rather big performance problem.

This problem comes mostly from the way Wabbajack finds matches in the files in your MO2 and the download archives. It does this by creating SHA256 hashes of all the files. So far we only cached the hashes of data inside the archives, but we generate hashes for the rest of the files on-the-fly. This means if you have 50GB of ins...

And now we start Dogfooding

In software development there's belief that developers should regularly "eat your own dog food" also known as "Dogfooding". In short it means personally using the product you're building. New features are nice, but until you actually try to use the product, you're never quite sure if it's going to work in the wild.

So this weekend I sat down and started installing the YASH'ed guide ( https://w...

SSE Support is Feature Complete

With the latest release (v. 0.6), we are now feature complete for SSE. The latest release includes support for nested archives. This means not only can Wabbajack install files that come from archives inside archives, but BSA files are considered nested archives. This means the following situations are now supported:

Extracting BSAs - Wabbajack can detect when files in MO2 originally started as files packed in a BSA.

Packed BSAs - Situations where loose files are combined in...

3 releases in 4 days

Quick update on the progress of Wabbajack, after the initial release last Saturday we got a nice influx of users on the discord (https://discord.gg/SSk4aB) as well as a nice set of less-common install configurations: like users without 7zip, users on Windows 7 machines, etc. All this has meant that we found quite a few bugs and in the mean time implemented some better crash logs so that when things do go wrong it's easier to d...