MoarDammit turned 8 this year in August and I thought it might be about time that I migrate and rewrite the backend for the site once again. This time however there was a real reason! The original MoarDammit in 2012 was a hand-rolled framework that very quickly showed growing pains for what I wanted to accomplish. I then looked around for some framework that would be sufficient and came across FuelPHP, which seems to have not had an update since then. I also attempted to build the site cakePHP, SailsJS, and AdonisJS. My previous update in 2017 goes over more history but it ended up that none of them really felt like a great choice at the time and I stumbled upon Laravel. Laravel has been the powerhouse behind MoarDammit for probably the last 5 or so years.

That changes today as the rewrite to NestJS is complete. The frontend client is still written in the ancient AngularJS (1.x) framework. In due time I expect to undertake the rewrite to Angular or potentially React and incorporate some new features but for now the same client was ported to the new build chain. What comes with these updates isn’t new features but better maintainability as all of my sites now run on some version of JS. No longer is it necessary to recall how to write PHP to fix any issue and I don’t need to figure out how to install the latest version of whatever PHP extension in order to install packages from composer. This alone will probably make it easier to take up occasional MoarDammit features in between inspiration to work on D20Kit and other products.

Not too long ago MoarDammit changed from being exclusively flash content to incorporate regular video media and youtube embedding. This has worked pretty well for the most part but considering a majority of the library is still in flash there will come a time which a migration may be necessary. I’ve tried to convert flash media to regular media before and it works for some but any dynamically driven swfs become corrupted or broken. After some research I came across Ruffle which shows some promise in being able to forever prolong the near decade old flash gallery. With this new migration and cost savings with moving to Azure have made it possible to expand the gallery without a huge increase in maintenance costs.

I wouldn’t take this post as any major flag that MoarDammit will be the priority but the learnings from the platform may apply to other ideas that are in the works. I’m a big fan of the versatility that NestJS has shown when building out multiple applications at this point. From building a large million-user scale authentication platform, realtime synchronous audio platform, web & discord hybrid client, and a server side rendered website (the static kind, not SPA). I don’t think there isn’t anything I couldn’t accomplish this framework and I’ve used a lot of frameworks in a variety of languages.

The Journey

Some may be wondering why I’ve migrated a dumb flash gallery for 8 years straight and my answer is simple, practice and experience. Sometimes when trying to expand ones own experiences it’s good to have a known goal. For example, if you don’t know how to do something but you know exactly what your end goal looks like you’re able to ask better questions. It also requires you to have patience and fully internalize that anything is possible and it is just a matter of breaking down the “how” to find the right questions to ask. So with this mindset you build the very first crude version that does nothing more than a glorified switch statement

if($_GET['c'] != ""){
	switch ($_GET['c']) {
		case 'summer':
			generalFunc::goToPage("./spongebob.swf");
			break;
		case 'datfresh':
			generalFunc::goToPage("./kewlmang.swf");
			break;
		case 'coolmang':
			generalFunc::goToPage("./smoothjazz.swf");
			break;
		case 'mom':
			generalFunc::goToPage("./spaghetti.swf");
			break;
		case 'chillassdog':
			generalFunc::goToPage("./dopedog.swf");
			break;
		case 'atwar':
			generalFunc::goToPage("./skeletor.swf");
			break;

		default:
			generalFunc::goToPage("/");
			break;
	}
}

Talk about a dated approach, to add a video caused a deployment! Uh, well this is before I ever considered running my own infrastructure. MoarDammit started with shared hosting on HostGator who are still around today. I originally left HostGator when they refused to actually update any of the vulnerabilities in my shared hosting instance. With much discontent I said “God dammit, fuck it! I’ll do it myself!”, and so it was done. Moving between providers trying to find VPS hosting and I eventually landed on Linode who was a great host for many years. At least I thought they were until their many issues with security arose along with a “max packet count” issue my server was having. With another great discontent I figured AWS was the only other place to go as they at least “cared” about security and not just security theater. Then it suddenly hit me how expensive cloud computing truly is and began the still to this day tradition of changing hosting providers to get the best deal. AWS became expensive for what I really wanted to do and I wanted to try the new hot tech on the market called Kubernetes of which D20Kit, MoarDammit, and even this blog run on.

While some say Kubernetes comes with operational overhead by having to deal with many more problems, I say yes. It’s absolutely true. Kubernetes can be incredibly complicated but once you have your configuration figured out you only have to keep up with the rapid development of everything in the ecosystem, something that should be “easy”. At least if you’re required for your employment as a infrastructure engineer. I don’t blame anyone for avoiding Kubernetes but I put all my eggs into containerization. It was at this point I attempted to run Kubernetes myself and through much pain was able to build a generic cluster builder managed by salt. It would automatically create all the primitives within AWS to build out the VPC, routing table, subnets, service discovery and then standup etcd. The etcd cluster size was determined by the target environment, this was so I could build smaller development clusters with lower reliability and availability targets. I treated Kubernetes as immutable and something that could be rebuilt and replaced. This ideology has made it possible for me experiment with just how far it can be taken. What does it mean to really be multi-cloud?

It was around this time I got around to actually founding a legal entity for my company and formalize actually operating as such. With this came opportunity at Google to join their startup program on Google Cloud and get some credits to build out a working copy of my idea. This required figuring out how to build MoarDammit, which originally was just a separate folder on a single VM, to be it’s own container deployed on Kubernets atop a set of reliable VMs. This proved to be extremely difficult because of it being PHP and still requiring a “server” inside the container to serve the content. Hooked it up to one of Googles load balancers and holy shit, this 3 node cluster costs $250/mo just for the cluster. Preemtibles to the rescue! Oh well, I can’t run mysql in here anymore… Continue the pain and I come across that Netflix calls Chaos Engineering. Something that I love in name and practice, no infrastructure should be so sacred that I can’t just restart it in the middle of whatever I’m doing. I should be comfortable with my infrastructure to know what will happen in any circumstance and that things will repair and heal themselves. I broke out of using Google’s load balancer because I didn’t want to pay the rental cost for an underutilized tool that has the audacity to charge me per rule. I stood up a traefik LB that served content and integrated with Kubernetes. Finally, I have a cloud-native like environment where I can define mostly everything within Kubernetes and the site automagically is available assuming the DNS rules are also configured.

The tangent into Kubernetes wasn’t for nothing, building things out to be exclusively defined by yaml files made it so I can migrate to Azure with relative easy and I was able to migrate everything including the data within a few days. With all the advancements I’ve made to my automated infrastructure made it trivial to standup a migration project and see nearly immediate results in production. The whole rewrite was probably accomplished within a weeks time. There are still edge cases that are not handled that the original application did. However they were not important enough to prevent moving the migrated project live. I’ll probably never make up the cost of time it took to build out a chaos-tolerant architecture but it was such an interesting problem to try solve. It makes keeping MoarDammit around a rounding error instead of a cost cutting measure. I look forward to explaining a new sidecar project named Concord that was spawned off of D20Kit in a future post.