CMake: The Good, The Bad, The Weird

2024-11-21T00:00:00+00:00

CMake: the Good, the Bad, the Weird</h1>
CMake is one of the most popular build systems in the C/++ ecosystem according to a few recent developer surveys (IsoCPP survey</a>, CLion Survey</a>).
At the same time, it's kind of an infamous one -- anytime I see it mentioned, I see a lot of negative feelings and opinions toward the language. I think it's important to acknowledge peoples feelings and where they are coming from and their frustrations -- I have certainly ranted to my friends about CMake when I've been stumped by it before!
But I also find the build system useful, so inspired by the Lua: The Good, the Bad, The Ugly</a> I'm going to try to articulate my thoughts on what's good about it, what's bad about it, and what's kind of just a weird quirk about it.
I also strongly recommend for any developers who are learning or struggling with CMake to purchase Professional CMake</a> -- I have found it very helpful in explaining things where most other resources haven't, and it is consistently updated with new major versions of CMake.

The Good</h2>

CMake is extremely backwards compatible, and via the policy</a> mechanism forward compatible as well.</li>
CMake has a ton of flexibility and builtin integration with a surprising number of systems and tools like Doxygen, Python, Package Config.</li>
Custom targets and actions allow for implementing more non-standard actions.</li>
Relatedly, the `cmake</code> binary` `itself</a> acts as a sort of very light busybox</a> tool, allowing custom targets to not rely on system-specific names of tools.</li>`
`Being a fully developed (if idiosyncratic) language is useful for code re-use: if creating a new target in you're project involves a very large or non-standard amount of boilerplate, you can use macros or functions for setup. LLVM does this for in-tree passes and it seems fairly pleasant to use.</li>`
`In general, I find that once you've done all of the complicated setup and the pages you make to the build system are adding new files or tweaking flags, I find that CMake feels much more concise than Makefiles (the only other build system I have major experience with). At the same time, I think CMake allows for much more complicated logic involving flags or target than Make does and it tends to be a lot more readable when done this way.</li>`
`CMake supports really solid utility for consuming targets and libraries, whether that is through a source package manager like VCPKG, vendored-in subprojects, git submodules, or if you have libraries installed on you're system with appropriate CMake scripts.</li>`
`CMake can fetch dependencies at configure time using FetchContent</code></a>.</li>`
`This gives CMake a huge network effect -- most C++ (but not as many C) libraries I'm aware of have CMake files to allow them to be integrated like this.</li>`
`CMake has a very rich set of release management tools under the cpack</code> tool, allowing export to a large set of formats</a>.</li>`
`CMake also has a test runner utility system under ctest</code>. I haven't used it, but it looks like it could be quite useful, and an associated dashboard under cdash</code>.</li>`
`The CMake binary can actually be used to launch a build itself, through cmake --build <build> -j <num-threads></code>. This is very convenient and how I almost always invoke CMake builds.</li>`
`On Windows, CMake will attempt to locate a MSVC install and use it even if you are not inside the Visual Studio developer environment, which smooths out testing out builds on Windows a lot in my experience.</li>`
`compile-commands.json</code> can be output by CMake, which gives IDEs and other tooling introspection into how a build works.</li> </ul>`The Weird</h2> In general, CMake is a "have the newest version possible installed on your system" type tool, I almost always install it through official release instead of a package manager.</li> CMake shares some influence (but no code pedigree) with TCL based on some comments by developers</a>. This mostly manifests in the stringly typed nature of the value system: values can be strings, numbers, and/or list of strings depending on the context.</li> The other influence is that every builtin CMake command (and possibly user functions) receives it's arguments as strings after variable expansion; this effectively means every command is it's own DSL, which is why some special syntactic forms, like LINKER:</code> in target_link_options()</code>. This is also why variable evaluation works differently in if()</code> conditions -- the attempted expansion of strings without any leading $</code> sigil happens after the arguments are passed to the function.</li> CMake functions and macros cannot return values directly, so they must pass variables names in the outer scope that the function will set and expand.</li> At the same time, generator expressions</a> entirely live inside their own expression oriented scope, and sometimes require you to write out stuff that doesn't feel natural for them to evaluate to what you want them to.</li> Generator Expressions also are only allowed (or expanded) inside some particular commands.</li> CMake has two types of variable storage: "normal" variables that exist while configuration is running, and "cache" variables that are stored in a flat file database called a cache -- declaration of cache variables do not change their value if the flat file exists and that variable is defined, but syntactically they are treated almost equivalently.</li> CMake policies and some actions are not evaluated until the end of the file when the built up dependency tree is evaluated (unless you're using a policy scope? maybe?) so sometimes code seems to be executing out of order with little warning.</li> I believe builds for the Apple ecosystem were added much later, and thus often times have custom behavior you have to be aware of when targeting them. I wouldn't be surprised if most build scripts I wrote didn't work with XCode!</li> Any change to configure code requires you to rerun the configure, and potentially delete the build directory and restart -- often meaning tweaking small details can require full builds.</li> </ul> The Bad</h2> CMake is very, very complicated and takes a lot of time to learn -- I spent almost a week just reading technical documentation trying to get a sense of how a lot of systems works, and I still have tons of blindspots.</li> It feels like CMake has a lot of necessary boiler plate and defensive code required, especially if you are starting a new project. I personally find this to be enough friction that experimental or prototype projects normally start out with Makefiles or a shell script instead.</li> In my experience, it's very easy to accidentally write a build script that works in one use case, but not in a very similar case. Often this is because of how the test of the underlying build system output is run and thus not detectable at configure time, but still quite difficult.</li> A lot of the online help about CMake is not super good, and often times code snippets I find on Stackoverflow are not useful or are counterproductive. I absolutely do not hold this against those development authors! Programming is hard, and I'm sure most have found the tips they offered useful, it's just that they often don't work correctly in all use cases.</li> There are lots of just little inconsistencies, where two features that can interact don't work like they seem to be, or sometimes a specific command cannot handle a relative path. These all tend to be small things, but I'm sure every developers whose lost an hour on them absolutely remembers them.</li> Generator Expressions feel like a later addition to the language that do not mesh very well with it, and writing CMake code that handles being built in a multi-config build requires a lot of changes to how code obviously feels like it should be done with in.</li> In general, writing a CMake project for a library involves an even higher level of ensuring quality and that things work. I think if you are developing a library with a large userbase it's worth the effort, but it absolutely is an effort.</li> The provenance of variables is sometimes hard to find: sometimes it's set inside global code in a way you can't change, or involves string editing the various build flag globals. This has bitten at least a few developers!</li> CMake historically utilized tons of global various, but with the introduction of the 3.X versions of the tool, a more OOP-like system of creating targets with add_executable()</code> et al and editing them with target_()</code> commands.</li> </ul> So, is CMake worth it?</h2> It depends on you and your project! I think Cmake has a ton of benefits, but a huge learning curve and plenty of issues. I think the friction the defensive boilerplate coding required for CMake is maybe not worth it for exploratory or prototype code, but at the same time I think the benefits for very large codebases are worth it. At the same time, if you find yourself doing just fine with Makefiles, or Meson, or SCons, or Bazel, or xmake -- keep doing what you're doing! Build systems are to help build a project, and if you're use case is being met then I don't think you need to switch over to a new build system for hazy network effects. It's totally possible that in the future, CMake becomes a legacy thing that has a similar reputation to autotools, but for now it is if anything growing in use and adoption. I also personally think build tools need more love: it's easy to ignore them because they are cost centers for developing code, but they are a vital part of the development, release and management lifecycle.

An idea for C++ Mixins

2024-11-16T00:00:00+00:00

A C++ Mixin System</h1>
I've had this idea in the back of my head for a while of pervasively using mixins to add code and logic for more high level concepts, although this gets somewhat close to Rust style traits (and C++ concepts). This has existed in the back of my head for a long time as a way to model a framework or standard library implementaiton while also providing it for user types.
I think the big asterick to all of this design is that my ideal framework would not look like standard C++ but like a slightly weirder Rust stdlib:

Errors are signified through a `Result<></code> or Option<></code> like class, and both can handle reference like payload -- instead of exceptions, you can abort threads or the entire program.</li>`
`All constructors that aren't default, copy or move are 'data constructors' (each parameter is simply initializing the equivalent field)</li>`
`Non-trivial object creation is all done via a create()</code> factory funtion (or make()</code>, or w.e.) that returns an Result</code> value.</li>`
`All non-trivial objects that can be moved or copied have a default constructor that initializes objects into a` `NOP</a> like state -- C++ sort of implicitly requires them for a lot of stuff, so I think owning this and making it more of a first class concept would be nice.</li>`
`Any non-trivial interaction with an object in a NOP state will trigger an assertion error.</li>`
`Most methods that can fail but you'd like to go for happy path (like clone()</code> or serialize()</code> below) will abort if an error occurs, and they will have associated cloneTry()</code> or serializeTry()</code> methods that return an Result</code> -- implicitly i'm imaging most inner plubming methods return these.</li>`
`Aside from function paramters, pointers are wrapped in "utility types" which are just newtypes that better tie in what their semantics are (e.g. pointer to unsized array) that could allow the 'default' expected behavior for moving them.</li> </ul> I'm writing these down because I don't ever expect to implement this but my dream can at least be expressed in prose :).`Mixins</a> are a way to add code to an object or class without going through the normal inheritance system. I think viewed through that lens what I'm doing specifically isn't, but it feels close enough to make it 'count'? My example: a Clone</code> trait</h2> Borrowing an idea from Rust</a> where objects which represent handles to resources (including memory resources) can't be deep copied without explicit usage. So the requiresments here are: A clone()</code> method that creates a new vesion of an object.</li> </ul> Bonuses: Ability to assert at compile time that a class implements this interface</li> A standalone clone()</code> function, more as a demonstration than anything.</li> </ul> I think the standard C++ interface method would look like this: class IClone { public: virtual ~IClone() = default; virtual auto clone() -> Clone = 0; }; class String: public IClone { String(): data_(nullptr){}; String(const char data): data_(data) {}; String(const String&) = delete; String(String&& other) { this->data_ = other.data_; other.data_ = nullptr; } static auto create(const char data) -> Result<String> { auto new_data = std::strdup(data); return Result<String>::makeOk(String(new_data)); }; virtual auto clone() -> String { return new String(std::strdup(this->data_)); }; protected: char data_; }; </code></pre> But this does opt us into having a vtable (which is fine, honestly?). My idea uses the CRTP pattern</a>. One other issue: child classes implementing auto clone() -> Clone</code> must return a pointer or reference, and not any other class -- this is because of C++'s rule on overloading functions, which only allow covariance in function return types if they are returning pointers (see this blog post</a>). template<typename T> class MClone { // This is the implicit "shape" of the // implementation function, it may not // actually be defined here auto impl_clone() -> std::optional<T&> {}; auto clone() -> T& { reutrn static_cast<T>(this)->impl_clone.value(); }; }; </code></pre> And then implmenetation may look like: class String: public MClone<String> { public: protected: // needed if we want to call methods protected auto impl_clone() -> std::optional<String&> {} { return make_optional(new String(this->data_)); } char data_; } </code></pre> Right now, the differences between these are fairly small, but the CRTP method gives you a few cool options: Since the code isn't instantiated until template expansion occurs, it almost acts as a lazily instantiated way of introspecting a class </li> Since your using a (implicitly always safe!) static downcast to the child class, you could make the implementation method virtual if you expect the class to be substyped -- but non-virtual (and thus avoid vtable overhead) if you don't! class SomeClass: public MClone<SomeClass> { // expected to be overloaded by child classes virtual auto impl_clone() -> std::optional<SomeClass> = 0; }; </code></pre> </li> You can something similar to Rust's associated types</a> by checking for a child class type: template<typename T> class MClone { // T must have a `RetType` auto clone_impl() -> T::RetType {...}; }; </code></pre> </li> If you're implementation requires any state, you can essentially add the utility added by virtual inheritance without necessarily adding the overhead! For example, an intrusive reference counting type: template<typename T> class MRefCounted { // required impl method to get refcount field auto impl_refcount() -> uint32_t& {...}; // how to release resource once we're finished auto impl_release() -> void {...}; auto increment() -> void { auto& val = static_downcast<T>(this)->impl_refcount(); val++; } auto decrement() -> void { auto& val = static_downcast<T>(this)->impl_refcount(); if (val == 1) { val--; static_downcast<T*>(this)->impl_release(); } val--; } }; </code></pre> </li> It explicitly annotates at the class level (which is like, the opposite of mixins normally): this means you could add the appropriate static_asserts that some type implements an interface. You could add this as an added trait: #include <type_traits> template<typename T> struct is_clone { static constexpr bool value = std::is_base_of<MClone<T>, T>; }; </code></pre> and as an example of using it: template<typename T> auto clone(const T& value) -> Result<T> { static_assert( is_clone<T>::value == true, "Type must implement Clone" ); return value.clone(); } </code></pre> </li> </ul> But what about implementing this for other classes?</h2> I don't have a good answer for this, but an option may be something like Abseil's FTADLE Extension Points</a>? Is this worth it?</h2> I don't know, probably not! A lot of this is already sort of done by the C++ standard library with concepts</a>, so I mostly just think it's Neat and something you could add to earlier C++ versions. It's also something I think can show off the utilities of what the CRTP pattern can do.

Thoughts on Logging

2024-11-09T00:00:00+00:00

Thoughts on Logging</h1>
I read two blog posts recently that made me think about blogging: A footnote in Hillel Wayne's Why Not Comment</a> and another blog post about continious improvement of codebases.
I've been thinking about logging for a second, and so alos had the thought of thinking about it.

Log levels may be too granular</h2>

A lot of logging systems (in addition to allow custom logging) usually have a pretty large number of builtin log levels. Python has 5</a>, Syslog has 8</a>, and I've seen others move up to over 10.

I feel like its hard to intuit what level you should or should be using with so many levels -- is an error a WARN</code> or an ERROR</code> or CRITICAL</code>?

I feel this also makes filtering out particular levels more difficult: aside from VERBOSE</code>, it's hard to feel out what level something should be at.

Here's my personal, totally off my head cutting down of levels and semantics


ERROR</code> or FAIL</code>: a core or important subsystem or operation failed, system likely should panic or shutdown.</li>
WARN</code>: a subsystem failed or operation failed but the system can keep working, however this suggests code should be reviewed.</li>
INFO</code>: the default/something is going on message. If something like a config file is not found but that's handleable or expected it should go here.</li>

DEBUG</code>: verbose logging information for when something is going on.</li>
</ul>
If individual topics can have different logging levels, then DEBUG</code> would be a good candidate for enabling for particular files or subsystems.</p>
If you're building a very large or complete system, maybe an extra CRITICAL</code> level means to suggest the entire system (like an OS?) is failing in a critical method.</p>

TODO</code> or DEV</code> Level</h2>
In my experience, a lot of developers writing logs are continioug their use as print debugging, to show information about the internals of the system. I think this is good! I do it too.</p>
But it also means that when working with more complete or deployed code, a lot of the logs that show up are things that were incrementally added to debug a particular part of code that was problematic or having issues, but may not be as meaningful if you're triaging a whole system to get an idea of what's going on.</p>
My half baked solution is to add a special log level: TODO</code> or DEV</code>. This level would either be the highet level or same level as the highest so it always appears.</p>
Suggested practice would be that any developer working on a new feature would add all logging statements as TODO</code> unless you're fairly confidant it should be shown up as a regular logging statmeent, and then during code review all TODO</code> logs are either converted to their appropriate levels or removed.</p>
Maybe a custom a linting tool could be added to check for this?</p>
On the other hand, rewriting code is more likely to add bugs, and those debugging log statements added while debuging code may be better to keep because the original developer is aware of them, and a rewrite may remove/get rid of implicit context the original person had?</p>

SUCCESS</code> Level</h2>

The Python library Loguru</a> has an extra SUCCESS</code> level. I thought this was a good idea to indicate some sizable sub-operation suceeded, especially in a response/request or command based system.</p>
I also think it would make a good set of terms for .e.g. a web server:</p>
INFO    recieved request for /some/random/endpoint
</span>SUCCESS served response for /some/random/endpoint
</span></code></pre>
which is nice as a pair for</p>
INFO    recieved request for /buggy/endpoint
</span>WARN   failed /buggy/endpoint -- invariant violation
</span></code></pre>
OUT</code> as a logging level</h2>
This one is more hazy and maybe bad, but I also had the idea: if we're building up this complicated logging system, maybe regular print()</code> printing to stdout could be an alias for logger_subsystem::log(OUT, "whatever message")</code>, and then you could do the same level of feature/controls as other code does.</p>
I'm really torn on this one, because normally logging is asumed to have some level of metadata or formatting, whereas standard output is generally more unstructured and can have essentially arbitrary code.</p>

The Alone Time

2024-04-13T00:00:00+00:00

Author book site</a>

I enjoyed the setup for this book, but found the plot twists and changes more confusing and hard to follow as it went on, as the dual perspectives of the characters during the time of the plane crash and the characters in modern day, dealing with media attention. I think given time I could appreciate more the sort of messy web of half truths and manipulations work out, but the actual conclusion is also confusing and seems to not have the consequences other parts of the story have that set it up, and some of the revelations feel kind of cheap, and weirdly sensationalist in the way the magic thing works.

I don't know if I'd recommend this book, but I definitely felt myself appreciating it at times.

Slayer Shock

2024-04-13T00:00:00+00:00

Slayer Shock</h1>
3/5
game website</a>
itch.io purchase link</a>
Slayer Shock is what the video game equivalent of competent fanfiction is for Buffy the Vampire Slayer, which it carefully avoids mentioning on any of it sites despite the clear influence. I'm not very familiar with Buffy or Angel aside from a kind of high level overview, but it leans on not only the general idea of the show having a teenage woman fighting vampires and other horror themed threats along with a supporting cast, but the structural elements of a TV show -- each micro-campaign is built around discovering and fighting a major threat while handling the vampire threat in multiple different neighborhoods, whose procedurally generated missions act as general biomes.
I bought it for more than I think most people would say the game is worth it, but I've wanted to try it for a while: I appreciated the developers previous game Eldritch</a> and generally heard good things about the structured campaign of thriller/stealth Neon Struct</a>, although Slayer Shock had a pretty muted if not negative critical reception, and I think I get it, it's a game with a very particular vision of what it does, and it does that with a focused budget. Game dev is hard and takes more time and effort longer than anyone whose not intimately familiar with, and I have read some of the developer logs, I think they pulled off what they wanted pretty well.
It also blends genre in a bit of a funky way: it is a "roguelite" first person action RPG with stealth mechanics. It's a roguelite in that you have persistence after failing, but this is tied in at a campaign level: failing a mission gives partial rewards while the threat level in neighborhoods increase, which means that you're kind of racing an external pressure. To finish a campaign, you have to gain enough currency/experience points to purchase research (which also takes a number of missions ran afterward) to eventually find the season Big Bad and defeat them before all of the neighborhoods are fully overrun. At the same time, missions give particular rewards, including new shopkeepers, currency, weapon or passive buffs. The result of all of this is a final fight against a big bad who occasionally appears in nightmares, echoing Buffy's Master arc.
So that's the macro level of play, the meso level of play is also kind of hard to suss out, but it's kind of a tactical stealth action game: the procedurally generated levels aren't really meant to be explored, instead you generally can/should prioritize moving quickly and/or stealthily to your objectives, and only supply chests that you stumble across near you are likely worth it. There's a sort of persistent stealth element, but generally being alerted only grabs nearby enemies and at times is either impossible or something should you be using strategically. Additionally, higher rank enemies drop currency, which with an upgrade is a healing source. Once I stopped trying to fully explore and instead did like, tactical stealth combat I made headway in the game.
And at the micro level, the game is kind of a first person hack and slash/shooter, but with fairly simple gunplay and non-conventional weapons that fit the whole vampire hunting thing, along with light stealth.
While the game tries to liven up the coffee shop hub area with random interactions (and sometimes bonuses from those interactions) from your allies/shopkeepers, it is a bit spartan and simple, but definitely within scope. In the lore of Buffy, the titular Slayers are chosen by Fate(?) and are more or less meant to sacrifice their lives and connection to community to act in that role, which Buffy buckles by trying to balance being a normal person integrated into society and a vampire hunter. This game has numerous bits of dialogue that try to match the shows theme that connection is a strength, but at the same time your unnamed Slayer only engages with her allies for strategic utility and doesn't really interact with her society at all except for this. I'm sure this isn't intentional and you should imagine that she has more normal parts of life, but it does create an odd undertone to the whole thing, because whereas your teammates have crushes/arguments/falling out and a general sense of being organic people especially when reacting to random events (like discovering that a local bar uses vampire blood in the drinks... twice) your character isn't.
I think the season/episode structure of the game is cool, and a really good fit that also creates a natural sense of rising tension and danger, and the ways chain together does seem like it could be neat. I finished one season, and I'll see how after playing a few others.
I think this game couldn't have been made during Buffy's actual showing even if technically possible (a lot of retro games pull on having a GPU for extra effects and general luxury of RAM that means you can have large areas all in view), but also because this sort of experimental structure with both failure and rising tension was fairly niche, and even though you can point to Dark Soul's huge mechanical influence, dark souls is also mostly ok with you standing still while you gain resources that supplement skills and don't have an explicit possible set of failures.
I also like the aesthetic details of passive buffs being represented as bracelets on your characters arm, and that you can customize the nail polish and your characters skin color. Since you never leave first person, this gives you a fairly focused set of self expressions in your hands. Since my Slayer alternated between navy and dark red nail polish and got silver or metal bracelets, I think she qualifies as she a Dark</a> or True</a> Winter.
I think I'd recommend this game if this review sounds interesting, and you are OK with something interesting that won't blow you away, or you are a Buffy the Vampire Slayer fan who is willing to try something that isn't interested in being more Buffy, but is instead interested in trying to emulate some of its thematic content and its formal structure into a different genre. Also if you are an immersive sim person and are ok with something a bit more off kilter, then it is an ImSim without any 451 callouts in it.

What Moves the Dead

2024-02-01T00:00:00+00:00

What Moves the Dead</h1>
Amazon Link</a>
Rating: 5/5
I bought this book because a bookstore's cafe had this cute "buy a coffee, buy a book for $5", and it was a good use of $5.
This book is explicitly an expansion of Poe's Fall of the house of Usher, but using a 19th century-flavored natural philosophy plot. If you've seen the cover, it does involve fungus.
The reviews and book stubs I saw talked up horror and description, but maybe since I've read so many similar plots I mostly felt this fun and nostalgic. It's a good book, took maybe a day or 2 to finish, and has some fun, well written characters and a neatly resolved plot. This is apparently has a sequel involving the main character, a sworn soldier (in the internal world building, a kind of distinct social role for women taking on the masculine-ish, formal role of a soldier or officer), and there a good character.
I don't have anything to say against it, it was fun, well worth what I spent on it, and I had a good time with it sticking in my head

Ashes: Afterglow

2023-09-11T00:00:00+00:00

Ashes Afterglow review</h1>
download link</a>
Rating: 4/5
Ashes: Afterglow is a sequel to Ashes: 2063</a>, and the pair form a strongly linked duology. Funnily enough for something I thought was being praised for something being praised for conforming (in it's own, GZDoom engine way) to modern high budget gaming expectations, the version of Ashes: 2063 I played was in fact a updated version that backported several gameplay additions from Afterglow. Unlike the original, Afterglow is a partially open world game, or maybe more so a spared down metroidvania of sorts, with the town levels acting as longer running quest hubs instead of quick small diversions from before. There is also now an upgrade system for weapons, something that was present in a bit more of a diergetic and understated way in the original, along with More : more weapons, more enemy types, larger maps, more complex quests and dialog states, ending slides, a racing level.
One thing that does truly set aside games using GzDoom from more modern games, to me, is the sheer scope of the levels -- these are huge, long, involved affairs in a way single player games might not lean into as much nowadays, and it all runs pretty reasonable on a slightly few years old laptop. The presence of the hub system and persistence also means that the areas connected by hubs (generally, the first act and the second act) almost act like one huge level, unlocking shortcuts, progression, and a few sidequests.
I find Ashes: Afterglow is a bit hard to talk about, because it's a stunningly impressive in talent, time and labour, and I do not like it's fundamental gameplay. I could not do anything on this scope, and it's as far as I know a work of love that has lasted years. It's slightly hard to guess how long playtimes are given their not quite correctly tracked, but I'd guess Afterglow was 15 hours for me and 2063 6 (there's an additional fan campaign distributed with them that'd I'd guess is 3 hours long). The game is strongly wed to making vast, visually impressive and varied levels, and it's doom style large numbers combat. I think the combat overstays it's welcome past a certain point, and opening the map to see that a level has 100+ enemies really killed a lot of the motivation for me. At around the 60% mark or so, I simply turned on immortality and stopped interacting with the combat past a certain point. I might have robbed myself of some of the interesting struggle, but I also felt like I'd more or less fully experienced the system, and mostly wanted to see the maps and plot.
A kind of fun detail in the plot is your character, mostly self interested and to a certain extent just following the clues ancient tech points him to, sort of cleaves paths of destruction. This game has ending slides, which are a really nice detail, and they actually make engaging with one area in the second act lead to a bad ending for the town, which I thought was unusually thoughtful for the game. The second town also has a factional quest split, where you can either side with the upper elite or the lower class workers, or mostly go by default with the upper elite and not interfere much. The player character also develops a sort of violent reputation as a one man army of cruel destruction in the background but none of the NPCs repeating this refers to your character, and the PC can sort of awkwardly cough and go along with it and mostly keep his head down. This obliviousness nicely ties into how much the character changes while being overall ignorant of the larger conflicts beyond the extent he's been pulled into them.
Both this and it's predecessor have always been solid about horror, and the haunted submarine level "Anomaly 110" and a section in the intermission levels do some incredible work with atmosphere, sound design and tension.
I hate leaving such a negative impression about the game, because I can only try and estimate how much sweat was poured into this, and seeing some stranger who has made very little telling them it sucks is awful. The maps are gorgeous and large, and there is plenty of content. If I was young or stuck in college without a budget, the sheer amount of content and scope these games have and the replayability of 2's quest choices would make them beloved goldmines. If you like doom style combat and don't mind the scope, I would extremely recommend these.

Ashes 2063 Review

2023-07-28T00:00:00+00:00

Ashes 2063</h1>
download link here</a>
Rating: 4/5
Ashes 2063 is described as a doom mod (and, as of the standalone release, no longer technically a mod reliant on the base game either), but really it's just a new game built on the foundation of the GZDoom source port. It's part of a duology with Ashes: Afterglow, and in the style of doomwads that game begins with the map being considered later in the game, and 2063 acts as a story prologue. I didn't play Afterglow, and while it's installed I'm not sure I will either.
Ashes feels like a big mashup of a number of different post-apocalyptic movie and game franchises in a game that uses doom-style monster density but tries to hue hard on being a modern, professional style game. Often being pretty on the nose on what it's taking inspiration from, be that fallout, mad max, escape from new york, Metro or Stalker (and probably more). Mechanically it feels much closer to a modern FPS game in how gunplay works with reloading and relatively low max ammo counts, but it still uses Doom style ai for most of it's non-human enemies, and still has enemy counts that feel really large, not quite leaning into horde shooter but close.
I think the game works best when your flowing in exploration and engaging in these extra-Doom mechanics, like managing radiation, some light inventory management, smashing up the environment for resources, and handling small combat encounters with larger set pieces you can get a stronger hand on through earlier exploration. At worst, I walk into another large, impressively designed level and I spend 10-20 minutes chewing through a large combat encounter, and I would regularly put the game down when moving between them.
There is also a few sets of town areas that restrict weapon usage. The overall structure of the game is fairly linear (one level to another, no going back) but the two town levels have this very quaint, simple TTRPG adventure feel -- there's a few sets of direct choices to access bonus content or gain certain abilities, but always in a mostly quite direct way. Likewise the story path that propels you is quite straight forward through the areas, and you are given monetary goals that your character is shown to be explicitly interested in. This is all wrapped in a small surrounding layer of role-playing fluff that has you acting out a (straightedge?) post apocalyptic scavenger ala later mad max. It's fun, but there's not much there aside from a set of straightforward and sign posted interactions.
One of the middle level, the subway/tunnel level, really shines above the rest because the set piece has good continuity with what happens before and after, and incorporates both all of the major added mechanics (light management, some radiation stuff, weapons), the conversation systems, and you have a route described to you that you try and follow through environmental clues, along with a few set of twists in it's finale that uses environmental hints to hype things up and suggest what happens. It also ends with tons of enemies spawning and waiting out a slowly descending elevator that actually moved back up, so I simply went and cleared out the rest of the level which was absolutely not the intention for this sort of set piece.
What really stands out about Ashes 2063 is effort. This is a game running on GZDoom, and while that probably has a much simpler asset pipeline than something like modern Unreal, the amount of work here is absolutely stunning, and you can tell in the more ambitious settings they were trying to aim for an equivalent to AAA-level Polish. It's deeply impressive, but I also kinda wonder if what the glowing reviews I heard is seeing something kinda weird, and kinda indie, going for high levels of polish and praising it.
Ultimately, I think I'd recommend the game if you are a doom person or like post-apocalyptic action games and can handle the jank.

Baby's second garbage collector

2023-07-10T00:00:00+00:00

Baby's second garbage collector.</h1>
This blog post is meant to an iteration on the tutorial and introduction to garbage collection implementation presented in this classic blog post</a>. We are still working with simple garbage collectors, but this time another one with a bit more complexity.
I thought this would be cool because the original code already has a tiny but functional mutator and a set of tests meant to stress it; this means we can compare both the performance and implementation of the copying collector as well as the normal collector.

Source code for this article</h2>

The source code can be found at This github repo</a>.

The code is also hosted locally on this site, and you can download it as well with:

wget</span></span> https://jennyjams.net/copygc/src.zip</span>
</span></code></pre>
Copying collectors</h2>
A copying collector</em> is a garbage collector design that ties itself intimately in with the backing allocation system: you have two logical (and here, physical) memory regions. One of these is set to be active</em> region, and all new memory is allocated from this region.</p>
When the garbage collection subsystem determines that enough memory has been used that garbage needs to be collected, then the active region is swapped to inactive and vice-versa, then the garbage collector determines the set of live objects and evacuates</em> these objects from the now-inactive region to the newly active region. Then, we take any pointers in the now moved object, and check if the pointers they pointed to in the old region have already forwarded. If so, use the forwarded pointer, if not, perform evacuation.</p>
If an object has already been moved, then the object in the inactive-region can be reused to contain a pointer pointing to it's evacuated counterpart; this helps preserve the topology of the object graph and fix all of the reference cycles code tends to create.</p>
Cheney's algorithm</h2>
The particular algorithm used here is Cheney's Algorithm</a>, and it comes to us from the venerable 1970s.</p>

Swap the active heap</li>
Walk the set of roots, and evacuate the pointer to the new heap.</li>
After evacuating all of the objects, start walking through all of the newly allocated objects and inspecting each of their fields. If the field is a pointer and points into the old region, check if it has a forwarding address. If it does, set the field to the forwarding address. If not evacuate it.</li>
</ol>
Or, in pseudocode terms:</p>

</span>def</span></span> collect_garbage</span></span></span>(</span></span>)</span></span>:</span></span>
</span>  swap_heap</span></span></span></span>(</span>)</span></span>
</span>  for</span> root</span> in</span></span> roots</span></span></span>:</span></span>
</span>    if</span> field</span></span> in</span> old_region</span></span>:</span></span>
</span>      root</span>.</span>pointer</span></span> =</span> evacuate</span></span></span></span>(</span>root</span>.</span>pointer</span></span>)</span></span>
</span>  
</span>  while</span> not</span> worklist</span></span>.</span></span>is_empty</span></span></span></span>(</span>)</span></span>:</span></span>
</span>    obj</span></span> =</span> worklist</span></span>.</span></span>pop</span></span></span></span>(</span>)</span></span>
</span>    for</span> field</span> in</span></span> obj</span></span>.</span></span>fields</span></span></span></span>(</span>)</span></span></span>:</span></span>
</span>      if</span> field</span>.</span>pointer</span></span> is</span> in</span> old_region</span></span>:</span></span>
</span>        field</span>.</span>pointer</span></span> =</span> evacuate</span></span></span></span>(</span>root</span>.</span>pointer</span></span>)</span></span>
</span>
</span>def</span></span> evacuate</span></span></span>(</span></span>ptr</span>)</span></span>:</span></span>
</span>  if</span> has_forward_address</span></span></span></span>(</span>ptr</span></span>)</span></span>:</span></span>
</span>    return</span> ptr</span>.</span>forward</span></span>
</span>  new_object</span></span> =</span> allocate_from_active</span></span></span></span>(</span>)</span></span>
</span>  copy</span></span></span></span>(</span>new_object</span></span>,</span> ptr</span></span>)</span></span>
</span>  ptr</span>.</span>forward</span></span> =</span> new_object</span></span>
</span>  worklist</span></span>.</span></span>append</span></span></span></span>(</span>new_object</span></span>)</span></span>
</span>
</span></code></pre>
However, I feel like a lot of GC stuff can feel extremely abstract, especially when you start dealing with the quasi-recursive stuff, so I'm laying it out in pictures instead.</p>
Cheney's Algorithm, this time in pictures.</h2>
</p>
First, our initial state shows how one of the heaps is full of memory, and the collector has decided to begin garbage collecting. The set of boxes on the left is the Root Set (represented as a stack with 3 active members), and the two boxes on the right are the 2 heap regions.</p>
</p>
We begin by walking through each of the roots. The blue color represents objects that have yet to be processed.</p>
</p>
The first root is pointing to an un-forwarded object in the from-heap, so we forward it to the to-heap and update the from-heap object to point to it (represented by the purple and curved, dotted line).</p>
</p>
Now that we've forward what the first root was pointing at, we set it to point at the evacuated object in the to-heap, and remove it from the set of objects we care about, rendering it again in brown.</p>
</p>
Next, we handle the second root. Because it is pointing to an object that has already been forwarded because we handled it previously, we simply set the root to point at the evacuated object in the to-heap and remove the 2nd root from my concern.</p>
</p>
Next, we handle the final root. This is pointing to an object that itself has a pointer, so first we evacuate it and set the from-heap object to forward to it. However, because we just copied it, the evacuated object is itself still pointing to a from-heap object.</p>
</p>
Next, we set the final root to point toward the evacuated pointer. We are now finished with all of the root set, and next must begin handling the objects in the to-heap to move any objects that were not directly pointed to from the roots.</p>
</p>
We handle the first object in the from-heap. Because it simply holds an integer, there is no need to do any more work and we move to the next object.</p>
</p>
The second item in the to-heap is an object that contains a pointer pointing towards an object in the from heap. We must evacuate and forward that object, and add it to the set of object's we care about.</p>
</p>
Now that the second item in the to-heap is pointing to a forwarded pointer, we can fix it's pointer to point to the evacuated object in the to-heap, and thus remove it from our concerns.</p>
</p>
Next, we handle the final object. Because it contains no pointers, we simply remove it from our concerns.</p>
</p>
Finally, we have no more items to work through in either the roots or the heap. We can then clear the old heap and blow all previous objects.</p>
Garbage collection is now Complete.</p>
The world smallest allocator</h2>
Unlike mark and sweep, a copying collector relies on it's allocator to free memory, and thus we cannot use the standard library malloc()</code> and free()</code> function -- instead of calling a function to free memory on every object we've determined to be non-live, we implicitly free it when we swap the heap because all of the data in our allocator gets reused for future allocation.</p>
Thus, we will need an allocator!</p>
The simplest possible allocator is a bump</em> allocator: just a large contiguous region of memory, and an offset. Whenever an allocate(size)</code> call is perform, it returns a pointer to the memory that is offset</code> bytes into the memory region. The offset is then incremented by this size.</p>
This allocator doesn't really have a coherent free()</code> operation -- even if you logically free memory earlier on in the allocator, the allocator itself can't reuse it. Thus, the only meaningful way to reset memory is to wipe all</em> allocations and reset the offset</code> to 0.</p>
This property makes this allocator-pattern genuinely useful for situations like web-servers or video games, where a request or individual frame might generate large amounts of data that is not meaningful useful to keep around afterwards; the clear()</code> operation would be much faster than attempting to deallocate each of the objects.</p>
</p>
This is a diagram showing a set of objects that will be populated by pointers to objects allocated by the bump allocator, and the bump allocator itself. The bump allocator consists of a offset (or pointer) into the backing allocation, along with a large chunk of memory itself.</p>
</p>
Allocation is performed -- the object's pointer is set to point at the memory chunk plus the offset. Then the offset is moved down to point to the next free area of memory.</p>
</p>
And again, with a different size.</p>
</p>
And again.</p>
</p>
Next, we call clear()</code> and reset the offset to point to the start of the memory chunk, effectively freeing the memory; on more security minded systems we might memset()</code> the allocated areas to zero to prevent possibly leaking any information. All of the previous pointers into the memory are now dangling pointers -- even though they are pointing into valid memory that is mapped, the pointers to that memory are likely of the wrong type or value, and will be pointing into the middle of another object.</p>
</p>
Next, we allocate memory again, and we begin allocating from the beginning of the memory chunk.</p>
Now, to the code</h2>
Next, I present the code for each of the major parts we change out, and attempt to highlight code where I'm rewriting old code.</p>
Fixin' up our unified object model</h2>
Since we're updating an existing codebase to swap the GC strategy out, we need to perform some code to change things.</p>
First, we update the object model for our VM to remove the intrusive link list pointer, and to add another case to our union for the forwarding pointer. This is one of the canonical benefits of copying over more traditional mark-sweep, mark-compact, and reference-counting schemes: we don't need any extra metadata inside of the header.</p>
However, we do need to utilize a forwarding pointer. You'd think that might mean we'd need the overhead of another pointer, but since an object being repurposes to point to a new object has no other meaningful significance, we can just reuse the space for one of it's fields</p>
typedef</span> struct</span> sObject {</span>
</span></span>  ObjectType type;</span>
</span></span>  //</span> remove ``struct sObject* next;``
</span></span></span></mark>  union</span> {</span>
</span></span></span>    /*</span> OBJ_INT */</span></span>
</span></span></span>    int</span> value;</span>
</span></span></span>
</span></span></span>    /*</span> OBJ_PAIR */</span></span>
</span></span></span>    struct</span> {</span>
</span></span></span></span>      struct</span> sObject</span>*</span> head;</span>
</span></span></span></span>      struct</span> sObject</span>*</span> tail;</span>
</span></span></span></span>    }</span></span>;</span>
</span></span></span>    struct</span> sObject</span>*</span> forward;</span>
</span></span></span></mark>  }</span></span>;</span>
</span></span>}</span></span> Object</span>;</span>
</span></code></pre>
We also need a new tag to indicate an object is forwarded.</p>
typedef</span> enum</span> {</span>
</span></span>  OBJ_INT,</span>
</span></span>  OBJ_PAIR,</span>
</span></span>  OBJ_FRWD
</span></span></span></mark>}</span></span> ObjectType</span>;</span>
</span>
</span></code></pre>
Next, we update the VM model to store the code for the beginning and end of the heap work-list</p>
typedef</span> struct</span> {</span>
</span></span>  Object*</span> stack[</span>STACK_MAX]</span></span>;</span>
</span></span>  int</span> stackSize;</span>
</span></span>
</span></span>  /*</span> First and last object in heap */</span></span>
</span></span>  Object*</span> firstObject;</span>
</span></span>  Object*</span> lastObject;</span>
</span></span></mark>
</span></span>  /*</span> The total number of currently allocated objects. */</span></span>
</span></span>  int</span> numObjects;</span>
</span></span>
</span></span>  /*</span> The number of objects required to trigger a GC. */</span></span>
</span></span>  int</span> maxObjects;</span>
</span></span>
</span></span>}</span></span> VM</span>;</span>
</span></code></pre>
The world's smallest allocator, twice</h2>
In our case, we actually need 2 bump-allocators. Because only one one is every being allocated from, we can keep around a single offset, and also keep an extra bit of state describing which pool is being used. The boring names would be Region 1 and 2, or Regions A and B; I want to be cute and name them Regions Boba and Kiki instead</a> instead. Because I cannot spell and cannot be asked to double check things, all of the code uses Boba instead of Bouba. Consider this your daily reminder to go buy Boba.</p>
First, we need a reasonable max size -- I simply choose 2^16:</p>
#define</span></span> HEAP_MAX</span></span> 65536</span></span>
</span></code></pre>
Next, the data structure. In following the language in Cheney's algorithm and other papers, we call this a heap</em>.</p>
typedef</span> struct</span> {</span>
</span></span>  size_t</span> bump_offset;</span>
</span></span>  Region region;</span>
</span></span>  unsigned</span> char</span> region_boba[</span>HEAP_MAX]</span></span>;</span>
</span></span>  unsigned</span> char</span> region_kiki[</span>HEAP_MAX]</span></span>;</span>
</span></span>}</span></span> Heap</span>;</span>
</span></code></pre>
First, we need some code to generate a new Heap object. We memset()</code> the memory to zero to ensure no previous data remains.</p>
//</span> Allocates and creates new Heap
</span></span>Heap*</span> newHeap</span></span>(</span></span></span>)</span></span></span> </span>{</span></span></span>
</span></span></span>  Heap*</span> heap =</span> malloc</span>(</span></span></span>sizeof</span>(</span></span>Heap</span>)</span></span></span></span>)</span></span></span>;</span>
</span></span></span>  memset</span>(</span></span></span>heap-></span>region_boba,</span> 0</span>,</span> sizeof</span> heap-></span>region_boba</span></span>)</span></span></span>;</span>
</span></span></span>  memset</span>(</span></span></span>heap-></span>region_kiki,</span> 0</span>,</span> sizeof</span> heap-></span>region_kiki</span></span>)</span></span></span>;</span>
</span></span></span>  heap-></span>region =</span> RGN_BOBA;</span>
</span></span></span>  heap-></span>bump_offset =</span> 0</span>;</span>
</span></span></span>  return</span> heap;</span>
</span></span></span></span></span>}</span></span></span>
</span></code></pre>
It seem a little strange that, after writing a custom allocator, we would just malloc()</code> to get the memory for the allocator, but this is pretty common -- unless you are yourself implementing malloc()</code> or the lowest-level memory allocator for a system, normally you have to rely on some</em> function call to get you backing memory. And malloc()</code> is conveniently right here!</p>
Next, allocation!</p>
void</span>*</span> heapAlloc</span></span>(</span></span></span>Heap*</span> heap</span>,</span> size_t</span> size</span>)</span></span></span> </span>{</span></span></span>
</span></span></span>  assert</span>(</span></span></span>heap-></span>bump_offset +</span> size <</span> HEAP_MAX,</span> 
</span></span></span></span></span>      "</span>Attempted to allocate more items that can be in heap"</span></span></span></span>)</span></span></span>;</span>
</span></span></span>  void</span>*</span> pointer =</span> NULL</span>;</span>
</span></span></span>
</span></span></span>  if</span> (</span>heap-></span>region ==</span> RGN_BOBA)</span></span> {</span>
</span></span></span></span>    pointer =</span> heap-></span>region_boba +</span> heap-></span>bump_offset;</span>
</span></span></span></span>  }</span></span>
</span></span></span>  else</span> {</span>
</span></span></span></span>    pointer =</span> heap-></span>region_kiki +</span> heap-></span>bump_offset;</span>
</span></span></span></span>  }</span></span>
</span></span></span>  heap-></span>bump_offset +=</span> size;</span>
</span></span></span>  return</span> pointer;</span>
</span></span></span></span></span>}</span></span></span>
</span></code></pre>
We can't utilize this allocator unless we have a way of clearing out the dump allocator. Instead of the clear()</code> interface discussed above, we swap the active heap over, and then reset the offset to 0. This also updates some data in the VM to allow for our later step of pointer fixup, so we pass a pointer to the VM in rather than to the Heap.</p>
void</span> swapHeap</span></span>(</span></span></span>VM*</span> vm</span>)</span></span></span> </span>{</span></span></span>
</span></span></span>  if</span> (</span>vm-></span>heap-></span>region ==</span> RGN_BOBA)</span></span> {</span>
</span></span></span></span>    vm-></span>heap-></span>region =</span> RGN_KIKI;</span>
</span></span></span></span>    vm-></span>firstObject =</span> vm-></span>lastObject =</span> (</span>Object*</span>)</span></span> vm-></span>heap-></span>region_kiki;</span>
</span></span></span></span>  }</span></span>
</span></span></span>  else</span> {</span>
</span></span></span></span>    vm-></span>heap-></span>region =</span> RGN_BOBA;</span>
</span></span></span></span>    vm-></span>firstObject =</span> vm-></span>lastObject =</span> (</span>Object*</span>)</span></span> vm-></span>heap-></span>region_boba;</span>
</span></span></span></span>  }</span></span>
</span></span></span>  vm-></span>heap-></span>bump_offset =</span> 0</span>;</span>
</span></span></span></span></span>}</span></span></span>
</span></code></pre>
Also, since we're no longer using malloc()</code> to allocate objects, we have to update our newObject()</code> function.</p>
We also update how we allocate objects to remove the code from the intrusive linked list.</p>
Object*</span> newObject</span></span>(</span></span></span>VM*</span> vm</span>,</span> ObjectType type</span>)</span></span></span> </span>{</span></span></span>
</span></span></span>  if</span> (</span>vm-></span>numObjects ==</span> vm-></span>maxObjects)</span></span> gc</span>(</span></span></span>vm</span></span>)</span></span></span>;</span>
</span></span></span>
</span></span></span>  Object*</span> object =</span> heapAlloc</span>(</span></span></span>vm-></span>heap,</span> sizeof</span>(</span></span>Object</span>)</span></span></span></span>)</span></span></span>;</span>
</span></span></span></mark>  object-></span>type =</span> type;</span>
</span></span></span>  vm-></span>numObjects++</span>;</span>
</span></span></span></mark>
</span></span></span>  return</span> object;</span>
</span></span></span></span></span>}</span></span></span>
</span></code></pre>
Finally, we want to update our VM object to contain a Heap pointer. We also add the lastObject</code> pointer field that we use during the collection step.</p>
VM*</span> newVM</span></span>(</span></span></span>)</span></span></span> </span>{</span></span></span>
</span></span></span>  VM*</span> vm =</span> malloc</span>(</span></span></span>sizeof</span>(</span></span>VM</span>)</span></span></span></span>)</span></span></span>;</span>
</span></span></span>  vm-></span>stackSize =</span> 0</span>;</span>
</span></span></span>  vm-></span>firstObject =</span> NULL</span>;</span>
</span></span></span>  vm-></span>lastObject =</span> NULL</span>;</span>
</span></span></span></mark>  vm-></span>numObjects =</span> 0</span>;</span>
</span></span></span>  vm-></span>maxObjects =</span> INIT_OBJ_NUM_MAX;</span>
</span></span></span>  vm-></span>heap =</span> newHeap</span>(</span></span></span></span>)</span></span></span>;</span>
</span></span></span></mark>  return</span> vm;</span>
</span></span></span></span></span>}</span></span></span>
</span></code></pre>
Finally, as good responsible C citizens we free the heap object we've malloc()</code>'d when we free it's own VM.</p>
void</span> freeVM</span></span>(</span></span></span>VM *</span>vm</span>)</span></span></span> </span>{</span></span></span>
</span></span></span>  vm-></span>stackSize =</span> 0</span>;</span>
</span></span></span>  gc</span>(</span></span></span>vm</span></span>)</span></span></span>;</span>
</span></span></span>  free</span>(</span></span></span>vm-></span>heap</span></span>)</span></span></span>;</span>
</span></span></span></mark>  free</span>(</span></span></span>vm</span></span>)</span></span></span>;</span>
</span></span></span></span></span>}</span></span></span>
</span></code></pre>
No more marking, only forwarding.</h2>
Since we neither engage in marking nor sweeping, we can rip out almost all code related to mark and sweep code (which would no longer compile correctly anymore because we updated our VM and Object types).</p>
We need to first handle all of our roots. To do this, we simply walk over each of the root, and perform evacuation and forwarding on each of the pointed to objects:</p>
void</span> processRoots</span></span>(</span></span></span>VM*</span> vm</span>)</span></span></span> </span>{</span></span></span>
</span></span></span>  for</span> (</span>int</span> i =</span> 0</span>;</span> i <</span> vm-></span>stackSize;</span> i++</span>)</span></span> {</span>
</span></span></span></span>    vm-></span>stack[</span>i]</span></span> =</span> forward</span>(</span></span></span>vm,</span> vm-></span>stack[</span>i]</span></span></span></span>)</span></span></span>;</span>
</span></span></span></span>  }</span></span>
</span></span></span></span></span>}</span></span></span>
</span></code></pre>
When we forward an object, we first check to see if it has already been forwarded.</p>
If it has, we just return the forwarded pointer, which we know is in the to-heap.</p>
Object*</span> forward</span></span>(</span></span></span>VM*</span> vm</span>,</span> Object*</span> object</span>)</span></span></span> </span>{</span></span></span>
</span></span></span>  assert</span>(</span></span></span>inFromHeap</span>(</span></span></span>vm-></span>heap,</span> object</span></span>)</span></span></span>,</span> "</span>Object must be in from-heap."</span></span></span></span>)</span></span></span>;</span>
</span></span></span>  
</span></span></span>  if</span> (</span>object-></span>type ==</span> OBJ_FRWD)</span></span> return</span> object-></span>forward;</span>
</span></span></span>  //</span>...
</span></span></span></span></span></span>}</span></span></span>
</span></code></pre>
If it hasn't, then we allocate a new object in the to-heap, copy the contents of object in the from-heap over.
We then update the vm->lastObject</code> field to point one past the currently allocated object. This is one of the times in this codebase where i'm taking advantage of the fact all objects have the same overall size; if they were of a different size it would look more like vm->lastObject  = (Object*) ((char*)copied + getObjectSize(copied)</code>.</p>
Object*</span> forward</span></span>(</span></span></span>VM*</span> vm</span>,</span> Object*</span> object</span>)</span></span></span> </span>{</span></span></span>
</span></span></span>  assert</span>(</span></span></span>inFromHeap</span>(</span></span></span>vm-></span>heap,</span> object</span></span>)</span></span></span>,</span> "</span>Object must be in from-heap."</span></span></span></span>)</span></span></span>;</span>
</span></span></span>  
</span></span></span>  if</span> (</span>object-></span>type ==</span> OBJ_FRWD)</span></span> return</span> object-></span>forward;</span>
</span></span></span>
</span></span></span>  Object*</span> copied =</span> heapAlloc</span>(</span></span></span>vm-></span>heap,</span> sizeof</span>(</span></span>Object</span>)</span></span></span></span>)</span></span></span>;</span>
</span></span></span>  vm-></span>numObjects++</span>;</span>
</span></span></span>  memcpy</span>(</span></span></span>copied,</span> object,</span> sizeof</span>(</span></span>Object</span>)</span></span></span></span>)</span></span></span>;</span>
</span></span></span>
</span></span></span>  vm-></span>lastObject =</span> copied +</span> 1</span>;</span>
</span></span></span>  return</span> copied;</span>
</span></span></span></span></span>}</span></span></span>
</span></code></pre>
Finally, we set our old object to hold a forwarding pointer:</p>
Object*</span> forward</span></span>(</span></span></span>VM*</span> vm</span>,</span> Object*</span> object</span>)</span></span></span> </span>{</span></span></span>
</span></span></span>  assert</span>(</span></span></span>inFromHeap</span>(</span></span></span>vm-></span>heap,</span> object</span></span>)</span></span></span>,</span> "</span>Object must be in from-heap."</span></span></span></span>)</span></span></span>;</span>
</span></span></span>  
</span></span></span>  if</span> (</span>object-></span>type ==</span> OBJ_FRWD)</span></span> return</span> object-></span>forward;</span>
</span></span></span>
</span></span></span>  Object*</span> copied =</span> heapAlloc</span>(</span></span></span>vm-></span>heap,</span> sizeof</span>(</span></span>Object</span>)</span></span></span></span>)</span></span></span>;</span>
</span></span></span>  vm-></span>numObjects++</span>;</span>
</span></span></span>  memcpy</span>(</span></span></span>copied,</span> object,</span> sizeof</span>(</span></span>Object</span>)</span></span></span></span>)</span></span></span>;</span>
</span></span></span>
</span></span></span>  vm-></span>lastObject =</span> copied +</span> 1</span>;</span>
</span></span></span>  
</span></span></span>  //</span>set forwarding pointer
</span></span></span></span>  object-></span>type =</span> OBJ_FRWD;</span>
</span></span></span>  object-></span>forward =</span> copied;</span>
</span></span></span> 
</span></span></span>  return</span> copied;</span>
</span></span></span></span></span>}</span></span></span>
</span></code></pre>
To all of the indirectly traversable objects that I've left behind</h2>
Now that we've built the mechanism to forward our pointers, all managed objects pointed to by the roots have been handled. Next, we have to handle the objects that those moved objects still point to.</p>
In the original mark-sweep scheme, reachability had been solved before any memory management had occurred -- we recursively walk the object graph, bail if an object has already been marked to prevent loops, and then walk the linked list to purge all of the unmarked objects.</p>
Here, the reachability and marking works intermingled: we iterate through each of the objects in the new heap, and when we see it has a field pointing into the from-heap, we moved it and stick it onto the end of our heap, meaning, and once we get to scanning it we fix up any of it's pointers, until we hit the end.</p>
Taking advantage that all objects in our VM are the same size, we can just increment a pointer. If we didn't, we'd need to do work to increment via the object's size.</p>
void</span> processWorklist</span></span>(</span></span></span>VM*</span> vm</span>)</span></span></span> </span>{</span></span></span>
</span></span></span>  while</span> (</span>vm-></span>firstObject !=</span> vm-></span>lastObject)</span></span> {</span>
</span></span></span></span>    //</span>forward sub-pointers of Pair object
</span></span></span></span></span>    if</span> (</span>vm-></span>firstObject-></span>type ==</span> OBJ_PAIR)</span></span> {</span>
</span></span></span></span></span>      vm-></span>firstObject-></span>head =</span> forward</span>(</span></span></span>vm,</span> vm-></span>firstObject-></span>head</span></span>)</span></span></span>;</span>
</span></span></span></span></span>      
</span></span></span></span></span>      vm-></span>firstObject-></span>tail =</span> forward</span>(</span></span></span>vm,</span> vm-></span>firstObject-></span>tail</span></span>)</span></span></span>;</span>
</span></span></span></span></span>    }</span></span>
</span></span></span></span>
</span></span></span></span>    vm-></span>firstObject++</span>;</span>
</span></span></span></span>  }</span></span>
</span></span></span></span></span>}</span></span></span>
</span></code></pre>
Tying the garbage bag together</h2>
Finally, we change the gc()</code> function so that we now call into our new mark and compact scheme.</p>
void</span> gc</span></span>(</span></span></span>VM*</span> vm</span>)</span></span></span> </span>{</span></span></span>
</span></span></span>  int</span> numObjects =</span> vm-></span>numObjects;</span>
</span></span></span>  vm-></span>numObjects =</span> 0</span>;</span>
</span></span></span>
</span></span></span>  swapHeap</span>(</span></span></span>vm</span></span>)</span></span></span>;</span>
</span></span></span></mark>  processRoots</span>(</span></span></span>vm</span></span>)</span></span></span>;</span>
</span></span></span></mark>  processWorklist</span>(</span></span></span>vm</span></span>)</span></span></span>;</span>
</span></span></span></mark>  vm-></span>lastObject =</span> NULL</span>;</span>
</span></span></span></mark>  vm-></span>firstObject =</span> NULL</span>;</span>
</span></span></span></mark>
</span></span></span>  vm-></span>maxObjects =</span> vm-></span>numObjects ==</span> 0</span> ?</span> INIT_OBJ_NUM_MAX :</span> vm-></span>numObjects *</span> 2</span>;</span>
</span></span></span>
</span></span></span>  printf</span>(</span></span></span>"</span>Collected %d</span> objects, %d</span> remaining.\n</span>"</span></span>,</span> numObjects -</span> vm-></span>numObjects,</span>
</span></span></span></span></span>         vm-></span>numObjects</span></span>)</span></span></span>;</span>
</span></span></span></span></span>}</span></span></span>
</span></code></pre>
Pros and Cons of a copy-collector</h2>
So, now we have a copying collector. What did we gain, and what will we tearfully grieve?</p>
Pro: Removed object list overhead.</h3>
Directly, we've removed the overhead of an additional pointer that the VM only uses during garbage collection, which does save us 8 bytes per object, and that sort of cost per-item can turn out to be quite large.</p>
But in addition to that, we've also removed a temporal overhead in addition to a spatial overhead -- we no longer traversing a linked list, and that can be a huge change due to Cache coherency</a>. Because we're only ever stepping over an array, we won't clear the cache out, and we (potentially) might have no more issues with blowing out our memory cache and repopulating it.</p>
We also have a much</em> simpler allocator than your systems malloc()</code>/free()</code>, but we do so at the cost of that systems flexibility, as giving up any effects of hardening against possible exploitation or preventing data races. That isn't a big deal with our tiny mutator in a system that'll never go into production, but is something worth considering for more serious projects.</p>
Pro: Active work depends on the number of live objects</h3>
A mark and sweep collector needs to traverse the entire object graphs while cleaning up objects. In contrast, the copy-collector only needs to work for each of the roots, and each of the objects that has been moved into the to-heap. If we have a very large number of dead objects and a very small number of live objects, this can be a massive gain on the mark and sweep implementation.</p>
Pro: Reduces fragmentation</h3>
A mark and sweep garbage collector using malloc()</code> (or any other allocator that deals with variably-sized data) has to content with fragmentation</em> -- after long cycles of freeing and releasing memory, it's possible that we might have enough memory to allocate it, but we don't have contiguous chunks of memory that it could fit. When copying collectors move objects, they push them all to one side of the heap, meaning we now have a large contiguous area to allocate new memory from, and there's no awkwardly sized gaps.</p>
Con: Halves the effective free space</h3>
All of this talk about the utility and benefits we get from cache coherency is nice and all, but we're also overlooking a bit of a big one:</p>
We are only ever using half of our allocation space. The rest of it is always just sitting there, without anything to do, just dead. That's kind of a scary amount of stuff to give up -- normally when we think of overhead or potentially unusable space, it's small little chunks that add up, not a huge compromise outright.</p>
Con: We can't do extra work per deleted object?</h3>
Also a bit of a weird one, but one of the biggest benefits of bump allocators is also a weakness -- we blow away all of our unused memory with one simple command. This is perfect for the small mutator we work with, but if we were using C++ and those objects had non-trivial destructors, or we were implementing a language with finalizers</a>, then we lose access to all of those objects that might need to have additional code run, and we'd need to add some sort of mechanism to find and walk through all of the objects that need to be destroyed.</p>
Con: pointers to objects aren't stable and we need to patch up root objects.</h3>
Many programming languages need to offer an ability for code that isn't handled as part of the garbage collection system, like C extensions in CPython</a>. In Mark and Sweep schemes (or in CPython's case, a hybrid reference-counting/sweep scheme), pointers to objects handled by the garbage collector are stable until their free'd, so you can extend your system to tell the GC that these objects should be treated as roots themselves and not be de-allocated until their marked as no-longer a root. But in a system that moves the object, we need to update any pointer to the old object to it's evacuated counterpart, and this can significantly complicate the language API.</p>
Go forward and copy</h2>
And, well, that's a copying collector! Much like the original, this is a pretty trivial project and implementing this in production will probably run into huge number of bottlenecks and potential issues, but it's certainly not an unusable toy either -- this was</em> cutting edge research at one point! And versions of it are used in production, like in the android runtime(Video link)</a>.</p>
And also, I think this is doing some nice work describing a useful piece of low-level tech that certainly has explainers online and in textbooks, but not any as in-depth as either Nystrom's original post, nor with as nice pictures.</p>

Garbage Collection Handbook

2023-07-07T00:00:00+00:00

The garbage collection handbook</h1>
4/5
purchase link</a>.
This book is a bit of a hard one to talk about, because a new version</a> of the book is going to come out soon. If you are reading this, I'd recommend buying the newer edition when it comes out.
This book feels like a collection of blog posts on the topic of garbage collector design -- it first covers the 4 major algorithm types (reference counting, mark and sweep, mark compact, copying), then a discussion of it's overall merits, then a walkthrough of the more advanced sort of properties you would want in a garbage collection (like incremental, parallel, concurrent, and so on). It is a very thorough, and has some discussions of some programming-language specific features like finalizers, and finally a chapter on real time designs.
This book is also solid goldmine of citations to dive into the literature. However,because this book came out in 2012, and the citation in the book mostly range in the 1995-2009 range, which means there's around 14 years of progress you are missing out on.
Additionally, I bought the eBook version on Amazon, but that was a mistake because it's mostly a fixed-formatting export of the PDF and I couldn't read it on my kindle, and it was kind of unpleasant to read it on the desktop client. If you can't wait for the new edition, then I would recommend getting the paperback.

What I wished I knew when learning C

2023-06-30T00:00:00+00:00

What I wished I knew when learning C</h1>
This article is a response/take on both Daniel Bachler's article on learning F#</a> and Hillel Wayne's on the hard part of learning a new language</a>. I thought it would be educational and a fun exercise to read this article, but covering C.
Often teaching tutorials don't focus as much on the surrounding ecosystem of how to do development, what tooling to use, or some deeper lore. Almost all of this stuff is generally documented (and where a good source exists, I'll try and add a link), but there's nowhere that has attempted to create a consistent, coherent listing of most of this information.
This document is almost certainly incomplete! There's lots of information out there, and I probably got some of it wrong. I'm also not at all familiar with many ecosystems, like the Mac, and thus can't really speak to it.

Why would I want to use C?</h2>
C is many ways a foundational language to other ecosystems -- it is not unusual for other programming languages to depend partly or completely on a C compiler to bootstrap it's compiler or runtime, many languages use C as the defacto Foreign Function Interface, and for several systems system utilities are exposed as C libraries you call into. If the question of "Should I write this in C?" comes up, it's totally possible you might not have any other major options.

When you want to add extensions to multiple programming languages</li>
When you need to interface with a system or user library that only exposes a C API.</li>
When consumers of your code mostly utilize the C FFI.</li>
When you're writing something that requires not having a heavy runtime, like operating system or firmware on a resource constrained system.</li>
When using a more managed higher level language is too slow or has too much overhead (potentially in the context of implementing a fast/efficient part of a project)</li> </ul>
Why would I Not want to use C</h2>
C is a memory unsafe language, and you have to manage memory management yourself.</li>
When you are working with code that needs to secure or handle untrusted input.</li>
When performance isn't important, but speed of development or utility is, like in scripting or exploratory programming.</li>
When you need to do complex string manipulation.</li>
when you need to have a large set of mature, well developed data structures and algorithms.</li>
When you are working with complex graphs of varying objects that have non-trivial lifetimes, and thus would benefit from a garbage collector.</li> </ul>
Where can I run C code?</h2>
C is almost always compiled (although interpreters do exist), so generally anywhere that can run native code and you have a compiler you can run C. C also can be compiled to wasm or javascript through Emscripten</a>, and thus you can run C code in the browser.
The C++ (And Objective-C) connection</h2>
You often hear C and C++ mentioned in the same breath, or together as C/++, and that's because while these languages are often quite different, they have close histories and C++ includes almost all of the language elements and standard library functions that C does. All of the major compilers (MSVC, GCC, Clang) all can ingest both C++ and C (and Objective-C!), so using code written in C++ from C (or sometimes vice versa, called Hermetic C++). Despite this, the languages have different cultures, norms, and ways of doing things, and often people lean towards using one language or the other than more freely mixing them.
Installing a development environment: Toolchains and tools</h2>
The set of tools needed to build C are generally referred to as a toolchain, which traditionally includes the compiler, assembler, linker, pre-processor and a set of other associated tools. The specifics are toolchain specific but inside of the nix world, they mostly follow a similar pattern. You don't need to generally worry about installing a particular part of a compiler since they are packaged together
GCC</h3>
GCC (standing for the Gnu Compiler Collection) is one of the most common and well known of the major compilers available to devs. It is available for free (and is one of the canonical examples of Free/Open Source software), and generally can be installed via your system package manager. One of the advantages of GCC is it's age: it's a long running complex project, and it also has a number of compiler backend targets you can utilize, as well as a pretty large space of added extensions and utilities. Generally, GCC is the compiler I default to using unless I have a specific reason to choose another, although Clang is equally as worthwhile and useful.
`# install on Ubuntu > sudo apt install gcc </code></pre>`*
Clang</h3>
Clang is the other major open source nix toolchains, and was developed in part to accept most of the extensions and compiler settings that GCC provides, meaning most software that uses GCC can probably be compiled with Clang. Clang and GCC are mostly compatible in terms of performance, so either are a fine choice unless you're relying on a particular backend or feature. It is also helpful to try and build software with both because it might pick up a number of errors the other might not. One thing Clang provides is that it is (by default, anyway) a cross compiler with the supported backends built in, whereas with GCC you will need to install a build of the compiler for that specific backend.
`# install on Ubuntu > sudo apt install clang </code></pre>`*
MSVC</h3>
MSVC is Microsofts in house C and C++ Compiler (they treat the barrier between the two languages a bit fuzzier than clang and gcc), and while it is not open source it is freely available. To install it you have to install visual studio (which is not visual studio code, the naming trips me up sometimes too), and versions of the compiler have until recently been strongly bundled with the broader visual studio environment, which you can download here</a>.
While you don't need to actually use the Visual Studio IDE if you don't want to and can instead rely on the standalone CLI tools, it's not unusual that this will cause some problems because those are not natively added to theenvironmentt but are instead added to the environment through special shortcuts that open `cmd.exe</code> with all of the useful command line files added, see` `documentation for details</a>.`
`One of the smaller benefits of CMake is that, as long as your MSVC is installed in a standard location, CMake can actually pick up on it's location and natively call it's tools for both the configure and build steps.`
Mingw</h3> Mingw (and it's more up to date port Mingw-64) is a port of GCC designed to compile code for Windows, either as cross-compilation or hosted natively. Generally speaking, MSVC is the way to build native C/++ code for Windows; Mingw works and can handle compiling a large number of projects but the toolchain has some tradeoffs that using MSVC doesn't, like utilizing an old and undocumented systems library. Unless you have a strong reason to, I would recommend utilizing MSVC if you are building compiler native. It can be installed on most repos via their package managers, and can also be downloaded here</a>. How should I be writing this: Text editors and IDEs</h2> Since C is so old and utilized on so many platforms, most major text editors have some form of C/++ support, either builtin or through user provided packages. Visual Studio Code</h3> Visual Studio Code is a cross-platform general purpose text editor and "light IDE", with a pretty huge repository of official and community supplied plugins. It is responsible for the creation of the Language Service Protocol</a>, which has been adopted in other editors, and is overall a very handy and mature editor for a large variety of langauges. It also has a very useful remote editing mode. This is my go to default. For setting up C/++ development specifically, these are wonderful tools provided by Microsoft themselves. C/C++ Tools</a></li> CMake Tools</a></li> Makefile Tools</a></li> </ul> Visual Studio</h3> Visual Studio, in addition to hosting the C/++ toolchain and other tools, is a fully featured and fairly rich IDE. It has a number of officially supported plugins for all the major languages Microsoft supports, and has been working on the C editor experience for years. I don't generally use it, but it's a solid and mature bit of tech, and if you are targeting or developing on Windows it's a solid choice. CLion</h3> CLion was developed by Jetbrains, who have built a number of language specific editors, and CLion is their C/C++ specific tooling. It generally relies on CMake as its build system, although it can import projects utilizing Makefiles or other constructs. It's a paid IDE and costs about $100/year, or less if you buy the full bundle of Intellisense tools. I quite like CLion, although a few choices (like some of the default formatting) are a bit fiddly and I have to fight against or edit the configs, but is a rich editing environment and I'm happy to have paid for it. Unfortunately it's third party plugin ecosystem is much more sparse than the other code editors on this list. CLion can be installed here</a>. Vim</h3> Vim (and it's quasi fork Neovim) is a pretty old and advanced command line based editing environment. By default it only has syntax highlighting for code, but has a pretty huge developer code base so you can absolutely fill up with tons of utility. Generally some of the bigger advantages of Vim is that it's likely to be present most computers, and you can use it if you are developing code remotely via SSH. Where should I ask for help: Documentation</h2> Manpages</h3> The historical way of finding development for C language functionality have been the Manpages, a database of documentation and a command line client that is pre-installed on almost all nix style computers. Generally, utilities, the system libraries and many libraries will put documentation in a manpage, which are normally written in a simple type setting system called Roff. Manpages are ordered into sections which act as namespaces (See this handy manpage for details</a>). Generally, the sections you'll want are in section 1 (command line tools), section 2 (syscalls and their wrapper functions), section 3 (library functions), and occasionally section 7 (overview or general topics). Online versions of the manpages are hosted at several locations (The linux kernel project mains one</a>), but these might have differences from the versions installed on your computer. I would instead recommend the extremely useful DWWW</a>, a local webserver that renders manpages into HTML and indexes them to be searchable. Unfortunately, manpags are slightly out of vogue, so it's not unusual for projects to no longer provide manpages or provide fairly spartan manpages with the expectation you'll look elsewhere. Gnu Info</h3> Gnu Info was designed to act as an upgrade of sorts to the manpages -- manpages were small and standalone, while Gnu Info was meant to have more of a hypertext-style set of linked pages. Gnu Info isn't used that much (not even as much as manpages, in my experience), but many Gnu projects utilize it (oftentimes provided a very spartan manpage telling you to look up the information in Info), but otherwise I think it's mostly passed. Even the Gnu projects often provide online HTML or PDF formats for their documentation, which is what I tend to refer to. Locally built documentation: Doxygen</h3> Doxygen is both a tool and general format for providing documentation inline in a C++ file -- while it's not unusual for developers to use doxygen style comments to add documentation to their code but not actually use Doxygen proper, it's still pretty common for older projects. If a project has doxygen documentation, you can build it locally and thus have an up to date version. Active project often host their documentation online, but usually only for the latest or long term release. Microsoft Development Network (MSDN)</h3> Microsoft provides documentation for their compilers, language extensions, and libraries on MSDN</a>, and it acts like the manpages for Linux do, often as primary form of documentation. The C build process</h2> C has a somewhat peculiar way of compiling things if you're coming from other languages: Normally, you compile each .c</code> file to an object file .o</code>; then you end up linking these together into a final executable using a linker. If you have a project with the files: foo.h</li> foo.c</li> main.c</li> util.h</li> </ul> Compiling it using the traditional unix style toolchain would look like > cc -C foo.c -o foo.o > cc -C main.c -o foo.o > ld main.o foo.o -o main </code></pre> While I called this model "peculiar", the only element that is unusual is the use of textual inclusion for header files; most other aspects are pretty similar to what most compiled languages do, the traditional C development environment only makes this pretty explicit. One of the advantages of this is partial builds -- if you keep around the object files after compiling something, you only need to recompile the object files for the c files you update. Depending on your build system, the intermediate outputs of compilation might be stored either in the same directories as the source code (generally referred to as in tree builds), or stored in a separate build directory (out of tree builds). Makefile-based projects lean heavily towards in tree builds, and CMake leans heavily towards out of tree. Includes and the include path</h3> C has two flavor of ways of including header files #include <stdint.h> #include "somefile.h" </code></pre> Technically the definition is implementation defined, but universally the meaning ascribed to these is that <></code> searches through some global set of paths, whereas the ""</code> include style first searches the local directories for this. In both cases, there's a set of paths pre-determined by your development environment and toolchain, and you can add files to the include path as a command line argument, normally -I /include/path</code>. This is relied on heavily by project that move their header files to a separate directory, although usually that is handled by the build system in some flavor. Macros and the preprocessor</h3> The C preprocessor is also, technically, a standalone tool you can run standalone on non-C files, although the only other major example I can think of is Fortran. It is still quite strongly tied up with the language definition, and C macros are required to follow the rules of valid identifiers in C code (although as they are textual macros, you can do wild stuff with them like passing +</code> as an argument to a function). If you ever need to debug macro code, you can call gcc -E code.c -o code_preprocessed.c</code> in nix land, MSVC uses distinct arguments</a>. Assembly and the assembler</h3> Technically, the C code to object file pipeline has a 2nd intermediate step where you compile code to a format that is (in some ways) a textual representation of what native code for your target architecture looks like, traditionally called assembly, which is transformed into native code using an assembler. This format is normally not output unless you specifically ask for it. On nix machines it traditionally has the file extension .s</code> or .S</code> (if you use the preprocessor), while on Windows it is .asm</code>. In addition to it being useful to look at a textual output of your final code, you can also write files in Assembly and manually invoke the assembler like as foo.s -o foo.o</code>. This is usually something you don't need but is sometimes used if you need extreme performance and need to hand-tune the output of your code, or for implementing low-lelvel syscalls or libc features that cannot be done in C itself. Note that assembly syntax is extremely target and vendor specific -- each machine architecture probably has it's own syntax and quirks, and the X86 family of architectures makes sure to go the extra mile by having two syntaxes, and even two competing assemblers might differ in details beyond the base syntax. Linkage and the linker</h3> The final step of the traditional build pipeline is linking, where you take all of the intermediate code in the object files and fuse them together into a final executable or library. This step involves laying out sections of data in the executable, and making sure that function calls to functions defined outside of an object file are correctly linked up. There is a custom language to control and manipulate how the linker behaves called linker script. Unless you are writing a compiler this a pretty rare format to interact with. McYoung has a fantastic guide to the details of what interacting with linker script is like</a>. Bundling libraries</h3> In nix land you normally build a static library via utilizing ar</code> (originally an archive manager) to bundle all of your files together, and shared libraries via a special flag to the compiler. While these create file that toolchain (and for dynamic files, the loader) to use, there normally is a more complex set of steps for actually bundling code to distribute to other libraries. Libraries</h2> Whenever you need to rely on behavior that's not built into the language, you'll have to rely on a library. Libraries are a way of tying together functions for performing actions as well as interacting with the host systems. For instance, SDL is a library with functions that let you draw things to a screen; libjson is a library with functions for parsing and serializing data to the JSON interchange format, and so on. There's two functional varieties of libraries in use: static and dynamic libraries, both utilizing header files to define their interfaces. Header Files</h3> Header files are not technically considered a distinct construct by the C standard, but are so defacto their built into the naming convention of files: files meant to be #include</code> use a file extension of .h</code>. Header files are often used internally in projects to logically separate details of a project separate, and often it's normal for there to be a pair of .c</code> and .h</code> files for one part of a program, e.g. parser.h</code> and parser.c</code> for the code in a compiler to parse code. Header files are also used to describe the interface a library exposes to consumers of a library: normally header files contain prototypes for functions that the library implement, definition of types for them to operate on, and macros for both utility and to describe architecture specific behavior. The only way for a library to expose it's functions and constructs is via a set of header files. Static Libraries</h3> These are technically the older of the two concepts. A static library is a grouping of functions in a way that can be linked into a binary when that binary is being constructed. This allows you to fix in behavior and not rely on the presence of a dynamic library on the users system. Some of the advantages of static linking are that you can set behavior at compile time, and you can prune any of the functions and values that are not used by the binary. Some larger projects also use static libraries to break up implementation and usage -- if you're creating a video game, you could put all of the main logic in a library called libgame</code> in a function called startgame()</code>, and have one executable called game</code> that immediately calls startgame()</code>, and another executable that instead is a model viewer or sound text. The disadvantages of static linking are that you are permanently locking in information, rather than relying on potentially more update dynamic libraries. This could be very bad if the version of your networking library has an error that opens you up to security faults, or if a change in unicode adds a number of new digraphs an older version of the library couldn't handle. Dynamic Libraries</h3> Dynamic libraries are the opposite of static libraries: the functions you are using are not loaded in til the executable is run, and then those functions are fetched in so you are always using the implementation the system provides, even if you change those functions out. Dynamic libraries are also often used for plugins: since you can manually load in libraries and associated values with Unix dlopen()/dlsym()</code> and Windows LoadLibrary()/GetProcAddress()</code>, you can load in additional logic to be run without baking it in. A video game might do this to load in mods, or a text editor to load a plugin to get support for a new programming language. The advantage of dynamic libraries are that they are always loaded in fresh, and thus you can fix security faults or add new behavior to older programs, and they can also save on system RAM -- if you don't have to load a copy of the math functions into every running binary, then you save a good deal of extra space. The problem with dynamic libraries is to take advantage of these benefits, you have to make sure that the library is the right version to match what your executable expects to use, which in the worst possible case involves bundling copies of the dynamic library you are relying on with your code. Libc and other special libraries</h3> There is one library that is generally singled out for a special purpose: the standard library, usually called libc. Almost all hosted programs utilize libc, which exposes most of the functions in the standard, as well as manipulating some of the core runtime features like files and signal handling. While libc could just be a special library, it is often intimately tied into the runtime system, and some compilers hard code expectations about what will be in it. In addition to libc, a few other libraries exist in the sort of special place where they are likely tied to the compiler (or even part of libc) libm, which implements several math functions</li> libdl, which implements functions for working with dynamic libraries</li> </ul> Header only libraries</h3> Header only libraries are more of a concept in the C++ world, but do seem some usage in C. Because of the complexity of build systems and wide range of practices, sometimes developers take advantage of the way inline</code> allows for multiple definitions in a project to provide both the prototypes and implementations in a header file, which means that once you have the downloaded somewhere on the system you have no additional work to build it with aside from #include <headeronly.h></code>. C Build Systems</h2> Like other older languages, C predates the design trend where language compilers come with an extended set of tools bundled together and suggested structure for what a project should look like. Instead, there's a pretty wide range of different tools to use, and each implicitly suggests a certain structure. Why use a build system?</h3> You could just write a build script that looks like gcc .c -o foo</code>, but while that might work for very small projects it quickly becomes a headache if your project is very big, or have a more complicated layout. Build systems are a way of describing what actions to take to compile a set of source files into a final form (or several final forms). They allow to describe this in a more concise, declarative form: working with the command line version itself might get messy if you need to only compile foo_windows.c</code> on Windows, but need to compile foo.c</code> on both Linux and MacOS. Additionally, build systems keep track of when the last time you edited a file, and only compile the source code files that have been changed. This doesn't seem like a big deal if your dealing with a small project that compiles in under a second, but it really matters if you're working with a massive codebase that can take hours to compile. Make (GMake, BMake)</h3> Make is probably the oldest build system out there. While POSIX defines a very limited subset of rules and utilities, the major Make implementations (GNU Make and BSD Make) both have a much larger set of features. Generally, I use Make for for smaller projects and simple examples because it is very easy to spin up with, and to add items to it incrementally. I generally lean away from using it if the project is expected to be quite large, or rely lots of complex conditional logic. The sample for our little project might look like .PHONY: all clean CFLAGS= -Wall -Wextra all: foo foo: main.o foo.o @$(CC) $(CFLAGS) main.o foo.o $(LDFLAGS) -o foo main.o: main.c foo.h foo.o: foo.c foo.h clean: @rm -rf .o foo </code></pre> Ninja</h3> Ninja is like a much more constrained Make, but was instead developed for meta-build systems like CMake below. It was designed to be fast to execute and build, and to not be written by hand. CMake</h3> CMake bills itself as a "meta-build system" -- you write a project's build scripts in the CMake scripting language, and these are then compiled into a build target suitable for the system (such as Makefiles, Ninja, Visual Studio Code). It also acts as a sort of configuration system so that libraries using CMake can be easily built and consumed by other projects also using it with a minimal amount of glue code. CMake also tends to change the layout of projects -- while Make-style targets usually put header files the same file as implementation files, CMake projects tend to put them in a distinct location and then add that to the header include path via commands, which means your IDE will probably have to have some introspection of the CMakeLists.txt command or it's output to allow intellisense to handle things correctly. Generally I use CMake if I expect the project to be bigger, involve a lot of fiddly logic, or if I'm using C++. It has a bit of a nasty reputation, but as long as you internalize a good way of structuring things I find it isn't too bad, just verbose. I prefer using most of the style suggestions in An Introduction to Modern CMake</a>. Visual Studio build system</h3> Visual Studio C has it's own format (which is a specialized XML file) and build tool; but really would rather you edit it via an exposed gui interface. I've used it a decent amount, but it's quite hard to easily show GUI interfaces textually. I don't have anything against it -- if you are making a Windows-primary thing and using Visual Studio, you should probably either use this or CMake. Code checking and formatting</h2> Especially when writing C, it is useful to have tools that look for errors or other issues in static code as well as utilizing runtime debugging. These can be useful for automatic CI/CD checking of pull requests or code being added to master. Compiler warnings</h3> The first line of defense is compiler warnings -- by default, the C compiler will actually let a whole lot of possible errors it can detect statically fall through, and it is considered best practice to turn these on. Normally compilers provide flags for both individual warnings, and bundles of warnings grouped together you can enable. In GCC and Clang that usually looks like -Wall</code> or Wall -Wextra</code>, and in MSVC it looks like /W4</code>. Additionally, an IDE might take advantage of the flagged errors to show possible issues to you. Note that even using these bundles might not enable every possible warning Compiler warnings as errors</h3> Compilers also include the option to make any detected compiler warning an error; in MSVC land this is /WX</code> and in GCC/Clang land it's -Werror</code>. This is generally considered best practice, but a bit of a controversial one: some of the more advanced warnings in the compilers can act more like style guide rules, or flag situations that are unlikely to be an error most of the time, leading to something akin to alarm fatigue. At least in GCC, it is possible to manually specify only some warnings as errors</a>, which is probably the best choice for when adding additional code rigour to an existing code base without warnings as errors already enabled. If you are developing greenfield code, it is probably worthwhile to enable it. Clang Format</h3> Clang format is a configurable automatic code formatter in the style of gofmt</code>, though in the spirit of C's heterogenous environment is widely configurable and has several baked in pre-sets you can modify. I've never really used clang format, but most open source project's I've looked have a .clang-format</code> configuration file sitting somewhere in the top level of the codebase. Like a lot of other tools, it might be more useful to introduce this during greenfield development that add in later. Static Analysis Tools and Linters</h3> While the compiler can act as a primary sort of static code analysis warning, since it already has a pretty intimate view of the codebase while its compiling, more complex or advanced tools designed around checking and verifying C code have sprouted. However, their use always seems to be kind of rare -- I've used CppCheck twice, and I'm not aware of many open source projects that utilize them, but I could be totally mistaken. It also seems like until somewhat recently, these were mostly the domain of commercial software that was normally outside of personal developers budgets. The main ones I'm aware of that are free are clang-tidy and cppcheck, both of which are open source and easily available. Debugging C</h2> Debugging C is something a lot of newcomers don't think about at first, but its necessary as your codebase grows in scope to be able to inspect a program as its running to figure out possible errors. Getting comfortable with debugging techniques and tools are absolutely necessary for long-term maintenance. printf()</code></h3> It's kind of dumb to say this, but sometimes it's useful to rely on printing out information to have some idea of what's going on in your system. More advanced projects will develop some sort of logging library, which is just a fancier printf that lets you add some extra information about where a call was made and controlling what sort of depth of information gets printed. Debuggers (gdb, lldb, windbg)</h3> When printf()</code> doesn't cover it, or you're getting a SEGFAULT, then you should bring in debuggers. These take advantage of system hooks to allow you to execute a program line by line and inspect the values of variables and parameters, and to show you were segfaults or other signals are triggered. GDB is a perennial one, and provides both a command line interface and a remote protocol interface so that you can debug on remote machines. Windbg is a windows-specific debugger that has two frontends, and can also be used for debugging kernel level information. To use debuggers, you should probably put your code in debug mode/low optimization and add debug symbols, which are necessary for debuggers to introspect a binary. Valgrind</h3> Valgrind is a much more heavy weight debugger and effectively acts as a native code interpreter you feed a binary program in, allowing you to inspect for errors that are not obviously until runtime, including more subtle ones like misuse of file descriptors. It has a pretty major speed overhead, which may make it difficult to use for some projects or if they have memory usage. But you get a pretty wide suite of detection for errors. Sanitizers</h3> Sanitizers perform similar tasks to Valgrind (finding higher level errors at runtime), but unlike Valgrind does not require loading the binary into an interpreter; instead all of the runtime checks and tooling are installed into the binary during compile time, allowing you to simply run the binary like normal and only seeing errors if one occurs. Unfortunately, sanitizers are partially incompatible with each other, and thus the permutation of sanitizers you can have enable at once is constrained; in particular UBSan and ASAN are not compatible together. Standards and versions</h2> C is a standardized langauge, meaning there is a set of canonical documents describing it and the runtime, environment and so on. This is a trend that's become less popular in newer languages as compiler development has become more open source, but historically this was valuable to ensure your code didn't accidentally rely on vendor specific extensions that would make it difficult to port your code somewhere else. The Standards</h3> The major versions of C are laid out in the ISO Standards documents. Unfortunately, all of the actual standards documents themselves are behind a decently pricey paywall, so most of the community relies on utilizing the final drafts, since these are almost surely quite close to the final standard. These aren't just hypotheticals -- GCC and Clang both take an option specificizing which version of the standard you want to adhere too. Companies or individual projects often will strongly specify what standard is allowed, and different standards expose specific functionality. Developers interested in portability often target older versions of the standard, usually either C89 or increasingly C99. If you utilize the command line arguments to select a particular dialect, you might end up disabling access to some of the functions specified by POSIX or your local environment. Pre-standardization C</h3> C existed as a existed as a language for at least 18 years before the committee formed that would create the C standards, and during that time almost every vendor would develop it's own compiler with it's own features and syntax extensions. The closest thing that existed to a standard was the textbook written by some of C's original developers, The C Programming Language</a>, and the details of the language shown in the first edition are sometimes called K&R C. C89 (aka ANSI C)</h3> Draft available here</a>. This is the earliest standardized version of C. Often times, this is the baseline people wanting to make their code generally usable across a wide swatches of platforms will default to. The primary difference between it and later versions are that all variables must be declared at the the start of a block, and several standard types and functions were later added to the library. Fun fact: the reference compiler of C89 was actually written in pre-standardized C++! C99</h3> Draft available here</a> C99 is the other major version of C people treat as a generally safe common denominator. The major syntactic additions it added are variably length structs via flexible array members and variable length arrays (later walked back in C11), complex number types, the C++ //</code> single line comment style, variable declarations allowed in the middle of a block, and the addition of several types for fixed size arithmetic. C11</h3> Draft available here</a> C11 makes some of the new addition in C99 (like variable length arrays) optional. It also adds a limited form of compile time generics via the _Generic</code> directive, as well as support for threads and associated multi-threading primitives. C17</h3> Draft available here</a> C17 was a pretty small standard update that added few things, and mostly deprecated or clarified positions in the C11 draft. C2X</h3> Current working draft here</a> The current version of the standard being worked on appears to be a major change and thus this section likely to be out of date soon, this has changed a number of technical details (several old macros have now become keyword), introduced C++ style attributes, a preprocessor directive used for embedding files directly as binary arrays. POSIX</h3> POSIX refers to a set of specifications made in the 90s and later updated, trying to find the minimal common set of library interfaces, tools and behavior that were in common between them. The libraries defined in POSIX are the "rest" and a quick listing is avaliable here</a>. Glibc and GCC extensions</h3> GCC and the GNU Projects major libc implementation (glibc</code>) provide a huge number of extension functions and utilities, and software has come to rely on it. Like Windows, the glibc developers have focused on backwards compatibility, so it acts as a bit of a de facto standard much like the Win32 environment does. The Win32 Environment</h3> While Windows doesn't define any sort of standard describing it's libraries and interfaces, Microsoft has a pretty extreme dedication to reverse compatibility that means you can effectively rely on the Win32 C environment to act as a stable environment to build software against. Portability, Undefined behavior, and memory safety</h2> Undefined behavior is one of the other major weird parts of C, and something every developer is probably going to run up against. The hierarchy of behavior</h3> The C standards provides a sort of hierarchy of how conformant code can be to the standard. Like a lot of the C standard, the combination of legal-like language with a prose description of the operational semantics of a language are kind of mystifying at first if you are not familiar with it. Specified behavior: Code whose behavior an implementation of C must handle, and the result of which is expected to be the same across compilers.</li> Implementation defined behavior: Code that a legal implementation of C must handle, but the result of which can be set by the implementation.</li> Undefined behavior: Code that a conforming implementation of C is not required to handle, and the standard makes no requirements on what may occur -- the implementation is not at all constrained about what to do, and may (in the legalistic terms) do "anything".</li> </ul> However, at least in compiler circles land, Undefined Behavior is also used to describe any code that is not defined behavior for a particular implementation. In that case, I think it's maybe better to label it as "violations of the compiler invariants", which may cause compiler errors or result in a compiler outputting code with an error in it, or result in code that works under most cases but fails on a few. The C Abstract Machine</h3> When reading the C standard, most of the way it describes the language is using an ideal of "The abstract machine" -- a sort of idealized C interpreter for C that transforms and executes the code without many specific details. Accidental non-portability</h3> It's very easy to accidentally rely on a piece of implementation specific or even undefined behavior that happens to work like how you think it should on your development environment, only to then discover that the codes reliance on these features in a way that causes bugs if you try to port the code to another platform. This isn't a pure hypothetical, or something that comes up when working with obscure and exotic processors -- this happens to people porting a game from one of the most popular gaming platforms (the PC) to another majorly popular gaming platform (The Nintendo Switch): With that out of the way, the next step was multiplayer determinism. One big goal was that I didn't want to cut multiplayer from the game. Furthermore, I wanted players on PC to play with players on Nintendo Switch. This is the first time we had to make sure the game is deterministic between ARM and x86. We should be fine, C++ is portable, right? Just don't use undefined behavior. Turns out we use quite a lot of undefined behavior, both in our main code and in the libraries. For example, when casting a double to an integer, if the value does not fit in the integer, it is considered undefined behavior and the resulted value is different on ARM and x86 CPUs. </blockquote> https://factorio.com/blog/post/fff-370</a> You don't get portability for free</h3> You probably shouldn't be freely relying on undefined behavior that does work in your production environment because it's possible that will introduce bugs later on or leave code unable to compile if you upgrade your compiler version. But it's worth contemplating whether or not you want to commit to work and hassle of making code that can and should compile on even the most exotic of platforms, because despite C being a portable language it's very easy to write non-portable code unless your on guard and constantly testing, and sometimes committing to less portable code reliant on a particular environment just is the right choice for what you are doing. Dependency management</h2> Or, where are libraries exactly...? In what has probably become a theme now, C is old enough that it predates the sorts of environments that necessitate handling dependencies and libraries. System wide package manager</h3> This is a bit more of a Linux construct, but probably one of the default ways many software engineers rely on is using the system wide package manager and relying on your local distribution managers to have built and packaged up a library, header files, and any other additional configuration details in a well known place. For instance, if I want to use the cryptology library, then I can pull install libopenssl</code> (and on most systems, the other necessary build items in libopenssl-dev</code>) via the command line and begin developing my application. > sudo apt install openssl openssl-dev </code></pre> This is one of those things where most of the time, it just works no problems asked. It also has some nice advantages: if you are distributing your code as source, then you can build it and link it against the one installed on your system and can rely on things that are baked into the vendored version. Unfortunately, if you are using a rarer library, an older or newer version than supported, or performing cross or are relying on certain features or properties that the packager for your system has opted against it, then you might need to rely on another method anyway. Git submodules and building the library as part of the build step</h3> Another way of managing dependencies that doesn't rely on a platform or specific platform manager is to include other projects via the git submodule utility, documented here</a>. This is most useful if you aren't updating the code frequently and can mostly stick with a particular release of the library. One of the downsides is you are going to have to build the library at least once, depending on your build system. Some other libraries might have issues being built this way, but most CMake projects have made doing this a lot easier. Manual vendoring</h3> Another trick is to just take the source code of your dependency and check it in with your code, and several libraries are designed around this (e.g. Lua</a>, SQLite's amalgam</a>). If you want to add extra features or compile the code in a very particular way and don't think you will likely update the version, then this is also a pretty useful technique. However a lot of libraries will not necessary support doing this easily.

But what about implementing this for other classes?</h2>
I don't have a good answer for this, but an option may be something like Abseil's FTADLE Extension Points</a>?</p>

Pros and Cons of a copy-collector</h2>
So, now we have a copying collector. What did we gain, and what will we tearfully grieve?</p>

Where can I run C code?</h2>
C is almost always compiled (although interpreters do exist), so generally anywhere that can run native code and you have a compiler you can run C. C also can be compiled to wasm or javascript through Emscripten</a>, and thus you can run C code in the browser.</p>

Where should I ask for help: Documentation</h2>

Microsoft Development Network (MSDN)</h3>
Microsoft provides documentation for their compilers, language extensions, and libraries on MSDN</a>, and it acts like the manpages for Linux do, often as primary form of documentation.</p>

Ninja</h3>
Ninja is like a much more constrained Make, but was instead developed for meta-build systems like CMake below. It was designed to be fast to execute and build, and to not be written by hand.</p>

Code checking and formatting</h2>
Especially when writing C, it is useful to have tools that look for errors or other issues in static code as well as utilizing runtime debugging. These can be useful for automatic CI/CD checking of pull requests or code being added to master.</p>

POSIX</h3>
POSIX refers to a set of specifications made in the 90s and later updated, trying to find the minimal common set of library interfaces, tools and behavior that were in common between them. The libraries defined in POSIX are the "rest" and a quick listing is avaliable here</a>.</p>

Portability, Undefined behavior, and memory safety</h2>
Undefined behavior is one of the other major weird parts of C, and something every developer is probably going to run up against.</p>

Dependency management</h2>
Or, where are</strong> libraries exactly...?</em></p>
In what has probably become a theme now, C is old enough that it predates the sorts of environments that necessitate handling dependencies and libraries.</p>

Soft

CMake: The Good, The Bad, The Weird

An idea for C++ Mixins

Thoughts on Logging

The Alone Time

Slayer Shock

What Moves the Dead

Ashes: Afterglow

Ashes 2063 Review

Baby's second garbage collector

Garbage Collection Handbook

What I wished I knew when learning C

Horrorstor

Soft

CMake: The Good, The Bad, The Weird

The Weird</h2> In general, CMake is a "have the newest version possible installed on your system" type tool, I almost always install it through official release instead of a package manager.</li>

An idea for C++ Mixins

Thoughts on Logging

The Alone Time

Slayer Shock

Slayer Shock</h1> 3/5</p>

What Moves the Dead

Ashes: Afterglow

Ashes 2063 Review

Baby's second garbage collector

Pros and Cons of a copy-collector</h2> So, now we have a copying collector. What did we gain, and what will we tearfully grieve?</p>

Garbage Collection Handbook

The garbage collection handbook</h1> 4/5</em></p>

What I wished I knew when learning C

Where should I ask for help: Documentation</h2>

Ninja</h3> Ninja is like a much more constrained Make, but was instead developed for meta-build systems like CMake below. It was designed to be fast to execute and build, and to not be written by hand.</p>

Code checking and formatting</h2> Especially when writing C, it is useful to have tools that look for errors or other issues in static code as well as utilizing runtime debugging. These can be useful for automatic CI/CD checking of pull requests or code being added to master.</p>

Portability, Undefined behavior, and memory safety</h2> Undefined behavior is one of the other major weird parts of C, and something every developer is probably going to run up against.</p>

Dependency management</h2> Or, where are</strong> libraries exactly...?</em></p> In what has probably become a theme now, C is old enough that it predates the sorts of environments that necessitate handling dependencies and libraries.</p>

Horrorstor

Pros and Cons of a copy-collector</h2>
So, now we have a copying collector. What did we gain, and what will we tearfully grieve?</p>

Ninja</h3>
Ninja is like a much more constrained Make, but was instead developed for meta-build systems like CMake below. It was designed to be fast to execute and build, and to not be written by hand.</p>

Code checking and formatting</h2>
Especially when writing C, it is useful to have tools that look for errors or other issues in static code as well as utilizing runtime debugging. These can be useful for automatic CI/CD checking of pull requests or code being added to master.</p>

Portability, Undefined behavior, and memory safety</h2>
Undefined behavior is one of the other major weird parts of C, and something every developer is probably going to run up against.</p>

Dependency management</h2>
Or, where are</strong> libraries exactly...?</em></p>
In what has probably become a theme now, C is old enough that it predates the sorts of environments that necessitate handling dependencies and libraries.</p>