DEV Community: Mike Fiedler

Visual Audit Homebrew Programs

Mike Fiedler — Sun, 10 May 2020 15:04:42 +0000

I use a Mac computer as my primary development machine. My personal one is where I spend a lot of time experimenting, self-learning, trying things out.
One awesome way to install software on macOS is to use Homebrew, which boils down the process of downloading, compiling, installing software (and any associated dependencies) to a simpler process, often with a single command, e.g. brew install httpie to get the HTTPie command line client.

However, since it's so easy to install a variety of packages and programs, oftentimes I'll have installed a program to try something out, and never need it again. The total count of packages installed via Homebrew can be seen like this:

$ brew list | wc -l
325

I definitely don't recall installing all of these - and over time that becomes a heavier burden to carry. Each package can use up some disk space, but more importantly, every time a brew upgrade is run, it may install new versions of software you don't use, so that takes up more time - which is even more valuable than disk space. If you're running out of disk space, read:

Seriously, periodically do brew cleanup.

Collapsed PLUG ・ Nov 13 '18 ・ 1 min read

#homebrew

But since I want to actually figure out if there's programs I've installed and no longer use or need, I could read the output of brew list and brew uninstall <package name> for each one, but I'm bound to overlook some packages that are dependencies, and may no longer be needed.

I find it best to read visual "maps" that diagram the relationships between packages, so that if I uninstall a package that as a dependency on another package, and nobody else depends on that package, that I uninstall them both, thereby freeing up more resources.

A nice helper program for Homebrew is this extension called brew-graph:

martido / homebrew-graph

Creates a simple dependency graph of Homebrew formulae.

Attention: This repository has been renamed from brew-graph to homebrew-graph to adhere to the Homebrew naming conventions of tap repositories Please update your local clones or forks (for consistency only, GitHub makes sure everything still works for you):

git remote set-url origin https://github.com/martido/homebrew-graph

brew-graph

brew-graph is a Ruby script that creates a dependency graph of Homebrew formulae. The currently supported output options are DOT and GraphML.

In general, if you'd like to know more about Untangling Your Homebrew Dependencies, check out the blog post by Jonathan Palardy.

Installation

brew tap martido/homebrew-graph

Note: If you already have the brew-graph formula installed from the old tap repository, uninstall it first:

brew uninstall brew-graph
brew untap martido/homebrew-brew-graph

Usage

Type brew graph --help.

brew graph [options] formula1 formula2 ... | --installed | --all
Create a dependency graph of Homebrew formulae.

Options:

 -h, --help            Print this help message.
 -f, --format

…

View on GitHub

Once installed (read the README), I can create an image, along with the graphviz package's dot command:

brew graph --installed | dot -Tsvg -o homebrew-installed.svg

I could replace the svg part with png to get a static image - I like the SVG file format, since it opens in Google Chrome and I can run a "Find" function for any names, and could also change some of the variables if I wanted to via text editor, or with an SVG Editor

The image output is similar to the one in the header of this post - some items are isolated and appear at the top row - these packages have no dependencies, and nothing depends on them - often a tool to do a specific thing. Audit these for tools you use directly, and uninstall those that you don't recall using. You can always reinstall them later!

Now for the more complex ones - like ffmpeg for example. This package has a lot of direct dependencies. So if I wanted to create an image that is scoped only to that package, I could run:

brew graph ffmpeg | dot -Tpng -o homebrew-ffmpeg.png

And that looks like:

Which isn't easy to see, but it can be helpful to see the dependencies of a single package.
So if I wanted to uninstall all of the packages safely, I could run each one and if I get a warning that the package is in use, not remove it, or I could use another helpful external command - rmtree:

beeftornado / homebrew-rmtree

Remove a formula and its unused dependencies

homebrew-rmtree

Remove a formula and its unused dependencies

What is it?

It's an external command for Homebrew that provides a new command, rmtree that will uninstall that formula, and uninstall any of its dependencies that have no formula left installed that depend on them. The command will check all dependencies recursively starting at the one specified on the command line.

This is tricky business. So this command comes with a warning.

Warning

There are formulae that do not specify all of their dependencies. This means that it is possible that this command will remove something you still need or won't remove something you no longer want. Generally, it is pretty good Until someone comes up with a clever way around this, you need to be careful what you uninstall A formula could also depend on something you want to keep around, while nothing else actually depends on it (except…

View on GitHub

This provides a useful --dry-run parameter to observe what might happen if we wanted to remove a package, and see for ourselves what we would be removing. Example:

$ brew rmtree --dry-run ffmpeg
This is a dry-run, nothing will be deleted
==> Examining installed formulae required by ffmpeg...
 -  61 / 62

Can safely be removed
----------------------
ffmpeg
aom
lame
libbluray
...

This goes on for a while, so run it yourself and see what your computer can safely remove!

Reminder: As always, take these notes with caution. The risk of installing someone else's software on systems that may lead to undue exposure. This post and other software are things lot of us write, but there's nothing preventing malicious actors from doing nasty things over similar channels, so make sure you have other protections in place - like firewalls, malware/virus scanners, network monitoring/alerting, etc - and if it's open source code, you can read it too, and form your own opinions!

Hopefully this post has provided you with a couple of tools in your toolbox to help keep your computer in good shape, extending its lifespan and utility to keep learning and trying out new things!

Python to JSON and Back

Mike Fiedler — Fri, 24 Apr 2020 00:33:36 +0000

Photo by David Clode on Unsplash

As JavaScript Object Notation (JSON) has become the ubiquitous language of communication between APIs on the web, being able to take Python objects and serialize them to JSON strings is an important part of communicating with other services, clients, APIs, what have you.

The default Python json module supports both dump and dumps methods - the more common one used is dumps - dumping the object to a JSON String - all based on this Conversion Table - which pointedly omits the datetime objects - most importantly because the JSON Specification has no reference to any specific date/time format.
This is left up to the implementer, based on what format they need - leaving coders everywhere figuring out how to Do The Right Thing - since any date/time mismatches have potential to cause all manner of havoc if incorrectly parsed. If you want to read more on time, check out this collection of articles.

A simple demonstration of the failure condition looks like this:

>>> import datetime
>>> import json
>>> json.dumps(datetime.datetime.utcnow())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File ".../lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File ".../lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File ".../3.6.8/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'datetime' is not JSON serializable

And if you've hit this, and search the internet, you'll find all manner of approaches to resolve - a common approach seen is implementing a function that iterates through an object, and if it finds a datetime value, convert it to the ISO 8601 format, with datetime.isoformat(). Example of the conversion step:

>>> json.dumps(datetime.datetime.utcnow().isoformat())
'"2020-04-23T23:36:36.620619"'

However this then becomes custom code to iterate through objects, and sometimes we may need to iterate through deeply nested objects - for example a blog page, its posts and comments in a single JSON payload may all have timestamps that would need to be iterated through, detected, converted, and then json.dumps() the whole thing.

Some popular frameworks may handle this for you automatically - notable callouts are the Django REST Framework DateTimeField will default to ISO8601, as will the excellent pydantic library (here's the implementation under the hood).

However, oftentimes we may not want to take on importing a framework or a full-fledged library, but still want to avoid having to create custom solutions like this - and that's where we can lean on well-trodden paths from library maintainers, like the rapidjson library - and more importantly, the Python wrapper to this fast C library.

`rapidjson`

rapidjson aims to be a swap-in replacement for the native Python json module, and allows the coder to extend the behavior to support more behaviors with less effort. Note: This package must be installed via the python-rapidjson package name.

Keep in mind: there are a few incompatibilities you should be aware of when using rapidjson - but most implementations may not be using these, in fact the coercion of "true" strings to Python True values is actually an improvement!

So if we were to run our example from above, replacing json with rapidjson, we get the same result, albeit the traceback is a little clearer now:

>>> import datetime
>>> import rapidjson
>>> rapidjson.dumps(datetime.datetime.utcnow())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: datetime.datetime(2020, 4, 24, 0, 11, 16, 105719) is not JSON serializable

However now we can pass an argument to rapidjson.dumps() to inform it how we want to handle datetime instances, like so:

>>> rapidjson.dumps(datetime.datetime.utcnow(), datetime_mode=rapidjson.DM_ISO8601)
'"2020-04-24T00:14:35.531078"'

Voilá, we've got a valid ISO8601-formatted string of a datetime with little added!

I've put together a more complete example of full object deserialization and loading below - feel free to run, fork, extend, experiment with the code, and read more about the other datetime_mode options in rapidjson.

Enjoy!