You Can Just Say No to the Data - Jim Nielsen’s Blog
Appealing to data as the ultimate authority — especially when fueled by engineered desire — isn’t neutrality, it’s an abdication of responsibility.
Appealing to data as the ultimate authority — especially when fueled by engineered desire — isn’t neutrality, it’s an abdication of responsibility.
Matthias responds to my pondering about the point of “likes” and “shares”:
I like to think of Webmentions not as a measure of popularity. To me, they measure connection. Connection to individual people and connection to the community as a whole. Webmentions let you listen into the constant noise out there and, just like a radio telescope, pick up scarcely audible echoes of connection.
This describes how I iterate on The Session:
It comes down to this annoying, upsetting, stupid fact: the only way to build a great product is to use it every day, to stare at it, to hold it in your hands to feel its lumps. The data and customers will lie to you but the product never will.
This whole post reminded of the episode of the Clearleft podcast on measuring design.
The problem underlying all this is that when it comes to building a product, all data is garbage, a lie, or measuring the wrong thing. Folks will be obsessed with clicks and charts and NPS scores—the NFTs of product management—and in this sea of noise they believe they can see the product clearly. There are courses and books and talks all about measuring happiness and growth—surveys! surveys! surveys!—with everyone in the field believing that they’ve built a science when they’ve really built a cult.
The design process in action in Victorian England:
Recognizing that few people actually read statistical tables, Nightingale and her team designed graphics to attract attention and engage readers in ways that other media could not. Their diagram designs evolved over two batches of publications, giving them opportunities to react to the efforts of other parties also jockeying for influence. These competitors buried stuffy graphic analysis inside thick books. In contrast, Nightingale packaged her charts in attractive slim folios, integrating diagrams with witty prose. Her charts were accessible and punchy. Instead of building complex arguments that required heavy work from the audience, she focused her narrative lens on specific claims. It was more than data visualization—it was data storytelling.
One of my favourite episodes of the Clearleft podcast is on measuring design. This post from Chris is a complements that episode in a sensible and practical style.
What gets measured gets done. You are what you measure. Measurement eliminates argument. If you work in an environment that puts store in these oft-quoted business adages then I urge you to take a moment to challenge your calculations. Let’s review our metrics to ensure they can stand up and be counted.
A thoroughly researched look at what our baseline criteria should be for making websites today:
The baseline for web development in 2022 is:
- low-spec Android devices in terms of performance,
- Safari from two years before in terms of Web Standards,
- and 4G in terms of networks.
The web in general is not answering those needs properly, especially in terms of performance where factors such as an over-dependence on JavaScript are hindering our sites’ performance.
It’s somewhat damning to Safari to see it as a baseline browser, but with Internet Explorer out of the picture, something has to be the lowest common denominator. In this case, Safari is quite literally the new IE.
We’ve got click rates, impressions, conversion rates, open rates, ROAS, pageviews, bounces rates, ROI, CPM, CPC, impression share, average position, sessions, channels, landing pages, KPI after never ending KPI.
That’d be fine if all this shit meant something and we knew how to interpret it. But it doesn’t and we don’t.
The reality is much simpler, and therefore much more complex. Most of us don’t understand how data is collected, how these mechanisms work and most importantly where and how they don’t work.
Even if you can somehow justify using tracking technologies (which don’t work reliably) to make general, statistical decisions (“fewer people open our emails when the subject contains the word ‘overdraft’!”), you can’t make individual decisions based on them. That’s just wrong.
Prompted by my post on tracking, Chris does some soul searching about his own use of tracking.
I’m interested not just in the ethical concerns and my long-time complacency with industry norms, but also as someone who very literally sells advertising.
He brings up the point that advertisers expect to know how many people opened a particular email and how many people clicked on a particular link. I’m sure that’s right, but it’s also beside the point: what matters is how the receiver of the email feels about having that information tracked. If they haven’t given you permission to do it, you can’t just assume they’re okay with it.
Well, this is just wonderful! Jim has written copious notes after listening to my favourite episode of season three of the Clearleft podcast, measuring design:
I’m going to have to try really, really hard to not just copy/paste the entire transcript of this podcast. It‘s that good. Don’t miss it.
A good post by Andy on “the language of business,” which is most cases turns out to be numbers, numbers, numbers.
While it seems reasonable and fair to expect a modicum of self-awareness of why you’re employed and what business value you drive in the the context of the work you do, sometimes the incessant self-flagellation required to justify and explain this to those who hired you may be a clue to a much deeper and more troubling question at the heart of the organisation you work for.
This pairs nicely with the Clearleft podcast episode on measuring design.
I’ve noticed a trend in recent years—a trend that I’ve admittedly been part of myself—where performance-minded developers will rebuild a site and then post a screenshot of their Lighthouse score on social media to show off how fast it is.
Mea culpa! I should post my CrUX reports too.
But I’m going to respectfully decline Phil’s advice to use any of the RUM analytics providers he recommends that require me to put another script element on my site. One third-party script is one third-party script too many.
An oldie but a goodie. If you think you’re getting statistically significant results from A/B testing, you should probably consider doing some A/A testing.
In an A/A test, you run a test using the exact same options for both “variants” in your test.
The typography of horology.
I hadn’t come across this before—run Lighthouse tests on your pages from six different locations around the world at once.
This is very handy indeed! Quick one-line JavaScript helpers categorised by type.
And, no, you don’t need to npm install any of these. Try “vendoring” them instead (that’s copying and pasting to you and me).
I spent most of the weekend reading through this and I’ve still barely scratched the surface—a lot of work has gone to the analyses and write-ups!
The sections on accessibility and performance get grimmer each year but the raw numbers on framework adaption are refreshingly perspective-setting.
Goodhart’s Law applied to Google’s core web vitals:
If developers start to focus solely on Core Web Vitals because it is important for SEO, then some folks will undoubtedly try to game the system.
Personally, my beef with core web vitals is that they introduce even more uneccessary initialisms (see, for example, Harry’s recent post where he uses CWV metrics like LCP, FID, and CLS—alongside TTFB and SI—to look at PLPs, PDPs, and SRPs. I mean, WTF?).
I must admit I’ve been wincing a little every time I see a graph with a logarithmic scale in a news article about COVID-19. It takes quite a bit of cognitive work to translate to a linear scale and get the real story.