<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>linkjob-notes</title>
    <link>https://rant.li/linkjob-notes/</link>
    <description></description>
    <pubDate>Wed, 18 Nov 2026 06:45:10 +0000</pubDate>
    <item>
      <title>HyperLogLog Explained Without the Math Anxiety</title>
      <link>https://rant.li/linkjob-notes/hyperloglog-explained-without-the-math-anxiety</link>
      <description>&lt;![CDATA[Counting distinct items sounds trivial until you have a billion events per day. Storing every user ID in a set to count unique visitors costs gigabytes. HyperLogLog (HLL) does it in about 12 KB with roughly 0.8 percent error.&#xA;&#xA;The core intuition&#xA;Hash every item to a random-looking bit string. In random bits, a run of k leading zeros shows up about once every 2^k items. So if the longest run of leading zeros you have seen is 20, you have probably seen around a million distinct items. Duplicates hash to the same value, so they never change the answer. That is why HLL counts distinct items.&#xA;&#xA;Why many buckets&#xA;One &#34;longest run&#34; estimate is very noisy. HLL uses the first few bits of the hash to pick one of m buckets (for example 16,384) and tracks the max run per bucket. It then combines the buckets with a harmonic mean, which dampens outliers. More buckets means lower error: the standard error is about 1.04 / sqrt(m).&#xA;&#xA;Properties that make it useful&#xA;Mergeable: the union of two HLLs is just the per-bucket maximum. You can count daily uniques per server and merge them into weekly uniques without reprocessing raw data.&#xA;Fixed memory: size depends on m, not on how many items you add.&#xA;No deletes: you cannot remove an item. If you need that, look at other sketches.&#xA;&#xA;Where you meet it&#xA;Redis has PFADD and PFCOUNT. BigQuery, Snowflake, and Presto expose APPROXCOUNTDISTINCT. Analytics pipelines use HLL for unique users, unique IPs, and cardinality checks on columns.&#xA;&#xA;In an interview&#xA;If a system design question asks for &#34;unique viewers per video,&#34; mentioning HLL with its error bound and mergeability is a quick way to show depth. For practicing how to explain trade-offs like this clearly, see AI interview copilot (I help build Linkjob AI).]]&gt;</description>
      <content:encoded><![CDATA[<p>Counting distinct items sounds trivial until you have a billion events per day. Storing every user ID in a set to count unique visitors costs gigabytes. HyperLogLog (HLL) does it in about 12 KB with roughly 0.8 percent error.</p>

<h2 id="the-core-intuition">The core intuition</h2>

<p>Hash every item to a random-looking bit string. In random bits, a run of k leading zeros shows up about once every 2^k items. So if the longest run of leading zeros you have seen is 20, you have probably seen around a million distinct items. Duplicates hash to the same value, so they never change the answer. That is why HLL counts <em>distinct</em> items.</p>

<h2 id="why-many-buckets">Why many buckets</h2>

<p>One “longest run” estimate is very noisy. HLL uses the first few bits of the hash to pick one of m buckets (for example 16,384) and tracks the max run per bucket. It then combines the buckets with a harmonic mean, which dampens outliers. More buckets means lower error: the standard error is about 1.04 / sqrt(m).</p>

<h2 id="properties-that-make-it-useful">Properties that make it useful</h2>
<ul><li><strong>Mergeable</strong>: the union of two HLLs is just the per-bucket maximum. You can count daily uniques per server and merge them into weekly uniques without reprocessing raw data.</li>
<li><strong>Fixed memory</strong>: size depends on m, not on how many items you add.</li>
<li><strong>No deletes</strong>: you cannot remove an item. If you need that, look at other sketches.</li></ul>

<h2 id="where-you-meet-it">Where you meet it</h2>

<p>Redis has PFADD and PFCOUNT. BigQuery, Snowflake, and Presto expose APPROX<em>COUNT</em>DISTINCT. Analytics pipelines use HLL for unique users, unique IPs, and cardinality checks on columns.</p>

<h2 id="in-an-interview">In an interview</h2>

<p>If a system design question asks for “unique viewers per video,” mentioning HLL with its error bound and mergeability is a quick way to show depth. For practicing how to explain trade-offs like this clearly, see <a href="https://www.linkjob.ai/" rel="nofollow">AI interview copilot</a> (I help build Linkjob AI).</p>
]]></content:encoded>
      <guid>https://rant.li/linkjob-notes/hyperloglog-explained-without-the-math-anxiety</guid>
      <pubDate>Fri, 13 Nov 2026 13:27:25 +0000</pubDate>
    </item>
  </channel>
</rss>