Aditya Thebe Personal Blog RSS Feed

Bypassing DRM protection to download a movie

Sat, 24 May 2025 19:00:00 GMT

In this article, I'll walk you through the process of how I managed to reconstruct a complete video file from encrypted video stream segments. At the end of the article, you'll also find the source code for the script that I used to download the video. I recently bought a movie from a platform that was very restrictive. You could only watch the movie for a very limited period of time and from a very limited number of devices. I'm a media hoarder and naturally I wanted to download the movie for later. However, downloading it was not trivial as you'll see in a bit. I couldn't resist the temptation to roll up my sleeves and reverse-engineer the stream. With a few hours of investigation and scripting, I was able to bypass the protection and piece together the entire movie using & . Needless to say, the movie was not distributed and is only being used personally. The usual approach Generally, whenever I need to download a video, I grab the URL and pass that on to my trusty tool. For other websites, I open the developer tools, go to the network tab, filter by media, and then grab the video stream URL. But this time, the movie didn't have a single URL. In fact, there were no network calls showing up under the media category at all. Reconnaissance What I noticed is that there were a lot of HTTP calls to different segments of the video. This endpoint returned a binary data of content type - which is just a stream of digital media. I tried downloading one of the segments and playing it with VLC but that didn't work. In between those calls, there were also calls to get some sort of key This endpoint also returned binary data. The name in the url was quite interesting. Upon googling, I found out that drm stands for Digital Rights Management and it relies on encryption to protect the content. At this point I was almost certain that the endpoint was returning a small encrypted segment of the video and the endpoint was returning the encryption key for the web client to decrypt it. If I could figure out the encryption algorithm used then I could download all the segments and the keys and then decrypt them one by one. Encryption algorithm The endpoint returned a 128-bit encryption key (e.g. 9036fb5d2f6af3c153acecd37bde6da7 in hexadecimal). That's a strong signal that it's most likely using AES-128 - a very popular symmetric encryption algorithm. With the ciphertext (from ) and the encryption key in hand, I tried decrypting it using However, when I tried to play the resulting file in VLC, it didn’t work – nothing happened. I went back to examine the network traffic and found another intriguing endpoint: It returned a plain-text response like this M3U8 Now that's something! It's a file which contains a list of urls to all the segments of the video file. There are some pretty interesting stuff in here. Let's go through them | Line | Directive | Description | | ---- | -------------------------- | ------------------------------------------------------------- | | 2 | | uses the extended version 3 of that supports encryption | | 5 | | Indicates that each url points to a video | | 6 | | we see the encryption algorithm used - AES-128 | | 7 | | Indicates that this segment is 4 seconds long | In the directive, we can also see . That most likely represents the initialization vector of the AES-128 encryption. Let's try decrypting once again using the CBC mode And voila! I was able to play the video. Code Now all I needed to do was write a script to download all the segments and their keys and then decrypt them individually. And finally piece them together with . You can find the source code at github.com/adityathebe/drm. -- In hindsight this was pretty straightforward. The endpoints hinted the use of DRM and it turns out most modern DRM technologies use AES with 128-bit keys [1]. To validate the approach, I purchased another movie and successfully downloaded the entire film. References

🕰️ Now

🏠 In Kathmandu
Working at @Flanksource

Go v1.21+ Auto-Downloads Toolchains and Verifies Them Using a Transparency Log

Sun, 07 Dec 2025 00:00:00 GMT

Before Go 1.21, if you wanted to build a project requiring a newer Go version, you had to manually install that version yourself. Since 1.21, go automatically downloads the required toolchain (the go binary, compiler, linker, etc.) and uses it to build the project - all without touching your global Go installation.

While very convenient, there's also a huge security concern in downloading a binary from the internet and running it on demand, completely transparently. That's why the downloaded toolchains are locally verified against a transparency log.

Transparency logs are append-only logs that enable public auditability. They use Merkle trees - a highly optimized data structure for fast consistency verification.

Go has been using a module checksum db (Go sumdb) - a transparency log - which stores the checksum for all the go modules (that the mirror sees). The same checksum db has been updated to support the toolchain checksum.

Who Audits Go’s Transparency Log?

It's reassuring that Go verifies the downloaded binaries for the toolchain, but who verifies that the same transparency log is being distributed to all the users?

Let me break this down:

A Go client downloads a toolchain and locally verifies that the toolchain's checksum is present in the transparency log.
A different client could download a different toolchain, and Google could serve a completely different transparency log that's internally consistent with that toolchain.

So two users could have two different toolchains, yet both believe they have the correct one. This is called a split-view attack.

The solution is to audit that google serves the same transparency log to all users. This is where auditors come in.

Andrew Ayer in his blog post - https://www.agwa.name/blog/post/verifying_go_reproducible_builds - mentions that he has been running an independent auditor for Go's checksum database since 2020 - https://sourcespotter.com/

Sourcespotter independently builds the toolchain for every new release and verifies that against the Go sumdb.

Verifying Reproducibility vs Consistency

It's important to distinguish what exactly the auditor is doing.

For regular Go modules, the auditor only verifies consistency. It doesn't download every module, build it, and check that the checksum matches. That would be impossible at scale since there are millions of modules. Instead, it just watches the log over time, making sure Google doesn't alter old entries or serve different versions to different people.

I don't understand how this works exactly. A naive solution would be

store_checksum_for_all_modules_to_db()

loop:
  for module in fetch_checksum_for_all_modules_from_sumdb():
    checksum_in_db = db.get_module_checksum(module.path)
    if not checksum_in_db:
      store_to_db(module)
      continue

    if checksum_in_db != module.checksum:
      alert("mismatch")

Of course, it doesn't work this way. In fact, the auditor stores a single root hash (not millions for each module) and can mathemtically verify that the tree hasn't been tampered - I just don't know how.

For toolchains however, the auditor also verifies reproducibility. Since there are relatively few toolchain releases (a few thousand across all versions and platforms), Source Spotter can actually download the source code, build each toolchain independently, and confirm the resulting binary matches the checksum Google published. This ensures the published binaries really are what you'd get if you built from source yourself.

Why is Golang's atomic.Bool 32 bits (not 8 bits)

Thu, 06 Nov 2025 00:00:00 GMT

TIL that Go uses uint32 instead of uint8 or just bool. At first glance, that seems counterintuitive because ideally a bool is just a bit. Why do we need the extra 31 bits?

The runtime picks a 32-bit slot so it can rely on the same word-sized compare-and-swap instruction on every architecture Go supports. Some CPUs expose byte-sized atomics, but others do not guarantee them (or require strict alignment), so Go standardizes on word-aligned storage instead.

// A Bool is an atomic boolean value.
// The zero value is false.
//
// Bool must not be copied after first use.
type Bool struct {
	_ noCopy
	v uint32
}

Let's see how CompareAndSwap works for a bool

// CompareAndSwap executes the compare-and-swap operation for the boolean value x.
func (x *Bool) CompareAndSwap(old, new bool) (swapped bool) {
	return CompareAndSwapUint32(&x.v, b32(old), b32(new))
}

Source: https://github.com/golang/go/blob/47a63a331daa96de55562fbe0fa0201757c7d155/src/sync/atomic/type.go#L27C1-L30C2

CompareAndSwapUint32 is a Go-level wrapper around a lower-level atomic CAS operation in the runtime

This function is implemented in assembly.

TEXT ·CompareAndSwapUint32(SB),NOSPLIT,$0
	JMP	internal∕runtime∕atomic·Cas(SB)

On amd64, bool swap is implemented as shown below in assembly.

// bool Cas(int32 *val, int32 old, int32 new)
// Atomically:
// if(*val == old){
//     *val = new;
//     return 1;
// } else
//     return 0;
TEXT runtime∕internal∕atomic·Cas(SB),NOSPLIT,$0-17
    MOVQ ptr+0(FP), BX    // Load pointer address into BX
    MOVL old+8(FP), AX    // Load expected value into AX (required by CMPXCHG)
    MOVL new+12(FP), CX   // Load new value into CX
    LOCK                   // Lock prefix for atomicity
    CMPXCHGL CX, 0(BX)    // Compare AX with [BX], if equal swap with CX
    SETEQ ret+16(FP)      // Set return value based on ZF (zero flag)
    RET

Source: https://github.com/golang/go/blob/47a63a331daa96de55562fbe0fa0201757c7d155/src/internal/runtime/atomic/atomic_amd64.s#L22-L37

Backup your GitHub repositories now!

Sun, 19 Oct 2025 00:00:00 GMT

Over the past few days, I've been seeing a surge of posts on Twitter regarding GitHub account suspension.

https://x.com/grimcodes/status/1977471153011933375

https://x.com/ThePrimeagen/status/1978261656573501510

This isn't new. In February 2024, even GitHub's co-founder and ex-CEO, Chris Wanstrath, had his account suspended. Of course, GitHub immediately responded and removed the suspension for him but the rest of us may not be so fortunate.

Here's a bunch of other such incidents

A student lost access to their bachelor's thesis and work repositories for 2 months due to GitHub Education verification issues
Developers waiting up to 120 days for appeal responses
Account suspended when trying out Codespaces and no response for over a month
Account suspended when submitting multiple bug reports

GitHub's automated systems can flag accounts without warning, and appeals can take months. The solution? Keep your own backups.

Backup with gickup

I'm using gickup to backup my repositories hosted on GitHub. I run it on Kubernetes as a CronJob, but you can also use gickup's built-in cron runner.

I schedule daily backups, which is sufficient since actively developed repos are typically already cloned on my machine.

I'm currently backing up to the local filesystem, which is backed up separately but gickup also offers backing up to various remotes like - GitLab, Gitea, Gogs, OneDev, Sourcehut, S3.

Disclaimer: This post is not sponsored or affiliated with gickup in any way. I'm sharing my personal backup solution that works for me. See the "Alternative Tools" section at the end for other options.

Setup

Create a GitHub Personal Access Token with repo scope - https://github.com/settings/personal-access-tokens
Configure - create a conf.yaml with your GitHub username and backup destination

Here's my conf.yaml

source:
  github:
    - token: ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      user: adityathebe
      exclude:
        - adityathebe/DefinitelyTyped # too Large, idc
        - DefinitelyTyped

destination:
  local:
    - path: '/data/github-backups'
      structured: true
      mirror: true
      lfs: false # setting this true was 100x slower

Instead of providing the github token on the config file, you can also set the GITHUB_TOKEN env var.

Key points:

Source: All my GitHub repositories (excluding a few large ones)
Mirror mode: This backs up all git refs (branches, tags, and pull request refs)
Structured: Creates organized folders for each repository

Once configured, it's completely hands-off.

Restoration

git clone gickup/github-backups/github.com/adityathebe/nvim.git

Cloning into 'nvim'...
done.
admin@truenas[/mnt/mega/aditya/backups]$

Alternative Tools

While I use gickup, there are several other tools available for backing up GitHub repositories:

git-backup - A simple tool for backing up git repositories
Gitea mirror feature - Use Gitea's built-in mirroring capabilities to backup repositories
GitHub-Backup - Another backup solution for GitHub repositories

Note: I haven't personally tested these alternatives, but they may better suit your specific needs.

Delete all files in a Cloudflare R2 bucket using rclone

Wed, 15 Oct 2025 00:00:00 GMT

TLDR

This post explains how to delete all files in a Cloudflare R2 bucket using rclone instead of the web UI or AWS CLI.

Quick steps:

Get credentials: Create an Account API Token in R2 dashboard with "Object Read & Write" permission
Configure rclone: Add R2 credentials to ~/.config/rclone/rclone.conf with type=s3, provider=Cloudflare
Delete files: Run rclone delete r2:/bucket-name/ -P

Object storage services do not allow deleting a bucket without deleting all the files in it first. R2's web UI doesn't have a "Select All" button that would make deletion a simple 2-click process. If you have tons and tons of files, the web UI becomes impractical for bucket cleanup.

Cloudflare suggests using the AWS CLI, but I prefer rclone (https://rclone.org/) because it provides a unified interface for many cloud services - not just object storage like MinIO, S3, and Backblaze B2, but also services like Google Photos and Google Drive. For example, I've used rclone to upload photos to Google Photos and download files from Google Drive.

rclone has a massive list of services that it supports. See here - https://rclone.org/#providers. Install rclone (guide: https://rclone.org/install/) and let's begin by creating an access key for your bucket.

Get the Access key

On the web UI, go to R2's dashboard and on the top right you should see "Manage API tokens" button.
Create an Account API Token (not the User API Token). You need to select "Object Read & Write" permission with the correct bucket.
On the next page, you should see the Access Key ID and the Secret Access Key. Keep note of them and also the endpoint - we'll need them in a while.

Setup rclone

You can either use the setup wizard using the rclone config command or just use this template and place it in ~/.config/rclone/rclone.conf.

[r2]
type = s3
provider = Cloudflare
access_key_id = 6b82af45b9a94e281cd5ba7b111b68fe
secret_access_key = 61f58485d481ad5980635c388e5740f9a8abf94ab5b5acc442c1effb08a081b6
endpoint = https://5bdd9766a7ad14ae145db477f519gs21.r2.cloudflarestorage.com

Verify that the config was created properly.

rclone listremotes --long

r2:                  s3

Emptying the bucket

My bucket is named longhorn so this is the command I ran. The -P flag shows the progress.

rclone delete r2:/longhorn/ -P

Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                29 / 37, 78%, Listed 100
Deleted:               37 (files), 0 (dirs), 41.254 MiB (freed)
Elapsed time:         1.5s
Checking:
 * backups/backupstore/vo…c759acc982e2aff860.blk: deleting
 * backups/backupstore/vo…f23cab54f4acad0772.blk: deleting
 * backups/backupstore/vo…f27e50d08a22c62ab5.blk: deleting
 * backups/backupstore/vo…e80924cddebd00517b.blk: deleting
 * backups/backupstore/vo…92910deb42c89ff38f.blk: deleting
 * backups/backupstore/vo…6262258659adcdee58.blk: deleting
 * backups/backupstore/vo…f3ebd0010bfda1e995.blk: deleting
 * backups/backupstore/vo…606dd94efb994185b1.blk: deleting

Fix invalid version: unknown revision from go get

Mon, 13 Oct 2025 00:00:00 GMT

go get by default talks to a proxy server - https://proxy.golang.org. The proxy server then fetches the module, caches it and returns it to you.

If you request a version before it's published, the proxy gets a 404 and caches that negative response. You'll then encounter an error like this:

go get github.com/flanksource/duty@v1.0.1111

go: github.com/flanksource/duty@v1.0.1111: reading github.com/flanksource/duty/go.mod at revision v1.0.1111: unknown revision v1.0.1111

I've run into this issue even when running go get immediately after a new release. According to the proxy documentation, new versions may not show up immediately but should be available within one minute:

"In order to improve our services' caching and serving latencies, new versions may not show up right away ... The new version should be available within one minute..."

- https://proxy.golang.org/

This poisoned cache can persist even after the version is published, which is why you may still get 404 errors for up to 30 minutes even after a release.

"it may take up to 30 minutes for the mirror's cache to expire and fresh data about the version to become available"

- https://proxy.golang.org/

To understand what's happening behind the scenes, use the -x flag to see which servers go get contacts:

go get -x github.com/labstack/echo

# get https://proxy.golang.org/github.com/@v/list
# get https://proxy.golang.org/github.com/labstack/@v/list
# get https://proxy.golang.org/github.com/labstack/echo/@v/list
# get https://proxy.golang.org/github.com/labstack/echo/@v/list: 200 OK (0.914s)
# get https://proxy.golang.org/github.com/labstack/@v/list: 404 Not Found (0.940s)
# get https://proxy.golang.org/github.com/@v/list: 404 Not Found (0.940s)
# get https://proxy.golang.org/golang.org/x/crypto/@v/list
# get https://proxy.golang.org/github.com/labstack/gommon/@v/list

As you can see, every request goes through the proxy.

The Solution

To bypass the poisoned cache, you can override the default proxy using the GOPROXY environment variable. Setting it to direct skips all proxy servers and fetches directly from the module repository.

GOPROXY=direct go get -x github.com/labstack/echo

mkdir -p /Users/aditya/go/pkg/mod/cache/vcs # git3 https://github.com/labstack/echo
# lock /Users/aditya/go/pkg/mod/cache/vcs/84bf749277ea54683052733eb9a9f7ea7f43e18090bed6ec5b8281cdee0a1f11.lock
mkdir -p /Users/aditya/go/pkg/mod/cache/vcs/84bf749277ea54683052733eb9a9f7ea7f43e18090bed6ec5b8281cdee0a1f11 # git3 https://github.com/labstack/echo
cd /Users/aditya/go/pkg/mod/cache/vcs/84bf749277ea54683052733eb9a9f7ea7f43e18090bed6ec5b8281cdee0a1f11; git ls-remote -q https://github.com/labstack/echo
1.158s # cd /Users/aditya/go/pkg/mod/cache/vcs/84bf749277ea54683052733eb9a9f7ea7f43e18090bed6ec5b8281cdee0a1f11; git ls-remote -q https://github.com/labstack/echo
cd /Users/aditya/go/pkg/mod/cache/vcs/84bf749277ea54683052733eb9a9f7ea7f43e18090bed6ec5b8281cdee0a1f11; git init --bare

Now go get clones the repository directly using Git, completely bypassing the proxy cache and fetching the latest version from the source.

References

Git Objects

Sun, 12 Oct 2025 15:45:00 GMT

I have been using Git for almost a decade now and I'm ashamed to admit that it was only today that I decided to get into the internals of Git. I feel like I've tried doing this before, but there just wasn't enough motivation at the time.

But today was the perfect day for it because as it just so happens I lost one of my uncommitted files due to git reset --hard. It was one file - a new weekly journal I was about to publish that had notes I'd written throughout the week. I know I had read somewhere that git still keeps file around so there's a possibility I could recover it.

I asked OpenAI's Codex to recover it. It ran a whole bunch of git commands and finally reported that it was impossible to recover. My file was permanently gone! Then, I asked Claude Code to do it. Interestingly, it was able to recover my files after running a few git commands.

I went through the commands that Claude ran to understand how it was able to do so, and one thing led to another, and I was already deep into Git documentation and learning about the four fundamental objects in git.

Since git-scm.com already does a fantastic job of explaining what the fundamental objects are, I'll focus this post on the hands-on experience of actually viewing and exploring them. I'll also not be talking about the file recovery process - that's for another time.

Git Command Categories

Before jumping right into the git's objects internals, I think it helps to understand the git command categories

Source: https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain

As a precursor, I think it's helpful to understand that git classifies the commands as

Porcelain: User facing, polished and highlevel
Plumbing: Low level. End users generally do not use it.

As far as I know, it's a Git specific taxonomy.

"because Git was initially a toolkit for a version control system rather than a full user-friendly VCS, it has a number of subcommands that do low-level work and were designed to be chained together UNIX-style or called from scripts. These commands are generally referred to as Git’s “plumbing” commands, while the more user-friendly commands are called “porcelain” commands."

Git Objects

Git has 4 fundamental objects - blob, tree, commit, and tag.

Let's create an empty repo and explore how git stores these objects and what they are.

mkdir my-project
cd my-project
git init

Initialized empty Git repository in /Users/aditya/my-project/.git/

Git stores everything about the repository in a .git directory. Let's take a peek.

tree .git

.git
├── config
├── description
├── HEAD
├── hooks
│   ├── applypatch-msg.sample
│   ├── commit-msg.sample
│   ├── fsmonitor-watchman.sample
│   ├── post-update.sample
│   ├── pre-applypatch.sample
│   ├── pre-commit.sample
│   ├── pre-merge-commit.sample
│   ├── pre-push.sample
│   ├── pre-rebase.sample
│   ├── pre-receive.sample
│   ├── prepare-commit-msg.sample
│   ├── push-to-checkout.sample
│   └── update.sample
├── info
│   └── exclude
├── objects
│   ├── info
│   └── pack
└── refs
    ├── heads
    └── tags

9 directories, 17 files

By default, git creates a few place holder directories even on an empty repository. The .git/objects dir comes with 2 empty directories info and pack but let's ignore them for now.

Let's create a new file and make a commit to have some objects we can explore.

echo "hello world" > first.txt
git add .
git commit -m "my first commit"

[main (root-commit) 8dae705] my first commit
 1 file changed, 1 insertion(+)
 create mode 100644 first.txt

Commit

We have a commit with hash 8dae705. We'll be using the plumbing command git cat-file to view the objects. Let's start with the commit

git cat-file -p 8dae705

tree 2c06b36b1c3de1c0b0de9897b23c70bf835dc190
author Aditya Thebe <contact@adityathebe.com> 1760258223 +0545
committer Aditya Thebe <contact@adityathebe.com> 1760258223 +0545
gpgsig -----BEGIN SSH SIGNATURE-----
 U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgOzmsUfKA9Ly4Ir4snhc6l6dWsa
 TplTM0tvrD14WJQjwAAAADZ2l0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5
 AAAAQMO7YaDnbTdHKlp8Y42wpwAJM+EMzVRJoW5HeNU+Rc4HjvXAr+fH1vYxmIDkRJZGSA
 vgSRhG3q6QsAxzbXS/dQo=
 -----END SSH SIGNATURE-----

my first commit

A commit object is just a pointer to a tree object with some additional metadata like the author, commit time, signatures, commit message, and so on. In this case, the commit points to the tree 2c06b36b1c3de1c0b0de9897b23c70bf835dc190 (see the first line of the output)

If you notice, we use the -p flag which pretty prints the contents of the object. As you'll see in a while, git cat-file can print objects of all types. That should tell you that all objects have a hash (not just commits).

Try out these other flags...

git cat-file -t 8dae705 # show the object type

commit

git cat-file -s 8dae705 # show the object size in bytes

Tree:

Since we have the tree hash, let's take a peek into it as well

git cat-file -p 2c06b36b1c3de1c0b0de9897b23c70bf835dc190

100644 blob 3b18e512dba79e4c8300dd08aeb37f8e728b8dad    first.txt

git cat-file -t 2c06b36b1c3de1c0b0de9897b23c70bf835dc190

tree

The output lists all the files in the project at the time of commit. That's what a tree basically is - a snapshot of the directory listing of the project.

Let's create a few more files and see how the tree appears for them

echo "another file" > second.txt
mkdir "src"
echo "console.log('hello world')" > src/index.js

This is what the project should look like at this stage

tree

.
├── first.txt
├── second.txt
└── src
    └── index.js

Let's commit this.

git add . && git commit -m "second commit"

[main c5fa6f4] second commit
 2 files changed, 2 insertions(+)
 create mode 100644 second.txt
 create mode 100644 src/index.js

Let's get the tree hash for this new commit

git cat-file -p c5fa6f4

tree 8b5a93d517fa8c8d9825f3c711f7cef02310ad91
parent 8dae70599ee87f208e86ad4ff65e9b05bf777630
author Aditya Thebe <contact@adityathebe.com> 1760259042 +0545
committer Aditya Thebe <contact@adityathebe.com> 1760259042 +0545
gpgsig -----BEGIN SSH SIGNATURE-----
 U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgOzmsUfKA9Ly4Ir4snhc6l6dWsa
 TplTM0tvrD14WJQjwAAAADZ2l0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5
 AAAAQBJcQdTma1GZumNRb4ALX0ZY8zJJdtWq17Tzzcobg3QxgHeRUnwIxG1Rsf8IT1pT6O
 PQHeRS4OrzPS3zezQb8AY=
 -----END SSH SIGNATURE-----

second commit

git cat-file -p 8b5a93d517fa8c8d9825f3c711f7cef02310ad91

100644 blob 3b18e512dba79e4c8300dd08aeb37f8e728b8dad    first.txt
100644 blob 17d5d9edf31a80878ad4911017cbd6d1d03322b8    second.txt
040000 tree 354f422855eb94b6325210f7dee06ffa8f91dfd0    src

Perfect, this second tree shows exactly what the project structure was like at the time of the second commit. If you notice, the tree object stores the directory structure for just one level, and the subdirectory src is represented by another tree 354f422855eb94b6325210f7dee06ffa8f91dfd0

git cat-file -p 354f422855eb94b6325210f7dee06ffa8f91dfd0 # another tree

100644 blob cea4f10350c7c3a4acb44629234b5502c24d0989    index.js

We have explored commit and trees. But where are the files actually stored? That's where blob comes in.

Blob:

You must have noticed, when viewing the tree object, some of the listings were blob. Let's take a look at the blob representing first.txt file

git cat-file -p 3b18e512dba79e4c8300dd08aeb37f8e728b8dad

hello world

A blob object stores the content of a file without the metadata like size, extension, timestamp, filename, permission. Some of those metadata are stored in the tree object while some metadata are never tracked by git like the creation timestamp.

Tags:

Tags are pointer to another git object - most commonly a commit - with additional metadata like annotations, tagger information, and a message.

There are two types of tags in Git:

Lightweight tags: Simple pointers to commits (not stored as objects)
Annotated tags: Full objects stored in Git's database with tagger name, email, date, and a message

Let's create an annotated tag to explore it:

git tag -a v1.0 -m "First release"

To list all the tags

git tag

We do not need the tag's hash as cat-file command can also accept the tag name directly.

git cat-file -p v1.0

object c5fa6f4cfca9fd790cb9bded39a2ae48983eddfd
type commit
tag v1.0
tagger Aditya Thebe <contact@adityathebe.com> 1760262129 +0545

First release

The tag points to object c5fa6f4cfca9fd790cb9bded39a2ae48983eddfd of type commit.

Run git log and you'll see the commit with that hash.

References

Running AI agents in parallel

Sat, 11 Oct 2025 14:30:00 GMT

Coding agents roles and commands

Src: https://blog.fsck.com/2025/10/05/how-im-using-coding-agents-in-september-2025/

Roles

Got some good prompts from this article. Now I run multiple sessions that take on different roles. Example:

Planner
- interactive
- plans in chunks
Implementer
Architect verifies the work of the implementer

I really like the planner session where I brainstorm with the AI to come up with a design. I have found that it asks really good questions and addresses things that I had missed in my initial design. But it also asks too much questions which are not that important and I want it to decide for me.

Reviewing PR comments

The only problem with tools like this is that our robot buddies are quite credulous. If you paste in a list of instructions for how to update a codebase, Claude's just going to take you at your word and make the changes, even if what you're asking for is crazy and wrong.

I faced a similar situation too. When I ask Claude to address GitHub Copilot's review comments, it blindly addresses them, taking them at face value.

I'm using this command ~/.claude/commands/fix-pr-comments which works so much better.

Please list all the review comments in this GitHub PR: $ARGUMENTS.

Follow these steps:

- Use `gh api` to fetch comments not the `gh view --comments` command.

  example: gh api repos/flanksource/mission-control/pulls/2173/comments

A reviewer did some analysis of this PR.
They're external, so reading the codebase cold.
This is their analysis of the changes and I'd like you to evaluate the analysis and the reviewer carefully.

Note: If the suggested fix are marked as resolved, skip them

1) should we hire this reviewer? If it's an automated review give it a score from 1-10 on how good he reviews were.
2) which of the issues they've flagged should be fixed?
3) are the fixes they propose the correct ones?

Anything we *should* fix, put on your todo list.
Anything we should skip, tell me about now.

With this prompt, the agent usually discards nitpicks and incorrect comments on the PR.

DoltgresSQL

Src: https://www.dolthub.com/blog/2025-10-03-100-real-world-postgres-dumps/

I came across this article and found out that this company was building a MySQL and PostgreSQL drop-in replacement in Go from scratch! That sounds really exciting but also massively challenging.

This article in particular talks about using real world postgres dumps in their test suite which is pretty clever.

I will definitely be keeping an eye on this project and hopefully consider contributing.

Containers on MacOS !!

Src: https://www.youtube.com/watch?v=x1t2GPChhX8

Native containers are finally here in macOS.

When I migrated from Linux to macOS, one of the challenges I faced was deciding what Docker engine to use. Most people use Docker Desktop, but I decided to use OrbStack because people on Twitter were saying it is much more performant.

I haven't really noticed any performance issues, but it's also because I'm not running anything heavy on it. However, you can tell that starting and stopping containers is so much slower.

It also feels a bit weird that I need to open a GUI to start a Docker service.

Key things about macOS containers:

Development after hype cycle means there is no need to move fast and break things.
Native apps available as containers
Native to OS. No need to download 3rd party like docker on Linux.
OCI compliant. Support Dockerfile.

Notifications on Claude Code

One of the things that I really liked about AMP was the notification bell you get when the agent completes the task or is waiting for a user input. For some reason the notification bell does not work for me with Claude code.

Fortunately, Claude Code now provides a various set of hooks. I have added these two hooks that will play a sound when the agent stops or needs to notify me about something.

{
  "hooks": {
    "Stop": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "afplay /System/Library/Sounds/Blow.aiff"
          }
        ]
      }
    ],
    "Notification": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "afplay /System/Library/Sounds/Glass.aiff"
          }
        ]
      }
    ]
  }
}

Moving to Claude Max Plan

I have also gone all in with Claude Code subscription and upgraded to the Max Plan ($100/mo). I was rarely hitting the token limits on the $20/mo Pro plan but I really wanted to give Opus a try - which is only available on the Max Plan. However, after using Opus and Sonnet 4.5, I don't think Opus is any better. In fact, it's probably worse.

In order to keep my budget of $100 a month for AI tools, I had to cancel my OpenAI subscription as well as the Cursor subscription. I don't really miss Cursor, and even if I do, the free plan will be sufficient. However, I'm starting to like OpenAI's Codex CLI, which probably means I might need to go over my budget and get the OpenAI $20/mo subscription.

Trying out Codex CLI

As I mentioned, I've been playing with Codex CLI. I don't really like its UX. Its permission system doesn't really make sense to me, and I hate that it doesn't even have syntax highlighting. It's also much slower than Claude.

But - what I really like about this is that it was able to solve problems that Claude couldn't. Not to say that it's better, because there have been many cases where Claude just performed better. But when you use Claude and Codex together, you get a much more intelligent agent as a whole, because you can ask Codex to review Claude's changes, and almost always it will find something that Claude missed - and vice versa.

I'll still be using Claude Code as my primary driver, but it's nice to have Codex CLI to verify Claude's work.

ARP - The DNS for IP addresses

Sat, 17 May 2025 15:35:00 GMT

In the OSI model, devices in layer 2 - like a Switch - work with MAC address while devices on layer 3 work with IP addresses. IP packets only know what IP address they are destined to but at the lower level we only have MAC addresses. This tells us that at some point we need to map IP address to MAC address.

As always, this post focuses specifically on Linux networking.

That's where ARP comes in. ARP (Address Resolution Protocol) maps IP addresses to MAC addresses. If you are familiar with DNS, ARP is similar to that [1].

DNS: Map Domain names -> IP addresses

ARP: Map IP addresses -> MAC addresses

ARP lives in-between L2 & L3 - often called L2.5.

It's clear that we need a record of mapping of IP addresses to MAC addresses. But where does that record live? And who populates it? If you consider DNS, the mapping needs to be supplied by a human and is stored in whatever nameserver is chosen.

ARP, on the other hand, doesn't have a pre-populated record to refer to. This is because unlike DNS, IP -> MAC address is much more dynamic. Rather, the mapping is queried when required.

In the example above, we need the mac address for 10.0.0.2. The networking stack sends an ARP request to all the devices connected to the network segment [2] requesting whoever has 10.0.0.2 to send it back the MAC address. The way to send this message to all the devices is by using the broadcast address [3]. The ARP request is received by all the devices in the network segment and the device that actually is assigned the IP address will send back an ARP response. This response is addressed directly to the MAC address of the requester rather than broadcasting it to everyone.

This mechanism prevents the network administrators from maintaining any sort of record as the protocol is self sufficient.

Caching

It would be very inefficient if we had to query the MAC addresses for every single IP packet. That's why the ARP responses are cached. The cache usually has a timeout period in the span of a few seconds.

There's also another more efficient way to invalidate the cache - it's by sending a gratuitous ARP message. Basically, when a device gets a new IP address it'll simply send a gratuitous ARP message broadcasting its own IP address. Other devices will then invalidate any stale records.

Users can also manually flush the entire cache as we'll see in a minute.

ARP inside the Linux kernel

ARP is maintained by the Linux kernel. More specifically by the IPV4 subsytem - https://github.com/torvalds/linux/blob/master/net/ipv4/arp.c.

IPV6 doesn't use ARP. It still does need a protocol to map IP addresses to MAC addresses, but that protocol is not ARP. Instead, it uses Neighbor Discovery Protocol (NDP).

ARP is completely transparent. The IP subsytem in Linux automatically issues ARP requests when it doesn’t have a MAC address for a given.

  ┌────────────────────────────┐                     ┌────────────────────────────┐
  │      Host A (sender)       │                     │      Host B (target)       │
  │────────────────────────────│                     │────────────────────────────│
  │  IP: 192.168.1.10          │                     │  IP: 192.168.1.20          │
  │  MAC: AA:AA:AA:AA:AA:AA    │                     │  MAC: BB:BB:BB:BB:BB:BB    │
  └────────────┬───────────────┘                     └────────────┬───────────────┘
               │                                                  │
               │ 1️⃣  **ARP REQUEST** (broadcast frame)            │
               │     ───────────────────────────────────────────▶ │
               │     Dest MAC: FF:FF:FF:FF:FF:FF                  │
               │     Src  MAC: AA:AA:AA:AA:AA:AA                  │
               │     Payload: “Who has 192.168.1.20?              │
               │               Tell 192.168.1.10.”                │
               │                                                  │
               │                                                  │
               │                2️⃣  **ARP REPLY** (unicast frame) │
               │     ◀─────────────────────────────────────────── │
               │     Dest MAC: AA:AA:AA:AA:AA:AA                  │
               │     Src  MAC: BB:BB:BB:BB:BB:BB                  │
               │     Payload: “192.168.1.20 is at                 │
               │               BB:BB:BB:BB:BB:BB.”                │
               │                                                  │
  ┌────────────▼───────────────┐                     ┌────────────▼───────────────┐
  │ 3️⃣  Host A updates its     │                     │ (Host B may also cache     │
  │     ARP cache:             │                     │  Host A’s mapping.)        │
  │     192.168.1.20 ↔ BB:BB…  │                     └────────────────────────────┘
  └────────────┬───────────────┘
               │
               │ 4️⃣  **Normal traffic** (e.g., IP packets)
               │     sent directly to MAC BB:BB:BB:BB:BB:BB
               ▼
        (Frame delivery continues as usual)



                        Fig: Created by OpenAI o3

Demo

Let's get our hands dirty. We'll use the mighty ip command line tool to investigate ARP records and get a live view of ARP messages. You can follow along this demo if you're on a Linux system (not unix - doesn't work on MacOS)

ip neigh

10.99.99.1 dev vmbr0 lladdr 74:fe:ce:6e:01:59 STALE
10.99.99.115 dev vmbr0 lladdr 9c:9d:7e:54:16:78 STALE
10.99.99.51 dev vmbr0 lladdr 30:f9:ed:c1:26:07 REACHABLE
10.99.99.8 dev vmbr0 lladdr 26:f4:91:9d:2c:86 REACHABLE
10.99.99.5 dev vmbr0 lladdr 12:31:43:ab:7d:aa STALE
10.99.99.131 dev vmbr0 lladdr 92:95:3e:3f:cd:81 DELAY
10.99.99.9 dev vmbr0 lladdr fe:7a:08:2c:fa:4e STALE
fe80::c20:96e8:c4b7:d575 dev vmbr0 lladdr 92:95:3e:3f:cd:81 STALE
fe80::fc7a:8ff:fe2c:fa4e dev vmbr0 lladdr fe:7a:08:2c:fa:4e STALE
fe80::103c:2fff:4455:e2e9 dev vmbr0 lladdr 2a:13:6d:9e:b9:ff STALE
fe80::32f9:edff:fec1:2607 dev vmbr0 lladdr 30:f9:ed:c1:26:07 STALE
fe80::28b1:57ff:fe29:ef4a dev vmbr0 lladdr 2a:b1:57:29:ef:4a STALE
fe80::ac09:e9ff:fee2:75d6 dev vmbr0 lladdr ae:09:e9:e2:75:d6 STALE
fe80::901d:ccff:fe64:ad04 dev vmbr0 lladdr 92:1d:cc:64:ad:04 STALE
fe80::c295:cfff:fe8b:a269 dev vmbr0 lladdr c0:95:cf:8b:a2:69 STALE
fe80::88a1:5fff:fe2b:fcd1 dev vmbr0 lladdr 8a:a1:5f:2b:fc:d1 STALE
fe80::1031:43ff:feab:7daa dev vmbr0 lladdr 12:31:43:ab:7d:aa STALE
fe80::21d8:9d95:d19d:af8f dev vmbr0 lladdr 58:24:29:7a:5f:a6 STALE
fe80::1c5e:aa1b:fa11:bebd dev vmbr0 lladdr 02:fa:76:28:bb:25 STALE
fe80::24f4:91ff:fe9d:2c86 dev vmbr0 lladdr 26:f4:91:9d:2c:86 STALE

we mentioned ARP doesn't work with IPv6 and yet we still see ipv6 above. That's because ip neigh returns ARP & NDISC cache entries.

Let's try cleaning up the cache

ip neigh flush all

Monitor live ARP message

We'll use tcpdump

sudo tcpdump -n -i enp8s0 arp

# replace enp8s0 with your network interface.
# use `ip link` to see the list of network interfaces in your system.

listening on enp8s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:15:43.403688 ARP, Reply 10.99.99.222 is-at 2a:b1:57:29:ef:4a, length 28
15:15:46.435738 ARP, Request who-has 10.99.99.222 (ff:ff:ff:ff:ff:ff) tell 10.99.99.222, length 28
15:15:49.467785 ARP, Reply 10.99.99.222 is-at 2a:b1:57:29:ef:4a, length 28
15:15:52.220755 ARP, Request who-has 10.99.99.5 tell 10.99.99.115, length 46
15:15:52.220947 ARP, Reply 10.99.99.5 is-at 12:31:43:ab:7d:aa, length 28
15:15:52.229263 ARP, Request who-has 10.99.99.115 tell 10.99.99.5, length 28
15:15:52.229574 ARP, Reply 10.99.99.115 is-at 9c:9d:7e:54:16:78, length 46
15:15:52.499759 ARP, Request who-has 10.99.99.222 (ff:ff:ff:ff:ff:ff) tell 10.99.99.222, length 28
15:15:55.420196 ARP, Request who-has 10.99.99.55 tell 10.99.99.4, length 28
15:15:55.531744 ARP, Reply 10.99.99.222 is-at 2a:b1:57:29:ef:4a, length 28
15:15:56.402074 ARP, Request who-has 192.168.1.254 tell 192.168.1.1, length 46
15:15:56.451366 ARP, Request who-has 10.99.99.55 tell 10.99.99.4, length 28
15:15:56.771925 ARP, Request who-has 10.99.99.51 tell 10.99.99.9, length 28
15:15:56.772240 ARP, Reply 10.99.99.51 is-at 30:f9:ed:c1:26:07, length 46
15:15:57.401975 ARP, Request who-has 192.168.1.254 tell 192.168.1.1, length 46
15:15:57.474439 ARP, Request who-has 10.99.99.55 tell 10.99.99.4, length 28
15:15:58.402074 ARP, Request who-has 192.168.1.254 tell 192.168.1.1, length 46
15:15:58.498718 ARP, Request who-has 10.99.99.55 tell 10.99.99.4, length 28
15:15:58.571763 ARP, Request who-has 10.99.99.222 (ff:ff:ff:ff:ff:ff) tell 10.99.99.222, length 28
15:15:59.522361 ARP, Request who-has 10.99.99.55 tell 10.99.99.4, length 28
15:16:00.547347 ARP, Request who-has 10.99.99.55 tell 10.99.99.4, length 28
15:16:01.603792 ARP, Reply 10.99.99.222 is-at 2a:b1:57:29:ef:4a, length 28
15:16:02.171191 ARP, Request who-has 10.99.99.1 tell 10.99.99.115, length 46
15:16:02.181128 ARP, Request who-has 10.99.99.2 tell 10.99.99.115, length 46
15:16:02.191115 ARP, Request who-has 10.99.99.3 tell 10.99.99.115, length 46
15:16:02.201213 ARP, Request who-has 10.99.99.4 tell 10.99.99.115, length 46
15:16:02.201264 ARP, Reply 10.99.99.4 is-at 08:97:98:75:cc:fd, length 28
15:16:02.211140 ARP, Request who-has 10.99.99.5 tell 10.99.99.115, length 46
15:16:02.211402 ARP, Reply 10.99.99.5 is-at 12:31:43:ab:7d:aa, length 28
15:16:02.221140 ARP, Request who-has 10.99.99.6 tell 10.99.99.115, length 46
15:16:02.222453 ARP, Reply 10.99.99.6 is-at 2a:b1:57:29:ef:4a, length 28
15:16:02.231200 ARP, Request who-has 10.99.99.7 tell 10.99.99.115, length 46
15:16:02.241218 ARP, Request who-has 10.99.99.8 tell 10.99.99.115, length 46
15:16:02.241411 ARP, Reply 10.99.99.8 is-at 26:f4:91:9d:2c:86, length 28
15:16:02.251118 ARP, Request who-has 10.99.99.9 tell 10.99.99.115, length 46
15:16:02.251278 ARP, Reply 10.99.99.9 is-at fe:7a:08:2c:fa:4e, length 28
15:16:02.261225 ARP, Request who-has 10.99.99.10 tell 10.99.99.115, length 46
15:16:02.271116 ARP, Request who-has 10.99.99.11 tell 10.99.99.115, length 46
15:16:02.281112 ARP, Request who-has 10.99.99.12 tell 10.99.99.115, length 46

Things we didn't cover

Security aspects (ARP spoofing / Dynamic ARP)

References

[1]. ARP and DNS are similar only at the surface level in the sense that both protocols map an address from a higher level (in the OSI model) to a lower level. They have a lot of differences - example: DNS is global and hierarchical while ARP is local and flat. So take this analogy with a grain of salt.

[2]. A network segment is a section of a computer network where all devices are directly connected and can communicate with each other at the data link layer (Layer 2) without needing to go through a router.

[3]. A broadcast address in layer 2 is - FF:FF:FF:FF:FF:FF. It's an address that doesn't belong to a single device but any message sent to that address is delivered to all the devices.

IP Routing in Linux

Sun, 27 Apr 2025 14:45:00 GMT

If you're looking to dip your toes into networking, IP routing is a great place to start. I think it's the sweet spot because it's low level and at the same time, not very alien - we have all heard of IP addresses. And not just that but IP routing is something that's much more applicable and relevant than starting from the very low level like the physical layer and data link layer.

In this article, we'll be covering how a linux networking stack decides to route packets out of it. We'll not be covering how the packets traverse the internet to reach their destination. Although, the concepts we'll cover will apply there as well.

In short, we'll be covering the decision making process of the linux system when it needs to send out a packet.

Here's a question for you - Say you're connected to your home network via ethernet and also to your mobile hotspot via wifi.

You're connected to two different networks

Your home ISP
Mobile network operator

When you visit youtube.com, does it use the ethernet or the wifi? Would it be any different if you were to visit gmail.com?

This is exactly the question that we'll be covering in this article.

Network Interface

Before we dive into ip routing, it's important to understand what network interfaces are as they are a crucial part of routing.

Network interfaces are devices that act as a point of connection in a network. They are the component that connect your device to a network. It's through them traffic flows in and out of your device. An interface has a MAC address associated with it that is used to identify it on the network.

For instance: an ethernet port or a wifi interface.

Virtual interfaces

An interface can also be emulated. These virtual interfaces are mainly used in Virtual machines, Containers, VPNs, and in software development. Examples:

lo loopback interface (127.0.0.1 aka localhost)
docker0 interface (used by docker containers)
tailscale0 interface (used by tailscale)
wg0 interface (used by wireguard)
veth interface (used by virtual ethernet pair)

Analogy to understand network interfaces

I'll use an analogy of social network to explain this.

Let's say you want to upload an image to Instagram. To do that, you need to have an Instagram account - there's no way around it. You need an entry point to connect to the Instagram world. Likewise, you need an interface to connect to a network.

Multiple interfaces

You can have multiple social media accounts to connect to different social networks. Likewise, your device can also have multiple network interfaces to connect to different networks. Example, your laptop most likely has an ethernet port in addition to a wifi interface.

Routing decision

To upload an image you would visit Instagram - to make a tweet you would visit Twitter - to send an email you would log into your email provider. In a similar way, IP routing involves which network interface should be chosen to send out a packet.

Pretty cheesy analogy? I think so too. Nonetheless, I think it does help a little.

Now, to answer the previous question, your device needs a policy to decide whether to use the ethernet interface or the wifi interface to reach a destination like YouTube.

`ip` tool

In Linux systems, the ip tool is the primary way to interact with networking internals and lower level networking APIs. It's part of the iproute2 software suite. It has several commands that manage various aspects of networking. For instance:

Command	Description
`ip link`	manage network interfaces
`ip addr`	manage IP addresses
`ip route`	manage routing table
`ip rule`	manage routing rules

There are numerous more but these are the ones we'll be focusing on for this article.

View all interfaces on your system

Try running ip link to see all the interfaces on your system. My Linux desktop has 2 physical interfaces and they are connected to different networks.

ip link

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether d8:43:ae:45:c5:62 brd ff:ff:ff:ff:ff:ff
    altname enxd843ae45c562
3: wlo1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DORMANT group default qlen 1000
    link/ether f0:20:ff:21:57:cd brd ff:ff:ff:ff:ff:ff
    altname wlp0s20f3
    altname wlxf020ff2157cd
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
    link/ether ea:d5:3b:18:3f:ad brd ff:ff:ff:ff:ff:ff

There's a lot of information here. I like to use the -br and -c flags to get a colored brief output.

ip -br -c link

lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
enp5s0           UP             d8:43:ae:45:c5:62 <BROADCAST,MULTICAST,UP,LOWER_UP>
wlo1             UP             f0:20:ff:21:57:cd <BROADCAST,MULTICAST,UP,LOWER_UP>
docker0          DOWN           ea:d5:3b:18:3f:ad <NO-CARRIER,BROADCAST,MULTICAST,UP>

I have 4 interfaces

lo a loopback interface (virtual). It's purely virtual and doesn't have a MAC address.
enp5s0 a physical interface connected to my home ISP with MAC address d8:43:ae:45:c5:62
wlo1 a wifi interface with MAC address f0:20:ff:21:57:cd
docker0 a virtual interface used by docker containers with MAC address ea:d5:3b:18:3f:ad

They are all enabled as shown by the "UP" flag inside the < and > brackets.

View IP addresses of all network interfaces

To view the IP addresses associated with the interfaces, use ip addr command.

ip -br -c addr

lo               UNKNOWN        127.0.0.1/8 ::1/128
enp5s0           UP             10.99.99.65/24 fe80::14ad:a0eb:c8f:c5b0/64
wlo1             UP             192.168.254.3/24 fe80::5fd6:c815:261b:5df7/64
docker0          DOWN           172.17.0.1/16

As you can see, the wifi interface (wlo1) and ethernet interface (enp5s0) are connected to two different networks as they have IP addresses on different subnets - 10.99.99.0/24 and 192.168.254.0/24.

IP destination types

Finally, one last thing to keep in mind is the different types of destinations.

A networking device must be aware of

its own IP addresses
addresses that are directly reachable
and the rest.

By directly reachable we mean the destination is on the same network as your device. No routing is involved here. Your device will use ARP to find the MAC address of the destination and send the packet directly to it.

For any other destination, the packet will be sent to a router.

Type	Description	Examples
local	These are the ip addresses of the network interfaces on your system	The loopback address `127.0.0.1/8`, the local IP addresses `10.99.99.65` & `192.168.254.3`
connected networks	These are the ip addresses of other devices connected to the same network	Other devices on `10.99.99.0/24` and `192.168.254.0/24`
remote	These are the ip addresses that are neither local nor connected networks	`1.1.1.1`, `8.8.8.8`, ...

Alright, now let's get into the nitty gritty of routing.

IP routing in Linux is handled by the IP routing subsystem called FIB (Forwarding Information Base). FIB is a general term, that's not specific to Linux, that refers to the data plane representation of a routing table. Cisco, BSD, etc. all use this term.

The subsystem consists of 3 components that work together to route packets:

Routes
Routing table
Routing policy database (RPDB)

Routes live in the routing table and RPDB contains rules that determine which routing table to use for a given packet.

A. Route

A route is an entry in the routing table that Linux kernel uses to decide where to send a packet (next hop) via what interface based on the destination IP address. This applies to both incoming and outgoing packets.

# dummy routes

- to reach (31.13.79.35/32) facebook.com send packets to the 10.99.99.1 (router) via wifi interface
- to reach (142.250.194.174/32) youtube.com send packets to the 10.99.99.1 (router) via ethernet interface

A route consists of the following components:

Component	Description
Destination	The destination IP address
Next hop	When the destination is a remote IP, the next hop is the next router to send the packet to. Usually the default gateway
Interface	The network interface to send the packet out
Flags	Special options (like UP, DOWN).
Metric	A priority number - lower metric means higher priority if multiple routes match.

`ip route`

Use ip route to see all the routes on your system. (This isn't entirely accurate as we'll see when we get into the routing table). Below is an example of actual routes on my Linux machine.

ip route list

default via 10.99.99.1 dev enp5s0 proto dhcp src 10.99.99.65 metric 100
default via 192.168.254.254 dev wlo1 proto dhcp src 192.168.254.3 metric 600
10.99.99.0/24 dev enp5s0 proto kernel scope link src 10.99.99.65 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.254.0/24 dev wlo1 proto kernel scope link src 192.168.254.3 metric 600

Here's a breakdown:

There are 3 routes and 2 default routes. The 3 routes target the connected networks (every device on the respective subnets) and the 2 default routes target the remote destinations. It's interesting to note that the wifi interface has a higher metric (600) than the ethernet interface (100). This means the kernel will prefer using ethernet to reach the destination!

That's why there's no next hop specified on them as they can be reached in the very next hop.

For any other destination, the packet will be sent to the default gateway, which is 10.99.99.1 for enp5s0 and 192.168.254.254 for wlo1, via the respective interfaces.

Default Gateway

Essentially, a default gateway is the fallback destination when no specific route is found. Think of it like the default case in a switch statement.

packet := new_packet(src="10.99.99.65", dst=destination)

switch destination {
  case "10.99.99.0"..."10.99.99.255":
    send_to_next_hop(destination, nexthop=destination, interface="enp5s0")
  default:
    send_to_next_hop(destination, nexthop="10.99.99.1", interface="enp5s0") // default gateway
}

packet := new_packet(src="192.168.254.3", dst=destination)

switch destination {
  case "192.168.254.0"..."192.168.254.255":
    send_to_next_hop(destination, nexthop=destination, interface="wlo1")
  default:
    send_to_next_hop(destination, nexthop="192.168.254.254", interface="wlo1") // default gateway
}

Next hop means the layer2 packet sends to the MAC address of the next hop. The IP packet still contains the original destination.

Query the route for a destination

You can even check which route a packet takes for a given destination using ip route get command.

Local addresses

ip route get 10.99.99.3

10.99.99.3 dev enp5s0 src 10.99.99.65 uid 1000
    cache

Remote addresses

ip route get 1.1.1.1

1.1.1.1 via 10.99.99.1 dev enp5s0 src 10.99.99.65 uid 1000
    cache

As you can see, 1.1.1.1 is not a device on the same subnet as my device. So, the packet will be sent to 10.99.99.1 - which is the default gateway.

The router will also have its own routing table. If the router doesn't have a route specified for 1.1.1.1, it will send the packet to its own default gateway which is the ISP's router.

Once the packet is inside the ISP's network, protocols like BGP (Border Gateway Protocol) may take over and figure out the best path to the destination.

Adding a new route

Let's get our hands dirty! I'll query https://ipinfo.io/ip to get my public IP address.

curl -s https://ipinfo.io/ip

43.231.211.110

In order to see which route this destination will use, we need to get the IP address for ipinfo.io. I'll use the dig command for it.

dig +short ipinfo.io

34.117.59.81

Now, let's see which route this destination will use.

ip route get 34.117.59.81

34.117.59.81 via 10.99.99.1 dev enp5s0 src 10.99.99.65 uid 1000
    cache

Sure enough, it's using the ethernet interface.

Let's add a new route so that this destination uses the wifi interface.

sudo ip route add 34.117.59.81/32 via 192.168.254.254 dev wlo1

Now, let's check which route this destination will use.

ip route get 34.117.59.81

34.117.59.81 via 192.168.254.254 dev wlo1 src 192.168.254.3 uid 1000
    cache

curl -s https://ipinfo.io/ip

120.89.104.16

And there it is — the public IP address has changed, confirming that traffic is now going through the Wi-Fi interface.

It's important to note that this route isn't persistent and will be lost after a reboot.

Let's clean it up

sudo ip route del 34.117.59.81/32

B. Routing Table

A routing table is a collection of routes. You can have multiple routing tables on a single system. By default, there are three built-in routing tables:

local
main (ip route uses this table by default)
default (often empty unless configured)

I couldn't find any command that lists out all the routing tables. Routing tables are defined in /etc/iproute2/rt_tables file. So just check that out.

It's possible that file is not present on your system. My desktop didn't have it as well. Here's what it looks like in one of my vms.

cat /etc/iproute2/rt_tables

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep

local routing table

The local routing table contains routes for local destinations (i.e. its own IP addresses). It's maintainted by the kernel itself and is used to route incoming packets or outgoing packets destined to the host itself.

ip route show table local

local 10.99.99.65 dev enp5s0 proto kernel scope host src 10.99.99.65
broadcast 10.99.99.255 dev enp5s0 proto kernel scope link src 10.99.99.65
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
local 172.17.0.1 dev docker0 proto kernel scope host src 172.17.0.1
broadcast 172.17.255.255 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
local 192.168.254.3 dev wlo1 proto kernel scope host src 192.168.254.3
broadcast 192.168.254.255 dev wlo1 proto kernel scope link src 192.168.254.3

The broadcast entries are interesting ones. Broadcast IP addresses are special addresses that deliver packets to all the devices on the network. If you send a packet to a broadcast address, it will be sent to all the devices on the network. Hence - the name.

main routing table

The main routing table contains routes for all other destinations (connected networks & remote destinations).

ip route show table main

default via 10.99.99.1 dev enp5s0 proto dhcp src 10.99.99.65 metric 100
default via 192.168.254.254 dev wlo1 proto dhcp src 192.168.254.3 metric 600
10.99.99.0/24 dev enp5s0 proto kernel scope link src 10.99.99.65 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.254.0/24 dev wlo1 proto kernel scope link src 192.168.254.3 metric 600

This is the default routing table that ip route uses.

default routing table

I'm not sure what the purpose of this table is. It isn't even present on my system.

ip route show table default

Error: ipv4: FIB table does not exist.
Dump terminated

C. Rule

A rule, determines which routing table to use for a given packet. Each rule has a priority, and rules are examined sequentially from rule 0 through rule 32767.

ip rule show

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

Routing Policy Database (RPDB)

Rules live in a database called the Routing Policy Database (RPDB).

RPDB (Rules: priority-ordered)
 └──> Routing Table (chosen by rule)
      └──> Routes (prefix + metric ordered)

Things we didn't cover

Why do we need more than one routing table?
Routing cache
Route matching algorithm (longest prefix match and lowest metric wins)
Rule types (unicast, multicast, prohibit, blackhole, unreachable)

References:

Golang Footgun: Why is this interface not nil?

Fri, 04 Apr 2025 23:30:00 GMT

Source: Gpt-4o

I'm gonna show you a snippet of a Go code and you tell me what the output is gonna be.

package main

import "fmt"

type MyCustomType struct {}

func getResult() any {
  var t *MyCustomType
  return t
}

func main() {
  result := getResult()
  fmt.Println(result == nil)
}

// Try it out: https://goplay.tools/snippet/-axDsS493LZ

Here's what went through my mind:

variable t is of type *MyCustomType
t is nil
getResult returns the nil t
so, result must also be nil

Let's see what happens when we run the code.

go run main.go
false

The crucial bit of information here is that the function returns an interface - in this case any or an empty interface interface{}.

Explanation

In Go, an interface is a pair of a type & a value (T, V). Eg: in var myInterface any = 3, the interface myInterface has a type int and a value 3.

For an interface to be nil, both its type and value must be nil. The following example shows a nil interface:

var a any
fmt.Println(a == nil) // true

// Try it: https://go.dev/play/p/X0h-DCswGPG

It's nil because it hasn't been assigned any concrete value yet.

On the contrary, take a look at the following code:

var a *int
var b any = a
fmt.Printf("typeof a: %T\n", a)
fmt.Printf("typeof b: %T\n", b)
fmt.Printf("a == nil: %v\n", a == nil)
fmt.Printf("b == nil: %v\n", b == nil)

// Try it: https://go.dev/play/p/3gq25jCzepx

The interface b is not nil as its type is *int and its value is nil. It's important to distinguish that the interface holds in a nil pointer, but it's not nil itself.

The above program outputs:

typeof a: *int
typeof b: *int
a == nil: true
b == nil: false

This is also addressed in the Go FAQ.

How to Check the Underlying Value for Nil

So, what's the correct way to check if the underlying value is nil when you receive an interface? The two approaches are:

Type assertion

package main

import "fmt"

type MyCustomType struct{}

func getResult() any {
	var t *MyCustomType
	return t
}

func main() {
	result := getResult()
	if v, ok := result.(*MyCustomType); ok {
		fmt.Printf("v is nil: %v, result is nil: %v\n", v == nil, result == nil)
	}
}

// Try it out: https://goplay.tools/snippet/MeMUtzKG9LT

Reflection

Type assertion works fine when you know exactly what type the function returns. In our case, we know that the function returns a pointer to MyCustomType.

But, imagine cases where getResult() could conditionally return various types. We would need to assert all the possible types that getResult() could return. Or, it could also return an unexported type so we cannot assert it.

In such cases, we can use reflection to check if the result is nil.

package main

import (
	"fmt"
	"reflect"
)

type MyCustomType struct{}

func getResult() any {
	var t *MyCustomType
	return t
}

func main() {
	result := getResult()
	fmt.Printf("result is nil: %v\n", result == nil) // false
	fmt.Printf("result contains nil: %v\n", reflect.ValueOf(result).IsNil()) // true
}

// Try it out: https://goplay.tools/snippet/ZxJNYBW0pVz

I must say that using reflection is generally frowned upon as it adds a lot of runtime overhead. There are also some caveats. Eg: reflect.ValueOf(result).IsNil() can panic for certain types like a string or an int.

Here's a generic helper function that can be used to check if the underlying value is nil.

func isNil(i any) bool {
	if i == nil {
		return true
	}

	v := reflect.ValueOf(i)
	k := v.Kind()

	// Check for kinds that have a concept of nilness
	switch k {
	case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Ptr, reflect.Slice, reflect.UnsafePointer:
		// For these kinds, IsNil() reports whether the value is nil
		return v.IsNil()
	default:
		// Other kinds (structs, basic types, arrays) cannot be nil themselves
		return false
	}
}

Git Hygiene: How to Automatically Delete Stale Local Branches

Thu, 27 Mar 2025 21:25:00 GMT

Feature branches in Git can quickly pile up. I like to keep my branches clean because when using auto completion I don't want to go through a list of branches that have already been merged or discarded. And I just like to keep things clean and tidy in general.

Created with Gpt4o

Primer on Git branches

Let's say you have a feature branch feat/notifications. You push that branch to the remote repository to create a PR. The remote is usually named origin so you will have another branch, in your local repository, called origin/feat/notifications that tracks the remote branch.

Branch	Description
`feat/notifications`	The local branch you created
`origin/feat/notifications`	The reference in your local repo that tracks the remote branch
`feat/notifications`	The branch on the remote repository

Try git branch -vv to see all the branches and their tracking branches.

git branch -vv
* feat/notifications a84be78 [origin/feat/notifications] feat: add notifications
  main               2831500 [origin/main] chore: remove deadcode

Delete remote branches locally

Now, assume the PR got merged and the branch feat/notifications was deleted from the remote repository.

Running git fetch -p (short for git fetch --prune) will delete the origin/feat/notifications branch from your local repository but not the feat/notifications branch.

If you list the remote branches again, you'll see that the branch is marked as gone.

git branch -vv
* feat/notifications a84be78 [origin/feat/notifications: gone] feat: add notifications
  main               2831500 [origin/main] chore: remove deadcode

So we have just deleted the remote tracking branch. But how do we delete the local branch?

When you run git fetch -p, it lists out all the remote tracking branches it just deleted. Then, for each of those branches, you need to manually run git branch -D <branch-name> to delete the branch from your local repository.

This is how I've been doing it so far but there's a better way to do it.

List out all the stale branches

git branch -vv | rg '\\[.*gone.*\\]' | sed 's/^..//' | cut -d' ' -f1

This uses a few unix commands to extract the name of the branches that have been deleted from the remote repository. Finally, we use xargs to delete the branches from the local repository.

git branch -vv | rg '\\[.*gone.*\\]' | sed 's/^..//' | cut -d' ' -f1 | xargs -r git branch -D

To break it down

git branch -vv - Lists all branches with their tracking information
rg '\\[.*gone.*\\]' - Filters branches that have been marked as "gone" (deleted from remote)
sed 's/^..//' - Removes the first two characters from each line (which are either * for the current branch or for other branches)
cut -d' ' -f1 - Splits the line by spaces and extracts the first field, which is the branch name
xargs -r git branch -D - Passes each branch name to the git branch -D command to delete it locally

Setup an alias in Git config

I have this as an alias in my git config in ~/.config/git/config

[alias]
    clean-gone = "!git branch -vv | rg '\\[.*gone.*\\]' | sed 's/^..//' | cut -d' ' -f1 | xargs -r git branch -D"

Now, I can simply run git pull && git fetch -p && git clean-gone to clean up those stale branches.

git pull && git fetch -p && git clean-gone
Deleted branch feat/notifications (was a84be78).

Why I switched from Arch Linux to MacOS

Sat, 22 Mar 2025 18:00:00 GMT

I've been using Linux for about six or so years now and five of those years have been with Arch Linux. I've customized my development environment a lot over the years and not to sound like a braggart, but I consider myself a power user. It was hard to imagine moving to a different operating system, because I had a very tailored development environment where I was very efficient.

Arch wasn't perfect, but it was close. If you aren't familiar with Arch - it's a Linux distro that's very barebones and requires the user to place pieces together to have a working system. It's like Subway - you choose all the ingredients. It's also a rolling release distro - meaning the app store almost always will have the latest version unlike Ubuntu or Debian were you will find versions that are older than 2 years. For that reason, Arch is known to be unstable but my 5 year old installation never let me down even with daily sudo pacman -Syu.

Pain of using a desktop machine

For the past year, I have been working exclusively out of my desktop machine. It has an Intel i9 14900k and 32GB of RAM and is seriously a beast of a machine. And before that I used to work from a 2019 Dell XPS 15 which I gave to my dad so the desktop is the only machine I was left with.

While the desktop is insanely powerful, it's not portable and susceptible to frequent power cuts. I live in a country where powercuts are a regular thing and my tiny UPS only lasts for about 5-10 minutes.

created with: Gpt4o

What made it worse is that suspending was not working for me. It had something to do with NVIDIA drivers and I tried to troubleshoot it but never had any luck. So, every time there was a powercut, even for a few minutes, I would have to completely shutdown the machine.

The noise, although relatively quiter than other desktops, was still a bit too much for me. I prefer a quiet environment to work.

Beyond these challenges, the most significant drawback was the desktop's inherent inflexibility. My remote job offers complete freedom regarding when and where I work. However, I couldn't take advantage of this flexibility - working from cafes or during travel was impossible, and I couldn't even work from my own bed. I was essentially tethered to my desk at all times.

So, really the main reason for me to switch to a Mac was that I needed a powerful laptop that was also power efficient - and no one does it better than Apple. There are a few laptops with arm cpus that are probably more power efficient than Apple Silicon, but nowhere near as powerful.

It wasn't the OS that made me switch to Mac, but it was the hardware.

So eventually after months or even years of convincing myself that a Mac wouldn't be worth it, I finally decided to make that switch. Spoiler alert, it has been absolutely worth it.

Don't think with your fingers

Sun, 16 Mar 2025 12:30:00 GMT

I have been writing blogs on this website for about 10 years now, and honestly, every time I've sat down to write a blog post, it has taken me hours. And I would never complete the blog post in one sitting. I would put in a draft and keep coming back to it for several days before finally publishing it.

That's because it was just too difficult or unnatural to write a blog post. The words just didn't come to me naturally. And I think that's because I used to think with my fingers. You know how when you think of words while your hands are on a keyboard, there's a disconnect? The words just don't come naturally to you - or maybe it's just me.

Whenever I first thought of writing a blog, I would have a draft in my head of what to write, and it would sound really good. But then when I sat down and actually started to write, I would go blank. And the chain of thought just gets broken constantly because you're typing at the same time you're thinking.

created with: Gpt4o

Now that's going to change because a few days ago I discovered an application called SuperWhisper. I came across it from a video by Andrei Karpathy where he shows how he uses LLMs and apparently the way he interacts with LLMs or even his computer is through dictation about half the time. So I installed this application and I've been using it for a few days. And honestly, it's a game changer. I cannot praise it enough.

So the first thing that came to my mind when I installed this application was that I could write my prompts to LLMs easily, but then it hit me that I could use this tool to write all my blog posts. And in fact, all of the writings I've done so far in this blog, I did that through this app.

And let me tell you the difference is, what's the word I'm looking for? The difference is... let me think... Claude just told me "night and day". So yes, the difference is night and day.

I think you can tell that this blog post feels much more natural than my other blog posts. I assume it feels like someone speaking to you because in a sense that's exactly it. I'm really speaking out these words. My eyes are closed and the words are just flowing out like a river.

I've also always avoided using LLMs for my blog posts because I didn't want to fill my blog with words that came from an AI model rather than from me. The nice thing about using SuperWhisper is that while you're still using AI, you're not using it to generate thoughts, but just to transcribe your own thoughts.

To top it off, the tool is completely* free. I mean it has premium offerings that will let you use much more powerful models. But I've been using the most basic model offered in the free tier. And it's so, so good. It also runs locally, so I don't need to be connected to the internet and my laptop is powerful enough that it's just instant.

The first time you install the software it gives you 15 minutes of credit to use the top speech to text models. I used the 3GB Ultra model that runs locally, but you can also choose a cloud model if you don't want to download anything to your device. It was quite good, but the free model I'm currently using is honestly good enough that I don't ~~want~~ need the premium version. So I'm just gonna stick to it for now & see how it works out. And if I feel like it's inadequate, I will be more than happy to get a subscription.

I think that's it for this post. I honestly cannot believe how much time this tool is going to save me, not to mention the improvement in the quality of my writing on this blog. This blog post took me just 5 minutes to "write" and then 20 more minutes to edit and finalize because I'm not articulate.

Motorcycle Trip: Kathmandu to Mustang

Sat, 08 Mar 2025 00:00:00 GMT

It was a trip planned for 5 days but unfortunately we had to cut it short.

Thasang, Mustang (near Holiday Camp)

Here's a very not-so-detailed itenerary.

Day	Time	Location	Details
Feb 5	09:20am	Left Pokhara	Fuel: ₹2,250
	10:20am	Dimuwa	Traffic violation
	10:50am - 01:00pm	Kushma	Food: ₹850 chicken rice
	01:25pm - 02:20pm	On the way to Beni	Puncture
	02:45pm	Beni	Fuel: ₹510
	03:25pm - 03:50pm	Tatopani
	04:10pm - 04:20pm	Rupse Jharna
	04:30pm	Ghasa	Police checking. Takes 2 minutes
	06:00pm	Marpha	Room: ₹1,200; Food: ₹650 chicken rice
Feb 6	10:10am	Reached Jomsom	Food: ₹650 chicken rice
	12:10pm	Dhumba Lake	Coffee: ₹100; Entry fee: ₹30
	03:00pm	Muktinath	Room: ₹2,000 (Dolma Inn)
Feb 7	11:00am	Reached Jomsom	Breakfast at MoonLight Hotel. It was good.
	12:40pm - 01:30pm	Chhusang	They register your vehicle if you cross this point.
	04:00pm	Lete	Took a lot of pictures. Incredibly beautiful place.
	06:00pm	Tatopani
	08:10pm	Kushma (The Cliff, Nepal)	Room with breakfast: ₹8,000, Bungee ₹9000

Notes to Motorcycle Renters

Where to Rent: Most bike rental shops are located in Hallanchowk, Lakeside. Nearly every store in the area, including barber shops and clothing stores, seem to have a few motorcycles available for rent.
Bike Condition: The motorcycles are generally in good condition, with some having less than 1,000 km on the odometer.
Price Variation: Rental prices vary significantly from shop to shop, so take the time to compare. For instance, we found an XPulse being offered for anywhere between Rs 2,400 and Rs 3,500 per day.
Pricing Based on Route: The rental cost depends on your destination. If you're heading to Lower Mustang, expect different rates than for Upper Mustang. We paid Rs 18,000 for a 5-day rental to Lower Mustang.
Security Deposit: Renters require an advance payment, including a security deposit. We paid Rs 40,000 in total, with Rs 22,000 as a deposit.
Helmets: Helmets are provided, but we opted to bring our own.
Required Documents: You must present your original driving license and citizenship ID.

Legality Issues

Technically, renting motorcycles is illegal since all rental bikes have private number plates. Renters will inform you of this beforehand, but it can still be a hassle. We rented a Royal Enfield Scram, and traffic police could immediately tell it was a rental. We had to pay Rs 2,000 as a fine for this, and turns out our bike had already been fined multiple times for the same offense.

Tips to Avoid Getting Stopped

Try to rent a newer bike to stay under the radar.
A bike with a Bagmati number plate is less suspicious.
A Gandaki number plate will attract attention and likely get you stopped.

Frequent tyre punctures

Ensure your rental bike has new tyres. We experienced three punctures in three days.
Workshops in the region do not repair tubes - they only sell new ones.
Each replacement tube cost us around Rs 1,000.

Podcast Notes & Review - Modern SQL

Sat, 11 Jan 2025 00:00:00 GMT

Episode: https://postgres.fm/episodes/modern-sql/transcript

This was one of my favorite episodes of postgres.fm - a podcast I regularly listen to. The guest - Markus Winand - had such a fresh perspective on the mundane and conventional wisdom of SQL that it honestly shifted my perspective and also inspired me to learn some newer features.

Most of us think of SQL as a query language - I mean, its very name indicates that - however Markus presents a different view. And his emphasis on SQL window functions has given me enough motivation to take them seriously. I'd always seen them as this advanced feature I could probably live without. But the way Markus emphasized them made me realize I've been missing out on some seriously powerful tools. It's funny how sometimes it takes someone else's excitement about a seemingly ordinary topic to reignite your own curiosity about it.

Flavors of SQL

SQL has different flavors and most people do not know that.
To most people, SQL means SQL 92.

I personally wasn't aware of this either.

SQL standard, formally known as ISO/IEC 9075, is maintained jointly by ISO and IEC. The standard has core features that every conforming SQL vendors like PostgreSQL, MySQL, SQL Server must implement. How the vendors implement those features are up to them. There are also a set of optional features that the vendors may choose not to implement. PostgreSQL, seems to be the one that is highly compliant with the SQL standard.

Here are some major SQL standards and their notable features:

SQL-92: Introduced JOIN syntax, CASCADE operations, and foreign keys
SQL:1999: Added triggers, recursive queries (WITH RECURSIVE), and regular expressions
SQL:2003: Introduced Window functions, MERGE statement, and XML features
SQL:2016: Added JSON operations, temporal tables, and row pattern matching
SQL:2023: Introduced property graph queries & JSON data types

Modern SQL

Modern SQL has features which are not rooted in the relational idea
- Window functions
- Recursive queries
- JSON
SQL 92 was almost feature complete. So it's difficult to convince people to use modern SQL.
Why should people use modern SQL when classic SQL has been serving them fine?
- Eg: With SQL 92, it was very common (almost required) to do self JOINs.
  - Self joins are complex & slow and Window functions are killers of self joins.
  - Window functions are more concise, clear, readable, writable, maintainable, and performant

"The most undervalued feature of modern sql is, and always was, Window functions."

- Markus

My impression is that - modern sql is anything that has to do with more than just the relational stuff. They also improve the language to do things, that may have been previously possible, but, in a more performant and concise way.

https://learnsql.com/blog/history-of-sql-standards/

NULL handling

Modern SQL also offers various ways to handle NULL values. Example: here are some features in modern sql regarding NULL value handling

IS DISTINCT FROM/IS NOT DISTINCT FROM are NULL safe comparison operator.
NULLS DISTINCT/NULLS NOT DISTINCT deals with how nulls are handled in unique contraints.

CREATE TABLE users (
    email TEXT UNIQUE NULLS NOT DISTINCT,
    backup_email TEXT UNIQUE NULLS NOT DISTINCT
);

Since Postgresql v15+, it's is now possible to have multiple rows with null values on a column with unique constraint.

I'm only learning that NULL values in SQL have some nasty qwirks. Only yesterday, there was a blog post on NULL values in SQL floating around in various subreddits & Hackernews.

Modern perspective on SQL

People in ORM world have misconception of SQL.
- SQL isn't just about relation anymore; not since 99.
- A very popular ORM framework in JAVA is JPA. It stands for Java Persistence API. However, SQL isn't just about persistence. It doesn't "just" store data.

"SQL isn't a query language; it's a transformation language. Look at CSS selectors - that's a query language. The focus is on transformation of data to answer a question at hand rather than to issue simple fetch queries."

- Markus

Absolutely loved this section. It really emphasises the power of SQL.

Data Normalization

Causes confusion among layman and new comers.
Just try to figure out in which tables will the data feel comfortable in. Forget the scientific definitions of normalization levels.

Very much in agreement with this sentiment.

Normalization forms are highly emphasized in schools and I don't think they should be. It's overwhelming to someone new to databases & learning all the normal forms and conversion between them is not something that a beginner should be spending their time on. It's like learning all about OOPs and then writing your first hello world.

What's new in `SQL:2023`

Biggest & killer feature - Property Graph Queries (SQL/PGQ). This allows data in tables to be queried as if it were a graph database.
JSON type

Building a mindset to use new features

Data types like json, jsonb, boolean, ... have 3 use cases

Persistent schema. Eg: SQL 2016 didn't have a JSON data type for the schema but the operations on JSON practically meant JSON data types existed in SQL.
During the transformation
- eg: boolean data type is mainly used during transfomation in (WHERE, ON, WHEN clause of CASE) rather than in the schemas. MySQL doesn't even have a boolean type.
- It's not coming from the disk and it's not going up to the client
API Layer
- where json can be very handy (json_array_agg)
- we don't want normalization in API Layer (normalization is fine for the disk)

Reflecting on My iPhone Experience: The Good and the Bad

Sun, 29 Dec 2024 00:00:00 GMT

Almost three years ago, I dropped my Google Pixel 4a from a motorcycle, damaging the display. Google has no official stores or service centers in Nepal and it was virtually impossible to get the replacement parts. I searched for it even in India and had no luck. I knew finding a replacement would take time and it was time for me to find a provisional cheap phone for the time being.

I obviously didn't want to cash out on a temporary phone but I also wanted a phone with a good camera. The best option for me at the time was to get a used older iPhone model. Eventually I bought a used iPhone Xs (64GB) for around NPR 30k (USD $240 at the time) which was surprisingly in really good condition. Sellers are notorious for claiming "not even a single scratch" but this time it was actually true!

After having used both platforms for a considerable amount of time, I think I'm in a good position to offer a fair assessment of them. So here are a few pros and cons from personal experience.

Disclaimer

I'm fully aware that I'm reflecting on my experience with a rather old iPhone, but most of the things on the list below are software related and I'm still on the latest iOS version.

On the flip side, Android users might argue this isn't a fair comparison since I'm comparing a mid-range Android phone with a flagship iPhone. But again the list is mostly software related, and I was using stock Android.

I also don't own a mac to fully benefit from the Apple ecosystem, but it's on it's way...

Pros

Software updates. It's ~~2024~~ 2025 and I'm still receiving major version upgrades albeit this is probably the last one.
Hardware
- Incredibly reliable camera. The camera's video quality is probably one of the major factors that'll keep me from switching back to Android. Pixel's camera was comparable to iPhone's but when it came to video, iPhone ran circles around Android.
- Speakers
- 3D touch
- Premium feel, good display and all of that ...
Airdrop. It's so convenient and fast to transfer photos.
Swipe down and search.

This is something that grew on me. I wasn't a fan initially. My frequent apps are on the homepage but for seldom used apps it's so much easier to swipe down and just search for it. I rarely swipe right of the homepage to find an app. I just search for it.

And this is a pattern in throughout iOS.
Link previews on a pop up on browser. This is one my favorites. I like how homogenous the browsing experience is no matter which app you're on. I can just hard press on a link and a tab pops up. So handy!
Focus modes. I'm sure Android has something similar by now or maybe it always did. But iOS actually made me want to setup focus modes. I have two Focus modes - work and sleep that activate on a schedule.
Do not disturb mode - I can't quite recall exactly what it was on Pixel - but DND didn't really feel like DND because you'd still get notifications (don't quote me on this one)
Silent button - A physical button to activate Silent mode. I love this! Although it's easy to forget to switch it off.
I like that the files app has native SMB support.
Beautiful fonts and coherent UI experience. As a Linux user, a consistent UI is something that really stands out to me because you rarely experience that in the Linux world.
Privacy
- Fine grained permissions for apps. I especially like that you can selectively allow which photos an app can access instead of allowing it your entire gallery or nothing at all. Also, you can select if you want to provide precise or imprecise location permission to apps but I think Android also has this maybe...
- Sign in with apple allows me to regsiter with a unique pseudonymous email address.
FaceTime and iMessage are also pretty cool.

I am aware that apps are generally better built on iOS, but I haven't personally experienced that.

Cons

Based on your usage/experience you might probably not experience some of these.

No access to file system. Major major bummer! I couldn't copy over all my mp3s like I could on Android. The hack to overcome this was to use "Documents" app and then transfer those mp3s to it and then use it as the music player 😐. No thanks!

This isn't a problem for me anymore as I now have a self hosted music server and there are some really good free open source iOS music players that work with it.
Custom photos picker.

I don't use iCloud because all my photos were already in a Google Photos (I've moved to Immich now). iOS doesn't allow using these 3rd party apps to pick files for upload. You need to download it and then select it from the Photos app.
Auto rotation.

This isn't about the screen auto rotation lock - or the Portait orientation lock. I'm especially talking about deciding video rotation when you have auto rotation disabled.

Android is really smart when it comes to rotating your screen based on your phones rotation. If you rotate your phone to the right, the video is rotated to the right. If you then rotate to the left, a small button pops up that'll change the screen's rotation to the left.

iOS, on the other hand, is so freaking dumb!! When I'm in my bed watching YouTube and go full screen, it rotates the screen in the wrong direction. Dude my phone is turned towards the right, why the heck is the video rotated towards left? Now I need to flip my phone.
Keyboard
- Worst auto completion by far! Keyboards on android not only have extremely good auto suggestions but can also remember custom words which is really handy when you use english letters for Nepali words.
- Hard to place a cursor in the middle of a word. You need to slide the cursor.
- Copy / Paste / Select All are atrocious!
Ecosystem
- Premium ecosystem. Apps that were free on android were behind paywall on App Store. This isn't really on iOS thing but if people are gonna boast about the ecosystem it's fair to mention all aspects of it. Example: I used to heavily use Syncthing to sync files between my computer and phone (like music and ebooks).
  
  This isn't a problem for me anymore as now I have NAS (network attached storage) that I can connect to from the Files app. And now, I also don't mind paying for apps.
- As a self hoster I use a lot of open source applications. And I've noticed that usually these softwares do not have an ios app but they almost always do have an android as they can simply distribute the APK.
- You also miss out on the hacker culture of Android where you can try out lots of APKs that are not in the PlayStore. Example: Cracked Spotify.
Application call logs in the phone call logs.

Why the heck are my Messenger and Telegram call logs mixed in with my phone's call history? I honestly don't get this. Hate it when I need to call someone and I tap the call history and Telegram pops up!
Background activites

iOS has a really strict and biased policy against 3rd party apps when it comes to background activity. On android, Google Photos or Immich can sync your gallery in background but iOS doesn't allow that. You have to keep your apps open to upload your photos.

On the same note, the native Mail is so slow to sync emails.
Can't force quit apps. Oh this makes me furious!

Hate it when I open Files app and then it resumes from the last directory I visited (even after restarts). What makes me really mad is when the last location I opened was a remote file server and if it can't connect to the remote server it just freezes. And yup I can't force quit it. Restarting the phone doesn't help either.
Setting default apps for URLs.

I used to use a 3rd party app (Apollo) for reddit and clicking a reddit link would not open in that app. It always opens up in browser. Ugh! On Android you can configure this.
Can't decline calls sometimes - there's just no decline button - WTF! I'm sure there's a setting somewhere to enable this but the fact that I need to configure this in the first place is kinda wild!
Nit picks
- Notifications. I remember hating this a lot when I first started using iOS but honestly it's fine now.
- Battery. I know I can't really complain about the battery after using a used 5 year old model but still the battery really really sucks.
- I can't edit a number from my call history before I call it. Android has a pretty handy "Edit and call" feature.
- All the settings are in the settings app! Need to configure camera? Go to settings -> apps -> Camera.
- Alarm. On Android when you setup an alarm it tells you exactly when it's gonna go off. i.e. if you set an alarm for 6am tomorrow at 10pm today, a small popup notifies you that the alarm's gonna go off in 8 hours. It's a small touch but really helpful to be extra sure that you setup the alarm at the correct time.
- The weather app has a nice UI but the location search is really poor. Four out of five locations I search for - I can't find them.

Removed recent improvements

Some of the cons that I've decided to remove from the list because they have been fixed in newer iOS / iPhone upates.

Doesn't suggest contact as I type the number. Really ? I need to type in the entire 10 digits before you show me if it's in my contact? This comes handy when you get an sms for a missed call and it just shows the number.
Thunderbolt port. Type-C all the way!
FaceID as a rider. Face id works on masks and helmets on newer iPhones.

I'm not going to mention Siri in here because I don't really use it and I also didn't really use Google Assistant.

How I use SimpleAnalytics to monitor traffic on this blog

Sat, 07 Dec 2024 16:00:00 GMT

Analytics are super fun to look at. When you first integrate any analytics tool to your webiste - you can't stop refreshing the page cuz it's just so fun.

I personally have been using SimpleAnalytics to monitor this blog for, I think, almost 5 years now.

Few things that I really like about it are

It has an API
True to its name - it's in fact simple (I don't even want to remember the google analytics dashboard)
It allows you to bypass ad-blockers by serving the "tracking" js via your own subdomain. (I must clarify that no personal identifiable information is collected. This blog serves it via stats.adityathebe.com. I've written about this before)
Exporting data. I'm a huge proponent of owning your own data. I have a homelab where I run bunch of open source softwares in favor of proprietary softwares just so I can own my data.

I'm sure it has a few other cool features too (I can see they've added an AI feature), but these are my favorites.

While I like their UI, I very rarely use it. Instead, I use their API.

I have a telegram bot which presents today's analytics in response to a /sa command.

The source code to this bot is private as it does a lot of other things than just serving the analytics.

However, I can share the bit of code that deals with simple anlaytics

func (t *SimpleAnalytics) callAPI(options map[string]string) (*APIResponse, error) {
	r := &APIResponse{}

	req, err := http.NewRequest("GET", t.apiURL, nil)
	if err != nil {
		return r, err
	}

	q := req.URL.Query()
	q.Add("version", "5")
	q.Add("info", "false")
	q.Add("fields", "histogram,pages,countries")
	q.Add("timezone", "Asia/Kathmandu")
	for k, v := range options {
		q.Add(k, v)
	}
	req.URL.RawQuery = q.Encode()
	req.Header.Set("Api-Key", t.apiKey)

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		return r, err
	}
	defer resp.Body.Close()

	if err := json.NewDecoder(resp.Body).Decode(&r); err != nil {
		return r, err
	}

	return r, nil
}

func (t *SimpleAnalytics) DailySummary() (string, error) {
	options := make(map[string]string)
	options["start"] = time.Now().In(nepalLoc).Format("2006-01-02")

	apiResp, err := t.callAPI(options)
	if err != nil {
		return "", err
	}

	return t.renderDailySummary(apiResp)
}

func (t *SimpleAnalytics) renderDailySummary(r *APIResponse) (string, error) {
	tableString := &strings.Builder{}
	tableString.WriteString("<pre>")

	// Views by Page
	tablePageView := tablewriter.NewWriter(tableString)
	tablePageView.SetHeader([]string{"Page", "Views", "Visitors"})
	tablePageView.SetBorder(false)

	for _, p := range r.Pages {
		tablePageView.Append([]string{p.Value, fmt.Sprint(p.Pageviews), fmt.Sprint(p.Visitors)})
	}

	for _, h := range r.Histogram {
		tablePageView.SetFooter([]string{"Total", fmt.Sprint(h.Pageviews), fmt.Sprint(h.Visitors)})
	}

	tablePageView.Render()
	tableString.WriteString("</pre>\n\n<pre>")

	// Views by Country
	tableCountries := tablewriter.NewWriter(tableString)
	tableCountries.SetHeader([]string{"Country", "Views", "Visitors"})
	tableCountries.SetBorder(false)

	for _, c := range r.Countries {
		tableCountries.Append([]string{c.Value, fmt.Sprint(c.Pageviews), fmt.Sprint(c.Visitors)})
	}
	tableCountries.Render()

	tableString.WriteString("</pre>")

	return tableString.String(), nil
}

It's been serving me great so far. ~~A slight complaint I have is that they only email based login.~~ Wait, they do actually support passwords now (or have they always had this ...?).

SET LOCAL, Stay GLOBAL - A PostgreSQL Cautionary Tale

Fri, 06 Dec 2024 22:00:00 GMT

Postgres's SET command allows you to modify runtime parameters or create your own custom parameters.

my_database> SET my_domain.name = 'adityathebe.com'
my_database> SHOW my_domain.name
-- +-----------------+
-- | my_domain.name  |
-- |-----------------|
-- | adityathebe.com |
-- +-----------------+

With the LOCAL option, the parameter only lives within a transaction.

my_database> BEGIN
my_database> SET my_domain.name = 'adityathebe.com'
my_database> SHOW my_domain.name
-- +-----------------+
-- | my_domain.name  |
-- |-----------------|
-- | adityathebe.com |
-- +-----------------+
my_database> ROLLBACK


my_database> SHOW my_domain.name
-- +----------------+
-- | my_domain.name |
-- |----------------|
-- |                |
-- +----------------+

Alternatively, you can also view the parameter using the current_setting function.

my_database> SET my_domain.name = 'adityathebe.com'
my_database> SELECT current_setting('my_domain.name', TRUE)
-- +-----------------+
-- | current_setting |
-- |-----------------|
-- | adityathebe.com |
-- +-----------------+

Adding RLS

With this in mind, I began writing a new Row Level Security Policy.

ALTER TABLE config_items ENABLE ROW LEVEL SECURITY;

CREATE POLICY config_items_auth ON config_items
  FOR ALL TO postgrest_api, postgrest_anon
    USING (
      CASE WHEN (
        current_setting('request.jwt.claims', TRUE) IS NULL
        OR current_setting('request.jwt.claims', TRUE)::jsonb ->> 'disable_rls' IS NOT NULL
      )
      THEN TRUE
      ELSE (
          ...
      )
      END
    );

The policy relies on the request.jwt.claims custom parameter. If the parameter isn't set, the policy simply returns true - meaning the access is allowed. Additionally, if the request.jwt.claims parameter is set, the policy still allows access if disable_rls key is present on it.

After pushing this change (along with a whole lot of other changes in the PR...), our test suite went haywire. Random tests would sporadically fail with varying errors on each run. One commonly repeated error was this error

invalid input syntax for type json
DETAIL:  The input string ended unexpectedly.
CONTEXT:  JSON data, line 1:

The error came from a view using jsonb_object_agg. My first guess? The RLS policy was blocking rows, causing the view to choke on empty data. I deleted all the rows on another test db and the view worked fine however.

But let's step back a bit. I never set request.jwt.claims parameter so the policy could not have blocked any rows. As far as this test is concerned, RLS policy should practically be nonexistent.

In the entire test suite, there was only one test where I had set the request.jwt.claims parameter. And that test ran entirely within a transaction to avoid any side effects. The parameter was also set with a LOCAL option. Since the transaction would be rolled back the parameter should not persist outside of the transaction. Even if the transaction was commited, the LOCAL option should ensure that it there's no trace of that parameter.

This is where it gets even weirder - I disabled that test and the test suite passed consistently. This didn't make sense. How does disabling a test that runs entirely on a rolledback transaction have any effect on other tests? Something was escaping our carefully constructed sandbox, but what?

I also made sure that this test never runs in parallel with any other tests, but that didn't help.

The Cause

After a bit of a research I found this weird behavior of SET command. If you set a custom parameter and then unset it, it actually sets the parameter to an empty string!

Let's try querying a non existing parameter.

my_database> SHOW your_domain.name
-- unrecognized configuration parameter "your_domain.name"

my_database> SELECT current_setting('your_domain.name');
-- unrecognized configuration parameter "my_domain.namespace"

Both the calls error out.

If we pass in the optional second parameter to current_setting it returns NULL instead of throwing an error.

my_database> SELECT current_setting('your_domain.name', TRUE)
-- +-----------------+
-- | current_setting |
-- |-----------------|
-- | <null>          |
-- +-----------------+

Let's try setting the parameter in a transaction.

my_database> BEGIN
my_database> SET your_domain.name = 'example.com'
my_database> SHOW your_domain.name
-- +-------------------+
-- | your_domain.name  |
-- |-------------------|
-- |   example.com     |
-- +-------------------+
my_database> ROLLBACK

Now, when I query the parameter I am expecting it to behave the same way as it did before - i.e. error out or return NULL.

my_database> SHOW your_domain.name
-- +------------------+
-- | your_domain.name |
-- |------------------|
-- |                  |
-- +------------------+

Well there you go! A local parameter that was created inside a transaction has basically escaped and left a garbage in the session. I couldn't find this behavior documented in the official docs.

In fact, according to this stackoverflow answer, you cannot delete the parameter at all. Not with RESET as well. The parameter lives forever within the current session.

So the json error I was seeing was coming from the RLS policy as request.jwt.claim was an invalid json - it was an empty string.

Debugging disk usage On Linux

Sat, 19 Oct 2024 00:00:00 GMT

My pretty powerful desktop with a intel i9-14900k feels a bit laggy sometimes. A very identical Linux installation on my 9th gen i5 laptop never had this occasional lag.

The weird thing is that the CPU usage is pretty low when the system is lagging. Upon further investigation I found out that the CPU load on the other hand was pretty high!

Yes, your CPU could be chilling at ~1% usage and still be peaking on CPU load.

> cat /proc/loadavg

# 0.71 0.58 0.56 2/1605 22984

That was quite baffling to me at first but it does make sense now. CPU load takes I/O into account. If your disk cannot keep up with your CPU, processes have to basically pause & wait for the disk to catch up. This increases the CPU load metric.

On my 24 core machine, the CPU load goes as high as 35-40 when I fireup my development environment after initial boot. That includes launching

a Tmux session with 3 different go builds occuring simulataneously and about 6 docker containers
and also launching the web browser with 6-7 tabs.

CPU Load

CPU load represents the average number of processes waiting for or using CPU time.

"Each process using or waiting for CPU (the ready queue or run queue) increments the load number by 1. Each process that terminates decrements it by 1. Most UNIX systems count only processes in the running (on CPU) or runnable (waiting for CPU) states"

-- Wikipedia

A CPU load of 1 on a single-core machine indicates full CPU utilization. On a quad-core CPU, a load of 2 suggests 50% utilization across all cores. When the load exceeds the number of CPU cores, processes begin to queue, and the system beings to lag.

In my case, the fact that the CPU usage is pretty low suggests that the high load is caused by IO.

Tools to debug slow disk

2 tools seem to be the universal choice to debug disk operations.

iotop

iotop lists out all the processes and the amount of data those processes have read from or written to the disk.

sudo iotop -aoP

# Flags
# -a accumulative mode where iotop shows the amount of I/O processes have done since iotop started.
# -o only shows processes actually doing I/O
# -P shows processes instead of threads

This is helpful to figure out what processes are hogging the disk.

> sudo iotop -aoP
Total DISK READ :       0.00 B/s | Total DISK WRITE :     178.60 K/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:       0.00 B/s
    PID  PRIO  USER     DISK READ DISK WRITE>  SWAPIN      IO    COMMAND
   5380 be/4 aditya      556.00 K     33.71 M  ?unavailable?  node /home/aditya/projects/adityathebe.com/.cache/tmp-5353-NH1acrOpTEuF
   3238 be/4 aditya       76.00 K     31.57 M  ?unavailable?  brave
   3277 be/4 aditya      128.00 K     17.25 M  ?unavailable?  brave --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US
    464 be/3 root          0.00 B     15.47 M  ?unavailable?  [jbd2/nvme0n1p2-8]
  28731 be/4 aditya        0.00 B      3.20 M  ?unavailable?  nvim --embed .
    523 be/4 root          0.00 B    128.00 K  ?unavailable?  systemd-journald
   3293 be/4 aditya        0.00 B     88.00 K  ?unavailable?  brave --type=utility --utility-sub-type=storage.mojom.StorageService --lang=en-US
    222 be/7 root       1024.00 K      0.00 B  ?unavailable?  [khugepaged]
  30530 be/4 root         32.00 K      0.00 B  ?unavailable?  python /usr/bin/iotop -aoP

I have sorted the list based on Disk Write. As you can see, node is the currently running process with the highest disk write.

Unfortunately the biggest flaw with this tool is that if a process dies iotop removes it from the list. It would have been nice to be able to leave iotop running in the background for few hours and then inspect the output later. Due to this reason, this tool wasn't really that helpful to me and I couldn't find any other tool that does that.

iostat

iostat is another tool that gives you the current disk read/s & write/s info along with couple of other similar metrics.

sudo iostat -xm 1

# Flags
# -x extended statistics
# -m Display statistics in megabytes per second

# argument
# interval: the amount of time in seconds between each report.
# if 1 is supplied, the output refreshes every 1 second.

This tool is really helpful to get the instaneous disk bandwidth.

sudo iostat -m 1
Linux 6.11.4-arch1-1 (desktop) 	10/19/2024 	_x86_64_	(32 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.32    0.00    0.56    0.18    0.00   97.94

Device             tps    MB_read/s    MB_wrtn/s    MB_dscd/s    MB_read    MB_wrtn    MB_dscd
nvme0n1          43.24         0.70         0.40         0.00       2799       1594          0
nvme1n1           0.10         0.00         0.00         0.00         10          0          0


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00    0.22    0.00    0.00   99.53

Device             tps    MB_read/s    MB_wrtn/s    MB_dscd/s    MB_read    MB_wrtn    MB_dscd
nvme0n1          81.00         0.00         0.45         0.00          0          0          0
nvme1n1           0.00         0.00         0.00         0.00          0          0          0


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.38    0.00    0.25    0.03    0.00   99.34

Device             tps    MB_read/s    MB_wrtn/s    MB_dscd/s    MB_read    MB_wrtn    MB_dscd
nvme0n1          10.00         0.00         0.12         0.00          0          0          0
nvme1n1           0.00         0.00         0.00         0.00          0          0          0

Ultimately, neither of these two tools really helped by get to the bottom of the issue however XD.

Video Files Demystified

Mon, 14 Oct 2024 00:00:00 GMT

For the past couple of years I have been running Jellyfin on my homelab; think of it as a self-hosted Neflix. I host all of my movies & TV shows in there and then watch them from various clients - mobile app, TV & web app. It's a great piece of software - however I don't particularly use its webapp. The reason being - it doesn't play some movies very well. It stutters so much and the server cpu usage goes through the roof.

And that's why I prefer to watch the movies on Jellyfin from my media player instead of from the web app. To do that, I simply grab the stream URL from Jellyfin and play it on the media player as

mpv '<jellyfin-stream-url>'

# or
# vlc '<jellyfin-stream-url>'

I had very little idea why some movies wouldn't play well on the web UI. And interestingly, on the other hand, the TV app handles pretty much any video with ease.

Now, I wasn't totally clueless here. I knew that some videos needed to be transcoded with ffpmpeg and that pins the CPU to 100% (I don't have GPU attached so the transcoding is entirely performed on the CPU). But I didn't really know why that was required for some videos and not for others. And honestly, I didn't know what "transcoding" was to begin with.

This curiousity led me to learn quite a bit about video files and ultimately to write this blog post.

The two defining characterstics of a video file

The unusual thing about video files is that the file extension doesn't tell you anything about the video format.

When you encounter an .mp3 file, you know that the file contains an mp3 encoded audio data. Likewise, a .jpeg file contains a jpeg encoded image and a .json file contains json encoded data.

Aside: I know you can rename the file to have an incorrect extension, but let's assume that the file is named correctly with the appropriate extension.

However, with video files, the extension doesn't give you the entire picture about what's inside the file. Your media player could support one .mp4 file but may not support another .mp4 file. i.e. two different mp4 files may have been encoded with two separate codecs.

A video content thus has two distinct characterstics that defines it

the file type (commonly referred to as the Container)
the encoding algorithm (Compression / Codec)

Containers

The thing that we commonly refer to as the video format like mp4, mkv, flv are actually the just the file types. They are containers.

Think of them as different types of lunchboxes:

The container alone doesn't reveal its contents.
Some containers are like multi-compartment lunchboxes, capable of neatly organizing different "food items" (such as multiple audio tracks or subtitle files).
What the container does indicate is its capacity and structure.

Some popular container formats are: mp4, mkv, 3gp, wEBm, flv, avi, ...

Container formats differ from each other in various ways but here are some prominent defining characterstics

Characteristic	Description
Compression	Different formats use various compression algorithms, such as H.264, VP8/VP9, or lossless codecs like FLAC.
Metadata support	Ability to store metadata, including subtitles, chapter markers, and custom tags.
Audio and video tracks	Some formats support multiple audio and video streams, while others are limited to a single track.
Compatibility	Different formats have varying levels of compatibility with different devices, operating systems, or software applications.
Licensing and royalties	Container formats may be open-source (e.g. mkv), royalty-free (e.g. webm), proprietary (e.g. MP4, QuickTime,)

Codecs

Codecs are the algorithms that encode the raw video data and compress them. The output of a codec is what's put inside the containers. Codecs could also mean a hardware that is specifically designed to efficiently run the codecs.

Here are some of the most prominent characterstics of codecs to consider:

Characteristic	Description
Compression Ratio	Amount of data reduction achieved by the codec, with higher ratios indicating better compression.
Licensing	Whether the codec is open-source (free) or requires royalties or licenses from patent holders.
Complexity	Computational resources required to encode and decode the video, affecting device performance.
Color Space	Range of colors supported by the codec, impacting suitability for different applications (SD, HD, HDR).
Bit Depth	Number of bits used to represent each pixel's color information, affecting overall quality and detail.

Some popular codecs are:

Codec	Description
H.264 (AVC)	most popular and most well-supported codec
H.265 (HEVC)	More efficient than H.264 i.e. similar quality with lesser file size. However, has lesser support and it's patented. In fact, on windows, you need to purchase the codec for $0.99 from Microsoft - but there are free alternatives.
VP9	(Open source) developed by Google and YouTube uses this.
AV1	(Open source) relatively newer and the considered the future of video compression. Isn't supported widely.

Why it's important to be aware of the codec

For General Users: Understanding codecs helps you troubleshoot issues with video playback. By knowing which codecs are supported by your media player, you can diagnose and potentially fix problems when a video file won't play.
For Videographers: Being mindful of the codec used in your footage ensures that you can work seamlessly with your preferred editing software or share your content without conversion issues. For instance, if you record with HEVC but Davinci Resolve doesn't support it, you'll need to purchase the codec to edit your files.
For Video Providers: Choosing efficient and compatible codecs for your videos saves time and resources by avoiding unnecessary transcoding requests from clients. By encoding in a format that's widely supported, you can ensure that your content is accessible and easily playable on various devices and platforms.

Looking back at the original problem

To circle back to the original problem, the reason why some movies weren't supported by the Jellyfin Webapp was because those movies were encoded with HEVC.

HEVC encoded movie on Jellyfin

My web browser (and most browsers today) doesn't natively support hevc encoded media files. The Jellyfin server then has to convert the hevc encoded files to a format that the client supports. This process of converting a video file from one codec to another is called Transcoding.

Most people running Jellyfin servers pre-transcode necessary files into h264 so all clients can support it and the playbook is smooth. Or, they have run the transcode on the fly when the client starts streaming.

On the other hand, h264 encoded movies are natively supported by most browsers these days.

AVC encoded movie on Jellyfin

Here's a list of codecs and their support on different Jellyfin clients

https://jellyfin.org/docs/general/clients/codec-support/

I have a handy function on my .zshrc that displays the file container format & the codec for a given media file.

video_info() {
  echo "File Container: $(file -b --mime-type "$1")" && \
  echo "Video Codec: $(ffprobe -v quiet -select_streams v:0 -show_entries stream=codec_name -of default=noprint_wrappers=1:nokey=1 "$1")" && \
  echo "Audio Codec: $(ffprobe -v quiet -select_streams a:0 -show_entries stream=codec_name -of default=noprint_wrappers=1:nokey=1 "$1")"
}

Who's responsible in supporting a codec

The media player application is responsible in supporting a given codec.
The operating system however provides system libraries and drivers that support various video codecs and facilitate communication between software and hardware components. The application can either implement their own support for codec or use the OS's system libraries.
The hardware accelerates specific codec operations using GPUs, SoCs, or specialized A/V codec chips.

Understand Public Key Infrastructure & Certificates

Mon, 29 Jul 2024 22:00:00 GMT

This article is targeted towards newcomers looking to explore the foundations of PKIs.
It should be treated as the first chapter of PKI 101.
Readers are assumed to have a basic understanding of Asymmetric Cryptography & TLS.

Public Key encryption has enabled us to establish a secure channel for communication in an untrusted environment.

If Alice wants to send confidential data to Bob, she needs to somehow obtain Bob's public key and then encrypt the message with that public key. Only Bob can decrypt the message. And likewise, Bob encrypts the message with Alice's public key, and only Alice can decrypt it. A secure communication channel is thus established.

While confidentiality is undoubtedly achieved with encryption, authenticity isn't! Let me ask you this - how does Alice guarantee that the public key is actually Bob's? Cuz remember, they are in a potentially compromised environment. Who is to say that Eve - a malicious actor - didn't offer her public key to Alice claiming it to be of Bob's.

Shortcomings of Public Key Encryption

If Alice doesn't verify that the public key is Bob's then she'll happily encrypt messages with Eve's key falsely believing that she's encrypting it for Bob.

You see the problem we have in our hands right? Just obtaining a public key isn't sufficient. We need a trusted entity that can tie down the public key to a certain identity. If the trusted entity states that it is indeed Bob's public key then for all intents and purposes that is indeed Bob's public key.

Certificate Authority

On the web, Certificate Authorities (CA) are organizations that aim to be such trusted entities.

They do all the heavy lifting of verifying the identity and then issue a signed digital document called a "Certificate" which acts as proof of identity.

When an unknown actor provides us with a certificate that was signed by a CA we trust, we can mathematically verify that the certificate was indeed certified by that CA and deem the certificate valid. This eliminates a massive burden off of us in individually verifying the identity of every single public key we deal with. We trust one (or more) CAs and then get the benefit of having the thousands/millions of public keys identified.

In our example above, Bob requests a certificate from the Trusted entity and then instead of providing a public key to Alice, he provides her with the signed certificate. Alice can then extract the public key from the certificate after verifying that the certificate is valid.

Digital Certificates

In summary, a digital Certificate serves as a signed proof verifying that a public key belongs to a certain subject. CAs are responsible for the creation, issuance, revocation, and management of Certificates.

TLS Certificates

One of the most popular types of certificate is the TLS certificate where the subject is a domain name. A domain owner must prove

his/her ownership over the domain
ownership of the private key to the public key

to the CA.

Below is a TLS certificate for my website (stripped off for brevity). You can see that the subject is set to adityathebe.com.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0f:59:b4:d4:06:47:9a:1e:df:00:02:3b:48:6c:72:d5
        Signature Algorithm: ecdsa-with-SHA256
=====>  Issuer: C=US, O=Cloudflare, Inc., CN=Cloudflare Inc ECC CA-3
        Validity
            Not Before: Feb 24 00:00:00 2024 GMT
            Not After : Dec 31 23:59:59 2024 GMT
=====>  Subject: C=US, ST=California, L=San Francisco, O=Cloudflare, Inc., CN=adityathebe.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
=============>    pub:
                    04:7d:2f:37:6a:b6:09:08:e4:00:18:54:6a:10:95:
                    25:c2:5a:09:a6:de:72:77:fe:ae:92:09:47:05:88:
                    de:f8:9a:89:0d:46:90:a8:ba:5b:03:69:88:32:12:
                    f5:ec:5a:42:51:7d:1e:93:60:d2:95:0d:5b:fa:fd:
                    b6:fc:64:e6:75
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            ...

The CA that signed and issued this certificate is Cloudflare Inc.

When you visit my website over HTTPs, your browser is presented with this very certificate. If your browser or OS trusts Cloudflare Inc as a CA then it can trust the certificate. And if it doesn't, then it'll show a warning and probably not load the website at all.

Trust store

Alright! So far we've established the fact that CAs are entities that facilitate authenticity in public key cryptography.

Now comes the question of which CAs to trust. Do you personally need to choose a certain bunch of CAs from a marketplace? Are there any popular well-trusted ones?

Well, good news! The operating system you use has already done that for you. Additionally, web browsers also have selected their own set of CAs that they trust. These chosen set of trusted CAs are stored in a place called Truststore.

If you're on Linux see the contents of /etc/ssl/certs/ca-certificates.crt for the list of root certificates.

cat /etc/ssl/certs/ca-certificates.crt | grep 'BEGIN CERTIFICATE' | wc -l
148

As you can see, on my Arch Linux machine there are 148 root certificates. On my Ubuntu server, there are 137.

The set of CAs considered trustworthy isn't uniform across all OSs and web browsers. For example: the CAs trusted by Google Chrome may not be trusted by Firefox and vice-versa.

But why is that? Why is one CA trustworthy to one vendor but not to another?

Root Programs

Each vendor has outlined a set of criteria for CAs to be considered trustworthy for them. This criteria is defined in a root program. There are several major root programs:

Supplementary Materials

An overview of the gzip & zip file formats

Sat, 20 Jul 2024 12:30:00 GMT

I just finished listening to an episode of the ShipIt podcast - Tars all the way down - where Jon “gzip enthusiast” Johnson talks about gzip, zip & Deflate. The episode goes on to talk about efficiently & randomly accessing the bits of compressed data from container images.

This episode led me to scratch my long outstanding itch to understand - or at least acquaint myself with - the popular compression tools. Naturally, this post isn't going to be anything in-depth. I took a peek at the compression world and boy it's deep.

In the Linux world gzip along with tar is extremely popular. It wouldn't be inaccurate, in my opinion, to say that gzip is the defacto compression tool & tar is the defacto archive format. It's the format most software is distributed.

On the other hand, on Windows, it's probably zip.

Let's explore their differences.

gzip and zip are file formats, not compression algorithms. They can also mean the command line tools.

gzip can only compress one file at a time. So, how do we gzip a directory then? That's where you need an archiving tool to first archive all the files and directories inside the directory into a single file - the archive. And then the archive is compressed. As gzip compresses the archive as a single unit, it can optimize better and produce a more efficient smaller output.

In the words of the creator himself

"tar + gzip compresses better than zip, since the compression of the next file can use history from the previous file (sometimes referred to as a "solid" archive). zip can only compress files individually.

-> See comments: https://stackoverflow.com/a/20765054/6199444"

On the other hand, zip compresses one file at a time and then creates an archive. Since it's only compressing one file at a time, it doesn't do as good of a job in compression. Usually, gzip produces smaller output.

However, an advantage that zip offers over gzip is that you can browse the zipped archive without uncompressing the entire thing. zip files have a central directory at the end, which provides a list of the contents. This along with the fact that each files are compressed individually makes this possible. That's why you might have noticed, that on Windows you can simply open & browse through a zip file without uncompressing it first.

Compression algorithms under the hood

gzip uses the Deflate method for compression which in turn uses LZ77 and Huffman encoding compression algorithms.

While zip also uses the same two algorithms, it can replace them with a wide variety of other algorithms that offer different balances between the speed of compression & the efficiency of the compresion. Example: zip can use LZMA instead of LZ77 and produce smaller output but at the cost of compression speed.

The ZIP format supports several compression methods:

Source: https://stackoverflow.com/a/20765054/6199444

0 - The file is stored (no compression)
1 - The file is Shrunk
2 - The file is Reduced with compression factor 1
3 - The file is Reduced with compression factor 2
4 - The file is Reduced with compression factor 3
5 - The file is Reduced with compression factor 4
6 - The file is Imploded
7 - Reserved for Tokenizing compression algorithm
8 - The file is Deflated
9 - Enhanced Deflating using Deflate64(tm)
10 - PKWARE Data Compression Library Imploding (old IBM TERSE)
11 - Reserved by PKWARE
12 - File is compressed using BZIP2 algorithm
13 - Reserved by PKWARE
14 - LZMA
15 - Reserved by PKWARE
16 - IBM z/OS CMPSC Compression
17 - Reserved by PKWARE
18 - File is compressed using IBM TERSE (new)
19 - IBM LZ77 z Architecture
20 - deprecated (use method 93 for zstd)
93 - Zstandard (zstd) Compression
94 - MP3 Compression
95 - XZ Compression
96 - JPEG variant
97 - WavPack compressed data
98 - PPMd version I, Rev 1
99 - AE-x encryption marker (see APPENDIX E)

Formats 0 & 8 are by far the most used ones - 8 more so.

Use cases

gzip (along with tar) is more common in the Unix world than zip. tar preserves all of the Unix file attributes, whereas zip was not designed to do that.

gzip also supports decompressing data streams which is why it's popular in network transfer. HTTP widely uses gzip.

zip is preferred when portability and random access are required.

No, your sudo password is not incorrect

Thu, 18 Jul 2024 23:50:00 GMT

I've ocassionally run into this situation where I just cannot get the sudo password right no matter how many times I retry it. Just to make sure, I carefully press one key at a time instead of typing fast in my regular pace. However, Linux still tells me my password is incorrect!

I've also tried typing my password on a text editor where I can see what I'm typing and it's flawless everytime. Copy pasting a valid password doesn't help either!

I've blamed my keyboard and myself for misconfiguring my linux installation for this and I used to simply restart my system. And after a restart, it would work.

Silently locking out the user

As it turns out, a few incorrect password attempts - I believe 3 - will lock you down for a period of time. But Linux doesn't tell you about that. It just tells you to "try again". That's the same thing it tells you when you do provide a wrong password.

To know if you're currently locked out you need to explicitly look for it.

faillock --user <user>

~/❯ faillock --user aditya
aditya:
When                Type  Source                                           Valid
2024-07-18 11:40:33 TTY   /dev/pts/11                                          V
2024-07-18 11:40:36 TTY   /dev/pts/11                                          V
2024-07-18 11:40:38 TTY   /dev/pts/11                                          V

If you see a record there, that means you have been locked out. Fortunately, you don't have to wait until the ban goes away. With the same command, you can lift the restriction.

~/❯ faillock --user <user> --reset

I found it pretty surprising that it doesn't even take sudo access to unlift the ban (If it did, this would all be pretty useless wouldn't it?).

Disclaimer: I'm on Arch (btw), and this may not be applicable to your system.

How to setup debugger for Golang in VS Code

Tue, 09 Jul 2024 23:40:00 GMT

VS Code comes with a native debugger by default. If you have Go's debugger delve installed on your system then you can simply go to the "Run and Debug" view and press the "Run and Debug" button. VS Code will then start your program and attach the debugger.

This approach serves fine for simple use cases. However, it's likely that your application needs some arguments or flags or environement variables. To pass those in, you can create a .launch.json file for vscode to configure.

Create `launch.json` file

The launch.json file in VS Code is a configuration file that defines how the debugger should run your application. It allows you to specify various settings such as the program to debug, command-line arguments, environment variables, and other debug-specific options. You can create the file manually in .vscode/launch.json or you can let VS Code create it for you.

Besides the "Run and Debug" button you should be able to see a create a launch.json file link. Select "Go: Launch Package" option and the file should be created.

If your application accepts a --config flag and an ENV environment variable, you can configure your launch.json file as follows

{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Launch Package",
      "type": "go",
      "request": "launch",
      "mode": "auto",
      "program": "${fileDirname}",
      "args": ["--config", "/tmp/config.yaml"],
      "env": {
        "ENV": "dev"
      }
    }
  ]
}

Attach debugger to a running process

I personally do not prefer to run applications from VS Code's debugger. Instead, I run all the applications in the terminal and then attach the debugger to the running process.

Let's setup launch.json to attach the debugger to a running process. You can delete the launch.json file and recreate it again. This time, choose the "Go: Attach to Local Process" option.

{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Attach to Process",
      "type": "go",
      "request": "attach",
      "mode": "local",
      "processId": 0
    }
  ]
}

This is the default configuration generated by VS Code. The only change we will need to make is the processId field. While it's named processId, it can also take in the name of the process. Say you're running a go binary named 'my-application', you can set the processId to my-application. We'll also change the configuration name to 'my application'.

{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "name": "my application",
      "type": "go",
      "request": "attach",
      "mode": "local",
      "processId": "my-application"
    }
  ]
}

That's all the configuration we need to do. Now, if you go to the "Run and Debug" view again, you'll see "my application" in the dropdown.

Select it and press the play button to it's left.

Personally, I use the keyboard shortcut Ctrl + p to bring up File Search modal. And then I typ in debug and a character.

Things to keep in mind

1. Building go binary with debug symbols

For best debugging experience, you need to build the binary with debug symbols. This will turn of compiler optimizations and enable debug symbols. To do so, you need to pass in the following flag

gcflags "all=-N -l"

# example
# go build -gcflags "all=-N -l" .

2. yama

By default, Linux kernel will not allow a process to attach to another process if it's not running as root. Due to this reason, attaching a debugger to our running application will not work.

To fix this, we need to modify the ptrace scope setting in the Linux kernel's Yama security module. The ptrace scope setting determines which processes can use ptrace-based debugging tools on other processes. The value 0 sets it to the least restrictive mode, allowing any process to ptrace any other process.

sudo su
echo 0 > /proc/sys/kernel/yama/ptrace_scope

Setting this to 0 is often necessary for certain debugging scenarios, including attaching a debugger to a running process. However, it's important to note that this reduces system security, so it should be used cautiously and typically only in development environments.

Setup DNS Over TLS with systemd-resolved & Next DNS

Sat, 11 May 2024 00:00:00 GMT

I've been using Adguard home for a few years as my DNS resolver. It runs in a virtual machine in my homelab. Some of the essential benefits it provides me are

Adblocking (ofcourse)
Privacy (I use quad9 and clouflare as the upstream DNS over HTTPS). My ISP sees nothing.
Bypass censorship (unblock 'DNS' banned sites in the country)
Monitoring (Logs & Usage overview. I don't use this a lot, but incase I need it - it's there)
DNS overwrites (I use it for a couple of my own websites that I frequently visit)

My Adguard Home instance

DNS Rewrites

I have setup my router to use the Adguard Home instance as the upstream DNS server which means every devices connected to my network get adblocking and censorship bypassing benefits out of the gate.

This has been a fantastic solution so far. The only downside here is that when I'm not on my network I lose all those benefits.

For this reason, I wanted to look for an alternative DNS solution that would work on whichever network I connect to. This means I needed a local resolver that would connect to a public DNS server over HTTPs or TLS.

systemd-resolved as the resolver

I had some experience using systemd-resolved before, so I looked at it as the first option. Currently, I have a bare minimum setup where Network manager sets up the DNS server on the machine.

Fortunately, network manager & systemd-resolved work together prefectly. The only change I needed to make on Network manager side was to tell it to use systemd-resolved as the DNS resolver

> cat /etc/NetworkManager/conf.d/dns.conf

[main]
dns=systemd-resolved

systemd-resolved was also disabled on my machine, so I had to enable it

sudo systemctl enable systemd-resolved
sudo systemctl start systemd-resolved

At this point, systemd-resolved was serving the DNS requests.

> cat /etc/resolv.conf

# Generated by NetworkManager
nameserver 127.0.0.53
options edns0 trust-ad

Configuring DNS over TLS

systemd-resolved doesn't seem to support DNS over HTTPS (DoH) but it does support DNS over TLS (DoT) which is fine with me.

DoH is the preferred option over DoT since the former is indistinguishable from a normal HTTPs connection whereas the later is easier to notice & hence censor because it works on port 853.

To configure the upstream DNS server, I simply edited systemd-resolved config and enabled DNSOverTLS.

# /etc/systemd/resolved.conf

[Resolve]
DNS=1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com
DNSOverTLS=yes
Cache=yes

Troubleshooting slow DNS responses

I could tell the DoT was working because I was able to access cencsored sites. However, the surfing felt noticeably slow. Sites that used to open in few milliseconds were now taking about 10 seconds (or more) just for the first contentful paint. Something was off ...

sudo resolvectl log-level debug

I enabled verbose logging on systemd-resolved and then tailed the logs

journalctl -u systemd-resolved -f

I could see a lot of Connection refused error and systemd-resolved would just use my router as the DNS server.

Connection failure for DNS TCP stream: Connection refused
Retrying transaction 10768.
Cache miss for 0pointer.de IN A
Transaction 10768 for <0pointer.de IN A> scope dns on wlp2s0/*.
Using feature level TLS+EDNS0 for transaction 10768.
Sending query via TCP since UDP isn't supported or DNS-over-TLS is selected.

Reference GH issue

I checked the status of systemd-resolved and I could see that the network interface was still pointing to my router as the DNS server even though the global config was pointing to the cloudflare DNS server.

> resolvectl status

# This text is altered as I'm now I'm using a private DNS server

Global
           Protocols: +LLMNR +mDNS +DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: foreign
  Current DNS Server: 1.1.1.1#cloudflare-dns.com 9.9.9.9#dns.quad9.net 8.8.8.8#dns.google 2606:4700:4700::1111#cloudflare-dns.com 2620:fe::9#dns.quad9.net 2001:4860:4860::8888#dns.google

Link 2 (enp5s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
Current DNS Server: 192.168.254.254

I think I just needed to point my lan interface to use the same DNS server.

I found this section quite confusing. resolvectl status shows that my LAN interface is using my router as the DNS server, however when I make a DNS query, it's still served by the cloudflare servers ...

For now, I've configured Network manager IPv4 & IPv6 method to point the DNS server to 127.0.0.1.

NextDNS

Cloudflare DNS works pretty well but now I've lost some of the benefits I had with an Adguard Home setup. Mainly, Ad blocking, DNS logs & privacy (cloudflare still sees all my dns queries although not as big of a deal as it can't really point back to my identity).

I gave NextDNS a try which I suppose is kind of like your own private Adguard Instance on the cloud. It has a free plan and offers

logs
analytics
dns rewrites
safe browsing
ad blocking

NextDNS Analytics

NextDNS Traffic Analytics

I updated systemd-resolved & networkd configs to use my private next dns resolver.

So far it looks amazing! Although I'm not sure if the 300k req/mo is going to be sufficient for me. I guess we will find out ...

Flying with NixOS

Sat, 09 Mar 2024 14:30:00 GMT

NixOS setup progress

So I finally fixed the previous issue with NixOS. I now have a remote NixOS vm that's just sufficiently configured to do some actual development work. I haven't really done any development yet because the VM only has 2GB of RAM and my proxmox machine is maxed out in memory. I ran some go builds and it took a while but it was a success.

I do notice some occasional lags while I'm ssh'ing into the vm. There's about 1% packet loss which is a lot! It could be because I have 4 different wifi networks at home!

I'll try to bump the RAM this week and maybe get an ethernet dongle and that should be enough to get a good stable connection to the VM.

Upgrade Immich

On my homelab, I run an Immich server. It's the single most important application that I host. If you're unfamiliar with it, it's basically a self hosted Google Photos alternative.

The application is really really good but it's still in its early development phase and introduces breaking changes often. They do have very good notes on every release though and I haven't really faced any problem upgrading it so far. Except, this time I did.

For some reason, I happen to be running postgres 16 while the recommended version is 14. The recent breaking release needed me to upgrade from 14.1 to 14.2 and that's when I realized I was running the wrong version of Postgres this whole time. There where some hiccups but I don't exactly remember what it was. It had something to do with the pg_vector extension. Something about that extension was changed in the Postgres 16 release.

It took a while - about 30 minutes, I think - but I managed to get it up and running. I'm still on postgres 16 though.

Dune Two

I watched Dune Two in theater and it was soo soo good! I had very little expectation going in since I didn't really enjoy the first one.

I had almost entirely forgotten anything about Dune. 30 minutes before the movie, I watched a YouTube refresher video and that made such a huge difference! I rate the movie 9/10 or even 10/10 if I'm being generous. The next day I watched Dune, the first part again, and that was equally as good. I can't wait to watch the next part.

I keep a log of all the movies I watch in Movary. Check it out.

Learning Rust

I gave Rust a shot this week. Before this, my only experience ever writing Rust was when I was applying for a role at fly.io. It was about a 100 lines of code and pretty much just unit tests.

I basically followed the official Rust book and got through the first 2 chapters. It was actually quite straightforward. Probably because I didn't get into the exciting part of Rust yet.

Actually, the use of traits in the guessing game chapter was confusing now that I remember it. I still don't get it but I think I'm supposed to not get it just yet.

Macros seemed really cool. I don't understand what the println! macro offers over a print function. What can that macro possibly generate at compile time that it makes it more efficient than a function? And the use of macros in sqlx was mind blowing! The fact that the library performs sql calls during compile time to guarantee that the sql queries are correct was something I had never heard before! There's so much happening during compile time and as a Go programmer, the compile time in Rust scares me.

I have heard that macros were quite complex and people don't really use it but that doesn't seem to be the case since there's a whole sql library that essentially works at the macro level. Also, No Boilerplate seems to really love it!

Mapping CAPS key to CTRL

I have mapped my CAPS key to the Control key. A lot of people map it to the Escape key as well. The reason behind this is that the CAPS key is much more accessible than the ctrl key; it's right below your left pinky and on the home row. Commonly used key chords like Ctrl+C, Ctrl+V are easier to press.

This is definitely going to take me some time to get used to it. In fact, I had even forgotten that I had this mapping turned on yesterday and just remembered it while writing about it.

I used xmodmap to setup the mapping. It required me to create a ~/.xmodmap file with instructions for the mapping. It wasn't quite comprehensive because you need to deal with key codes and key symbols. Undoing the map was also a pain because xmodmap doesn't keep a state of the mapping.

setxkbmap seems to be the modern and user-friendly way of doing it. I was able to reset the mapping I had messed up using setxkbmap -option us. And then the actual mapping is done using

setxkbmap -option ctrl:ctrl_modifier

This line goes into my ~/.xinitrc file.

Building an HTTP server from scratch - CodeCrafters

I came to know about CodeCrafters through Jon Gjengset. It has a handful of catalogs where you build popular softwares like Redis, Git, etc from scratch.

I tried the Building HTTP server course and it was quite fun. I din't really like that there were hints just below the course. I want to struggle and find the solution myself. Sometimes you just scroll down and accidentally see a comment that gives you some hint about the solution.

Besides that, the platform looks really good. I'll probably do a stream in future with one of the courses.

Developing on a NixOS VM in Neovim

Sat, 02 Mar 2024 12:40:00 GMT

Website updates

I added 2 new pages to my website & also exposed my movary instance.

/now

I've been seeing a /now page in quite a few blogs these days. The /now page, unlike the /about page, is more frequently updated.

/now is a more personal look at what people are doing right now, what they care about, and what they could use help with.

- https://aboutideasnow.com/about

/links

/links is another page that's quite common these days. People put all kinds of things in their /links page - like links to podcasts they listen to, their favorite books or blogs, etc.

I'm not quite sure what I should put in here but I mainly added this page to share the movary instance. It's a web app that's currently running on my homelab but exposed to the internet via Cloudflare Tunnel.

Developing on a VM

I've been wanting to develop on a VM for a over two years now. The goal is to have a beefy desktop at home and ssh into it for all development purpose from my laptop.

Developing on a VM means you can have a fairly basic laptop and still run huge projects. It can be kept in one corner of your house so you don't have to deal with noise. And remote development shouldn't be problem with Tailscale.

What about when I'm working away from home and there's a power outage? Who's gonna turn the PC back on? Well, turns out you can configure the BIOS to automatically turn on when there's power. You don't need to login because systemd turns on the ssh daemon automatically. This is how one of my coworkers used to work.

My laptop is getting old now and its fans are driving me nuts. The keyboard also gets quite hot but I rarely use it. A thin fanless laptop with a good battery backup developing off of a beefy nixos vm would be my dream setup for now.

Mt.lynch's homelab for all developerment https://mtlynch.io/building-a-vm-homelab/
"Between VSCode, WSL, and Intel’s latest desktop chips, I’ve been living with a PC that runs my programming tests faster than an M3 Max, and generally feels like a completely viable alternative to macOS." https://t.co/tP4UQNk1Ss
— DHH (@dhh) February 25, 2024

Nixos

I learned about NixOS a few weeks ago and I've wanted to migrate to it ever since. It's not the most friendly OS and it has the worst in the business error messages.

To test developing on a VM, I wanted to deploy it in a proxmox vm and see how that turns out. However, I haven't been able to deploy it and no one seems have the solution to the issue. Installing it is fine but migrating to flakes is where I'm stuck right now.

https://mtlynch.io/notes/nix-first-impressions/

Self Hosting Podcast

I installed AudioBookShelf this week to be my ultimate Podcast player. It looked great at first but turned out it has some fatal issues.

Can't browse the episodes.

ABS is fully local. That means discovering new podcasts through it is not possible. While I can search for podcasts I can't really see/listen to the episodes without first subscribing and then manually downloading episodes.
You can't download episodes from mobile

This was the fatal flaw that turned me away from ABS. If you add a podcast, it'll only download future episodes! You have to configure it to download past episodes. And worse, that's not possible from the mobile app.
Doesn't have an iOS app right now.

There's a beta release but it's over the limit.

I have installed AntennaPod which I think will be my ultimate player. I'll also try out podfetch or podgrab next week.

Neovim

This week I got a lot done with neovim. I did two days of development entirely on Neovim. It's fun but I'm still not as productive with it as I am with VSCode. But, I certainly feel much more comfortable now. In fact, I'm writing this journal in Neovim right now.

Things VSCode does better

Find and replace from all files in the workspace
Multi cursor
AI code completion
Discovering new plugins
Installing new plugins

Vimimum browser extension

This is an extension that lets you use vim bindings on your browser. I had tried this before but didn't like it as it interferes with your normal keybindings. Example: on YouTube, pressing "F" doesn't go full screen anymore. You have to get into the "Insert mode" and then press F. Also, the "Insert mode" has a floating bar on the bottom left panel that stays up when you're in insert mode; so after you press F you need to exit out of the insert mode again.

Ctrl+C to copy also doesn't work in visual mode as that's default for ESC binding. I've now disabled vimimium on youtub.com.

It's been an ok experience so far & it doesn't get in your way most of the time so I'm going to continue using it.

Vim setup series by TypeCraft

Watched this really good series by TypeCraft. The guy's really funny and you should check him out.

https://www.youtube.com/playlist?list=PLsz00TDipIffreIaUNk64KxTIkQaGguqn

Vim Plugins & exploration

Better escape is pretty cool

When I was starting out with vim, getting out of the insert mode was a pain because I had to press ESC key which isn't very accessible. You need to lift your hand for that and I'm realizing that this sounds like a first-world problem while I'm writing it. Nonetheless, it's still a problem.

Then I started using Ctrl+C which is much more easier but still not the best keymapping.

This week I found out about better escape which lets you exit insert mode by pressing jj in quick succession. AstroNvim ships this plugin be default so I got it for free.

This was definitely my favorite vim finding of the week.

Telescope for other stuffs

I use telescope for finding files and greping files but it seems to have lot of other cool use cases.

<leader>gt for git status and easier access to the diffs on the right panel
<leader>gb to show all the git branches was coool
similary, <leader>gc to show all the git commits was also pretty nice.

Preventing logictech from waking up

I have a MX Master mouse which has a USB dongle that needs to be plugged in to my laptop for wireless connection. It's been a pain so far to remember to plug it out when I suspend my laptop because it wakes the computer.

I added this udev rule and now all's good.

https://wiki.archlinux.org/title/Udev#Waking_from_suspend_with_USB_device

ACTION=="add", SUBSYSTEM=="usb", DRIVERS=="usb", ATTRS{idVendor}=="046d", \
  ATTRS{idProduct}=="c548", ATTR{power/wakeup}="disabled", ATTR{driver/1-1.1.1.4/power/wakeup}="disabled"

Bcrypt 72 bytes limit

When I saw this tweet, my immediate reaction was that this limitation is probably coming from the VARCHAR(72) field on the database and that sha256 hashing the password before bcrypting it would solve it.

Source https://x.com/_JohnHammond/status/1762360967844012418

Interestingly, there was a different genuine reason behind it.

I learned that bcrypt has a 72 bytes input limit! If your password is longer than that then the remaining bytes are simply discarded.

Sha256 hashing beforehand doesn't help because it outputs 256 bits (just 32 bytes) and the 72 bytes entropy that your password initially had gets reduced tremendously. https://stackoverflow.com/a/16597402

Whether the reduction is significant or not, it certainly doesn't help those who want to use long passwords.

Vitess

I watched this really interesting interview with Sugu Sougoumarane where they talk about Vitess - a mysql proxy that enables horizontal scaling with sharding.

Interesting bit:

At the 42 minute mark, he says that YouTube was initally built outside of the Google Monorepo and then the YouTube build was pulling code from his laptop's home directory (which was mounted to Google storage).
Vitess got its name from a japanese Anime. The word itself is derived from the French word "Vitesse" (meaning speed).

DevOps Reimagined

System Initiative is a company that's rethinking devops. Not sure how much I'm buying into it but it definitely looked interesting.

https://www.youtube.com/watch?v=zyEOYl23pd8&t=252s&pp=2AH8AZACAQ%3D%3D

I think their greatest asset is their mock up of cloud services like AWS for instance. You can run your infra in their mockup as a precheck before deploying which tremendously speeds up the feedback loop and makes testing changes much much faster & safer.

All about Git & Unix

Sat, 24 Feb 2024 23:00:00 GMT

I can't believe it's been 3 years since my last weekly journal! It felt more like a year. Anyways, here's what I've learned so far this week.

Git

Git configs

This week I added a tonne to my git config and it all started when I read this blog post by Julia Evans. While I enjoyed a lot of those, my favorite one was

[push]
	autoSetupRemote = true
	default = current

At work, I spin off git branches every time. Small fixes, chores & features are all in their own branch. The first time you push a new branch to the remote, you need to set the upstream

git push -u origin <new-branch>

You're used to just doing git push and it's annoying to see this error every once in a while. With the above config, your new local branch is automatically mapped to a remote branch with the same name!

Inside git

It always helps to know your tools better!

Scott Chacon's awesome talk on git

Scott Chacon is the co-founder of GitHub who's currently working on GitButler. His talk on git at FOSDEM 2024 is honestly one of my favorite tech talks. I just love geeks (or nerd ... what's the difference?) who are also great at public speaking and add a lot of humor 😁. The information/duration density in this video is pretty high!

I'm not gonna write a lot about it here as I highly suggest you to watch the video but here are some of the highlights

Git branch better https://gist.github.com/schacon/e9e743dee2e92db9a464619b99e94eff
Force with lease (git push --force but better)

Salaries made public

A list of compensation history of software engineers.

I found out all these websites (except Xe Iaso's) through jvt.me who I discovered through the Changelog podcast.

I think it's pretty cool but I'm not sure if I'll ever do it myself. 1% of me thinks it's interesting to share but the other 99% thinks that this will do more harm than good.

SSH Signing key

I recently added a new GPG key as a signing key to my GitHub account. If you're not aware of this, git has a feature to sign a commit which tells that the commit made from your email was actually made by you. Otherwise, I can just as easily set the commiter's email to YOUR email in my git config and and then push that commit to GitHub and GitHub will associate that commit to your account. By signing the commit, I can guarantee that the commit was made by me.

Once you configure the signing key locally in git, you also need to tell GitHub that you're gonna be signing with that key. Once that's setup, GitHub will show "verified" status in the commit history.

Verified commits in Github

What I didn't know was that you can use a SSH key to sign commits. This means I can use the same SSH Key to both authenticate & sign commits. The benefit of this is, to me, is simply that I just have one key to maintain & I can delete the GPG key.

People seem to not like the idea of using SSH keys to sign though as ssh keys are supposed to be ephemeral while the signing keys are supposed to be long lived. But no one says you can't keep your ssh keys for a long term.

Unix Magic in Unicorn

I finally caught up to this video on Unix Magic by Thorsten Ball. It's being sitting on my YouTube's "Watch later" playlist for a while now.

In this video, he talks about multi process workers in the Unicorn web server. Things I learned

IPC (inter process communication) through Unix Pipe:

As a Linux user, I interact with unix pipes numerous times daily. But I never imagined that it was a common way to communicate between processes.

`sockets` & `select`:

Sockets are one of the fundamental features to working with anything networking related in UNIX. Life cycle of sockets

socket
bind
listen
accept

select seems to be very similar to the select keyword in Go. In fact, Go might have probably derived the term from Unix. Basically, a select syscall allows us to accept from multiple sockets. i.e. if you have multiple sockets, accept on of the sockets blocks until it receives a message. No other sockets can be accept at the same time and select enables exactly that.

Signal

I'm not sure I fully understood this section but the interesting bit here was how signals are handled by Unicorn - or more specifically asynchronous signals. Techniques like self-pipe and queuing were used but I didn't fully get it.

Some other cool use cases of signals were

Scaling workers using TTIN signals was also pretty cool.
Hot reloading with USR2 signal.

Dockerfile isn't the only way to create Container Images

Sat, 24 Feb 2024 00:50:00 GMT

On a recent episode of the ShipIt Podcast, I learned that container images can be built without docker! This may be pretty obvious to someone with a foundational understanding of containers, but to me it was news.

To a lot of people, container images are synonymous to docker images. You have to have a Dockerfile and then you need to build it - which can take quite a bit of time to download a bunch of things - .... and finally ... you have an image, right?

Wrong

I mean that's also true but that's not the only way to get a container image.

Just like a Dockerfile, Podman (an alternative to docker) has its own Containerfile. I had heard about Podman before but never gave it a shot. There seems to be another tool - buildah which can also build container images without using Dockerfile. Chainguard builds a large host of images with Terraform and apko.

This tells us that Docker and Dockerfile is just another way to build a container image. So what exactly is a container image then?

It turns out that it's simply a bunch of tarballs.

It's actually several other things as well. The image also has the details of

what command to run after starting the container
what ports to expose
what env vars to set

These details and many other details are kept in a variety of manifests as defined in the spec. Example: Here's an image manifest file

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7",
    "size": 7023
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:9834876dcfb05cb167a5c24953eba58c4ac89b1adf57f28f2f9d09af107ee8f0",
      "size": 32654
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b",
      "size": 16724
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736",
      "size": 73109
    }
  ],
  "subject": {
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270",
    "size": 7682
  },
  "annotations": {
    "com.example.key1": "value1",
    "com.example.key2": "value2"
  }
}

But in the summary, the image is in two parts

the Filesystem
a collection of metadata

Building a Container Image

I don't know about you but this terminology subconsciously made me think that an image is a byproduct of a build process. Kind of like how you need to build source code to get a binary. You don't just right binary by hand, right?

But that's not the case. Since images are just a bunch of files, you can, in fact, ~~build~~ create one by hand. An image can be as simple as a folder with a single static binary in it like this hello-world image by Docker. Or it can even be an empty folder like the popular scratch image.

In hindsight, I feel silly to have not known about this. I am aware of OCI & I've also watched Liz Rice build container from scratch but I just didn't put 1 and 1 together - dang it!

References

Thanks

cycode.com for the featured image

My Backup Strategy

Sat, 03 Feb 2024 20:55:00 GMT

In 2023, I started dabbling with self hosting applications. It all started with a plex server and an SMB share acting as a poor man's NAS. After multiple iterations, I am now managing about 20-30 self hosted applications. Self hosting applications is nice because you know you own your data. It's a blessing but also a curse because now you have to manage that data as well.

When you use a cloud provider, the risk of data loss is practically zero. Whether they'll misuse your data or guarantee access to your data is a different story. But in terms of keeping your data from losing, they've got you covered.

3-2-1 backup strategy

I came across this strategy through this video by Jeff Geerling. It stands for

3 copies
in 2 different media
and 1 offsite backup

The offsite copy does not have to be on the Cloud.

There is another 3-2-1-1-0 strategy that extends on this where the additional 1 and 0 mean - 1 offline backup and a recoverable backup with 0 errors. With Restic, I do get to verify that my backups are not corrupted and can be recovered. However, I do not have an offline backup (yet).

Broadly speaking, I would categorize my backups into two types

Computer backups
Media backups

Computer backups are important, but the media backups are of far more importance to me as those cannot be recreated in case of data loss. Regardless, I have a similar backup plan for both of them.

Computer backups

I use Restic to backup all my configs and some other directories on my system to a TrueNAS server. You can find the backup script here. These are the directories I'm currently backing up with a few exclusions.

/etc
/root
/boot
/home/gunners/.config
/home/gunners/.local/share

Some sofwares, for whatever reason, have made the decision to use the XDG_CONFIG_HOME as a dump to put all the application data (i.e. logs, cache, session, ...). My XDG_CONFIG_HOME is 16GB right now of which more than 90% is just from Brave, Slack, Discord and VS Code.

~/.config disk usage with Diskonaut

Which is why there are some exclusions in the backup script.

# https://github.com/adityathebe/dotfiles/blob/master/.local/share/restic-backup/excludes.txt

/home/gunners/.config/asdf
/home/gunners/.config/BraveSoftware
/home/gunners/.config/Chromium
/home/gunners/.config/Code - OSS
/home/gunners/.config/Code
/home/gunners/.config/code-server
/home/gunners/.config/discord
/home/gunners/.config/Slack
/home/gunners/.config/go
/home/gunners/.config/gops
/home/gunners/.config/gcloud/logs

/home/gunners/.local/share/go
/home/gunners/.local/share/nvim
/home/gunners/.local/share/nvim.bak
/home/gunners/.local/share/TelegramDesktop/
/home/gunners/.local/share/.codeium/

/root/go

Anyways, this backup script is run twice a day with a cron job

# Restic backup
00 11,22 * * * /home/gunners/.local/share/restic-backup/backup.sh

Then, on the TrueNAS server I have a cron job that replicates the system backup to Backblaze B2 using rclone. This cron job runs twice a week and is set up using an Ansible playbook that you can find here.

Why not directly back up to Backblaze B2 ?

So, intially I was indeed directly backing up to Cloud. However, the advantages of not directly backing up to a cloud provider are

backups are faster (honestly not a big deal since it's a background job anyway)
but the more important reason is - the fact that I can perform the integrity check of the backup without needing to download the whole thing from the cloud. This saves time & money.

Media backups

Media backups include all the photos, videos and my music library. The NAS is the production copy for my media. So, to have another local copy on a different media, I simply backup to an external HDD with Restic.

I wouldn't say I love this approach because I need to have the external drive plugged in at all times to the NAS and also doing lots of IO operations through a USB is probably not a good idea (don't quote me on this one) - but hey, it works. And the alternative would be to have another backup system that would cost equally as the main NAS system.

The previous ansible playbook also sets up a systemd timer on the NAS to perform daily backups to the external drive.

cat restic-backup.service

[Unit]
Description=Restic backup

[Service]
Type=oneshot
User=root

CPUQuota=100%

Environment="RESTIC_REPOSITORY=/home/admin/seagate/backups"
Environment="RESTIC_PASSWORD=***************"


ExecStartPre=/usr/local/bin/restic check
ExecStart=/usr/local/bin/restic backup --verbose /mnt/mega/aditya
ExecStart=/usr/local/bin/restic backup --verbose /mnt/mega/photos --exclude="/mnt/mega/photos/immich/thumbs" --exclude="/mnt/mega/photos/immich/encoded-video"
ExecStartPost=/usr/local/bin/restic forget --keep-within 90d
ExecStartPost=/usr/local/bin/restic prune

As you can see, this systemd service is backing up two directories

/mnt/mega/aditya - this is my personal documents
/mnt/mega/photos - this is my photos library

to /home/admin/seagate/backups where /home/admin/seagate is where the external drive is mounted.

Then, as a 3rd offsite copy of the backup I, once again, use rclone to replicate the restic backup to backblaze.

sudo crontab -l

#Ansible: sync arch backups to b2
0 1 * * 2,5 rclone sync /home/admin/backups/arch b2:restic-backup-arch
#Ansible: sync nas backups to b2
0 1 * * 3,6 rclone sync /home/admin/seagate/backups b2:nas-backup-aditya

Why these tools?

Backblaze

The simple answer is that they are the cheapest solution out there. AWS Glacier could be cheaper depending on how much data you have and how often you need to retrieve it, but for me B2, was the best one.

They are also fairly reputed among data hoarders so I am going to stick with B2 for now.

Restic

Restic is one of my favorite open source tools out there. It's fairly aged, supports encryption out of the box (in fact it doesn't even support unencrypted backup), has configurable compression, allows verifying the integrity and recoverability of the backups, and supports a wide range of cloud providers as the backend. It's also written in Go and used by Fillipo.

I can do a "soft" integrity check on the backup with

sudo restic check -r ~/seagate/backups

and a "hard" integrity check with

sudo restic check -r ~/seagate/backups --read-data

rclone

Honestly, I chose this one simply because a lot of people have talked highly about it. It was also fairly easy to set up. Offers incremental backup and checksum as well.

Improve your Google dorking with keyboard shortcuts

Sat, 21 Jan 2023 23:11:00 GMT

I'm sure you probably already know about Google dorking (also called Google Hacking). For those who don't know what it is - it's a way to narrow down your google search by using filters provided by Google. Example: if you are searching for a pdf of a book, you can narrow down your search using the filetype:pdf filter. If you want your search to only include a particular site you can use the site:<sitename.com>.

Without the filetype filter

I frequently use the site filter - mostly with reddit site:reddit.com. Ocassionally I also use it with stackoverflow.com and github.com. However, it's not quite pleasant to type it out!

I just found out about a shortcut you can use to avoid this typing. I've only tried this on chrome-based browser (Brave in particular) but I'm sure there's a similar feature in Firefox also.

Steps to configure shortcuts

Right click on the address bar. Click on the "Manage search engines & Site search" context menu.

Right click on the address bar

You should see the "Search engine" & "Site search" sections. Click on "Add" in the site search section.

Add new site search

Finally just fill in the form

Here's an example of my shortcut for reddit site filter in google search.

Reddit site filter

https://www.google.com/search?q=site:reddit.com %s

Using the shortcuts

This is my favorite part. Usually shortcuts involve chording keys. There's at least 2 keys involved and one's probably the Ctrl key. However, to use the site search shortcut you just need to type in the shortcut key that you configured and simply press space!

Using site search shortcut

Search Engine

You can follow the same procedure to add shortcuts for search engines also. I don't really use this because Google works fine for me. But in case you need to search with DuckDuckGo, for example, you have the option to set a shortcut.

Pacman for newbies

Thu, 19 Jan 2023 13:45:00 GMT

Who this post is for

Users who are new to pacman
Someone who wants to see what other pacman users regularly use

What this post isn't

Principle of pacman design
Detail explanation of the commands
Replacement of the documentation (duh!)

Everything in this post can of course be found easily using pacman --help. But that's a documentation. It has a lot more than you'd ever need. This post will help you filter out the few good stuff (read: regularly used) from the vast documentation.

Here's a list of commands I personally use. I never had the need to use anything more than what's listed here.

The Basics

Upgrade local packages

sudo pacman -Syu

This command is probably the most used pacman command by quite some distance. It does 2 things

downloads a fresh package list.
upgrades all the installed pacakges that are out of date.

Pacman keeps a local copy of the package list in the system, so it doesn't have to download the list every time you run a command. The -y flag forces pacman to download a fresh copy of the package list from the repositories.

Install packages

sudo pacman -S <package-name>

Search for packages

sudo pacman -Ss <package-name>

Remove packages

sudo pacman -Rns <package-name>

Clean cache

sudo pacman -Sc

Still basics but rarely used

These are a list of commands that you probably wouldn't use on a daily basis.

List all packages that are no longer required

pacman -Qdt

Remove all packages that are no longer required

sudo pacman -Rns $(pacman -Qdtq)

Info of a package

pacman -Qi <package-name>

I use this command to look for a few things

packages that require this package
packages this package require
whether it was installed explicitly

pacman -Qi seahorse

...
Depends On      : gtk3  gcr  ...
Required By     : None
Install Reason  : Explicitly installed

List all the packages installed in your system

pacman -Q

# To get the count
pacman -Q | wc -l

List packages that were explicitly installed

pacman -Qe

Packages come with dependencies. This is will only list those packages that you explicitly installed.

List all the packages installed from aur

Not possible. But you can get a close to accurate result with

pacman -Qm

Thanks

Photo by https://medium.com/@aliakbarzohour/use-pacman-in-arch-linux-and-manjaro-c987e4a0da66

Here's how you can wake up early without waking up early

Thu, 05 Jan 2023 16:00:00 GMT

How nice would it be to have a turn off switch for your body? I would turn myself off every night at around 9/10 pm. Hold on ... that didn't sound right. A "sleep switch" might be a more appropriate word. Yes - that sounds better. I would press the sleep button every night at 9/10pm.

There are some people out there who have this gift - a virtual sleep switch they can turn on to dose off whenever & wherever they want. I, unfortunately, am not one of those people.

Why does it matter when you sleep if you get 8 hours of sleep, you say? Because, just as important as it is to get adequate sleep it's also important that you get the morning sunlight (shout out to Dr Andrew Huberman).

And also if you happen to live with someone else who isn't a night owl, it would be nice to have a coherent schedule so you can get chores done & have meals together.

There's one other problem for late risers - and it's that you don't really have a good sense of where you are at your actual day. You see, I personally wake up at around 11am. By the time I wakeup and get ready for work it's almost noon. This gives me a feeling that half of my day has already passed but in fact it hasn't - it just started. I just woke up! Even worse, if it's winter - it's already dark 5/6 hours after you wake up.

This has been a pretty major problem for me. This dissonance of your internal clock with your external clock affects how you set yourself up for the day. So, here's a hack I came up with (just this morning actually).

Follow a different timezone where you wake up at 6 in the morning.

I know it's super silly but it works. I have found that Cairo/Egypt (GMT+2) or Berlin/Germany (GMT+1) suits me perfectly. When I wake up at 11 am (GMT +05:45) it's 6:15am in Berlin.

Timezone difference: https://savvytime.com/converter/nepal-kathmandu-to-egypt-cairo-germany-berlin

Here's how my time differs

	Local Time	Berlin Time
Wakeup	11:00 am	06:15 am
Start Work	12:00 pm	07:15 am
First meal	02:00 pm	09:15 am
Sleep	02:00 am	09:15 pm

I have a much better track of where I am at the day now.

There you go - that's how you wake up early without actually waking up early.

Thanks

Photo by Quin Stevenson on Unsplash
https://savvytime.com/

My New Year's Resolution for 2023

Sun, 01 Jan 2023 22:05:00 GMT

For the past few days I have been trying to decide which of these two topics to begin learning first: databases or Kubernetes. I've grown a sudden interest in these topics because the companies that I want to work at had at least one of the above as a hard requirement - or sometimes both. Now, of course it's not the only reason why I'm interested in them. My interest goes beyond just fulfilling job requirements as they are both highly essential skills to have in any backend engineer's arsenal. But it was for sure the spark that ignited the interest/motivation in me.

While searching for learning resources and companies that are doing amazing things on these topics, I've come across a few talented people. One such personality is Kelsey Hightower. He's an extremely prolific speaker and also a highly decorated software engineer at Google. I've been watching his Talks on Youtube and also scrolling through his tweet history (because that's what I do when I find interesting personalities). One tweet in particular stood out to me

"What do you say to people who think you're a one trick pony?"

I've been a one trick pony my whole career. At one point I was all in on Python and contributed to tools like pip and virtualenv. Then it was Puppet, then Go, then Kubernetes, and now Serverless.

It's called focus. https://t.co/nU7YByRB4T
— Kelsey Hightower (@kelseyhightower) November 30, 2022

"Focus" - that's interesting! But let's put that word aside for a second cuz I have a few more anecdotes to share.

I'm highly active on Reddit - especially on technology subreddits.

My 2022 Reddit Recap

I rarely engage though - maybe once in a blue moon. I'm just a lurker.

Anyway, there's this another highly skilled developer - Amos (@fasterthanlime) - who just hosted an AMA (Ask Me Anything) on reddit. I wanted to share one of the questions on that AMA, but unfortunately it was deleted. Nevertheless, the answer alone should suffice to convey what I'm trying to get across here.

"I'm very selective about what I spend my time on, ..."

It's not rocket science but rather a common knowledge but when you hear it from people you look up to - it really hits you. Advices are nice but I think they only bear weight if they come from someone you respect. (Btw, I'm not saying @fasterthanlime is my hero. Our view on the Go programming language differs radically.)

There are a few others who I've been following just as much

Thomas Ptacek (I came to learn about him through my job application at fly.io but turns out he's a highly decorated security engineer)
Fillipo (Just as ~~old~~ young as me and has already led the Go security team at Google!)

A sudden realization

If you notice one common thing among these guys - it's

Focus

They specialize in something and they're very frugal with their time.

I made a sudden realization that focus is something that I've been lacking more and more recently.

Oh look - an interesting blog post on Reddit about creating SVG with Javascript - let's read it!

Shoot! I can barely understand what's going on.

Meh, back to Reddit.

Now what's this...? a reddit post on how 1Password's cryptography is more secure than Lastpass'.

...CLICK!!!

I don't understand a single thing about any cryptography jargon that's in this post.

I better scroll down to the comments. That might be interesting.

Why am I hopping through these posts reading about things that I have very little understanding of? Is that even "reading" because as soon as I stop understanding some of the jargon, I'll just try to skim through the post and quickly scroll down to read the comments. In the end, I mostly likely will end up without gaining any knowledge. And even if I do, I will forget about it because when will I ever use it?

On the flip side, I don't want to narrow down what I intake so much that I become a one trick pony. Diversity is important! But, the balance is even more important.

A look back at school

I have learned quite a few technologies in the past year but all of them in surface.

That's not how I learned things when I was at school! I used to sit down and grok on topics for hours to grasp things from the root. I used to love making notes - and in fact I was pretty good at it.

Where are all these good practices that I highly benefited from? I barely pickup pen & paper these days. I don't have any notes and I haven't learned anything in detail lately.

I think it's mainly because of focus. At school, I could stay focused on one thing because there were syllabuses! If I suddenly had a interest on something out of syllabus, I knew not to spend too much time on it because I had a syllabus to cover. The syllabus were a roadmap, an overseer to guide my focus. Oh, I wish there was a syllabus to go from Software engineer to a Staff Engineer - I would have aced it!

So here's my new year's resolution.

To maintain focus and balance in what I read.

Thanks to

Stefan Cosma for the Photo Stefan Cosma.
ChatGPT for providing me with better words.

Places To Go On A Ride Around Kathmandu

Sun, 24 Jul 2022 00:00:00 GMT

It has been around 2 months since I purchased my motorcycle and I've already exhausted pretty much all the ride destinations around Kathmandu. I guess all riders can relate to this :).

Apart from the obvious popular destinations, it takes a bit of an effort to find the lesser known ones. Few vlogs on YouTube and scouring through Google Maps have helped me find a few. Googling for desinations is pretty much useless as almost all of the handful of blog posts have an identical list of places and they focus more on the desination than the ride. I care about the enjoyable rides than the destination itself. This blog post is my attempt to provide information especially to motorcycle riders.

My Ride

Svartpilen 250 at Pilot Baba Bhaktapur

I have a Svartpilen 250, which does alright on decent offroad. The ground clearance is pretty much one of the least on any bikes you can find on the market right now. And that's why I personally prefer to avoid offroad.

I think it's best to categorize the list into two sections

Good Roads
Off-roads

Good Roads

1. Nagarkot

You saw this coming from a mile away, didn't you? Obviously, it's number one on the list because there's no rider who hasn't ridden the twisties of Nagarkot. One of my best rides, if not the best, has been on this road on a rainy morning.

Svartpilen 250 at Nagarkot

There's probably 3 or 4 different routes (or even more) to reach Nagarkot from Kathmandu and one, in particular, is my favorite. I'm not sure how to show that route here but it's the one that Google Maps suggest.

The road was under construction just a few months ago but at the time of writing this post, it's pretty good. What I like best about Nagarkot ride is that at some point in the middle you'll have to ride through the fogs and drizzle.

2. Dadagaun, Shivapuri National Park, Budhanilkantha

If I had to pick my favorite road on this list, it'd probably be this. You get to ride through a mild forest on a road that's probably the best around Kathmandu.

It's fairly close to where I live (just 16kms away) that sometimes I even ride here during my lunch break.

On The Way To Dandagaun, Shivapuri

Once you cross Mahargunj, the traffic is pretty light and even the road to Budhanilkantha is quite fun. Just 1 or 2 kms after Budhanilkantha, you'll start to make the climb up and then the fun begins.

Just be aware that this road goes through an Army premise and that means you'll need to register your vehicle before you enter. Make sure you have your driving license and bill book with you as you'll need to provide them during entry registration.

At Dandagaun Resort View From Dandagaun, Shivapuri

3. Kalu Pandey's Crematorium, Dahachowk

This road actually exceeded my expectation. I was originally headed to Switzerland park, which I found pretty underwhelming (there's literally nothing), but then decided to keep moving.

Road To Kalu Pandey Crematorium

To my surprise, I found the Kalu Pandey Crematorium at the end of the road. Yes, THE Kalu Pandey about whom we all read in our secondary school. There's no entry fee and the site is well maintained.

Kalu Pandey Crematorium

There're also a few resorts at the top and also a view tower (as seen in the image below).

View Tower Resort

4. Nuwakot (via Tokha -> Bardalinn Resort-Shivapuri).

In terms of what the journey has to provide, this is probably the top on the list. The road itself is decent as there are quite a few broken segments on the way but my god the scenery is just breathtaking. It feels like you're way away from Kathmandu and are in a different region of the country.

There's a waterfall on the way - Jhor Waterfall, right after you cross Tokha. It's meh! No idea why it's so hyped as you can see all these bikes parked.

Jhor Waterfall Bike Parking

Jhor Waterfall

Once you cross this waterfall it starts to get better and better. Although I have to warn you, the climbs are pretty stiff and maybe riding with a pillion on this road isn't the best idea.

National Park Area On The Way To Nuwakot

As you keep moving there's another much better waterfall - Mahtong falls, right beside the road.

Mahtong Falls

Mahtong Falls Svartpilen 250

There's a local restaurant which offers one of the best Sukutis. I don't remember the name and unfortunately it's also not registered on Google Maps but here's a link to the place. I don't even have a picture of the Sukuti which should you tell you how good it was as I couldn't stop munching XD.

5. Chobhar - Toudaha - Kritipur

Chobhar is arguably the most popular destination for riders simply because it's right around the valley. I've been here more than any other places in the list.

6. Dolalghat - Khadichaur

All the other destinations on the list are mainly twisties. If you want to enjoy speed with a twist of twisties, then it can't get any better than this road. The scenary is pretty good, the traffic is light and the road is as good as they come in Nepal. Honestly, I found it quite difficult to believe that a road like this is just an hour away from the valley. For first timers on this road, it'll be quite an experience.

Dolalghat

This is definitely a trip for the weekend as it can take up 4-5 hours to get there and be back.

Khadichaur Bridge

OffRoads

Chitlang - Markhu
Manichud

Hands-on Aerospike with Go

Sat, 08 May 2021 21:20:00 GMT

This is an extremely basic introduction to Aerospike. It's basically a compilation of resources to help you quickly get started.

SQL Analogy

I think the best way to start learning Aerospike is to establish an analogy with SQL databases.

Conceptual components of Aerospike Data model

Aerospike	SQL
NameSpace	TableSpace
Set	Table
Record/Object	Row
Bin	Column
Key	Primary Key

Knowing this much will be sufficient to get us started.

Install Aerospike

I'd recommend installing Aerospike with Docker just because it's simple that way. For this article, I'm using version 5.5.0.2.

# Download aerospike docker image
docker image pull aerospike:5.5.0.2

# Start docker
docker container run -p 3000:3000 --rm aerospike:5.5.0.2

Aerospike server, by default, runs on port 3000. The -p flag tells Docker to map the container's port 3000 to our system's port 3000. The flag --rm will automatically delete the docker container after we stop the aerospike server.

You also need to download the Aerospike Go package which is officially available at github.com/aerospike/aerospike-client-go

go get github.com/aerospike/aerospike-client-go

With both of these installed, let's get started.

1. Connect to Aerospike from GO

package main

import (
	"log"

	as "github.com/aerospike/aerospike-client-go"
)

func main() {
	client, err := as.NewClient("localhost", 3000)
	if err != nil {
		log.Fatal(err)
	}
	client.Close()
}

These few lines of code will connect to Aerospike and then close immediately.

2. Create a Set

Naturally, the next move you'd think would be to create a Set as we do in relational databases. However, that's not required in Aerospike because our table (Set) does not have a fixed schema.

The data model is determined through your use of the system. For example, if you wanted to add a new data type to a record, you would write that data type into the record without having to first update any schema

3. Inserting data

Although we didn't need to explicitly create a Set, we do however require one. Let's call our Set "Person".

And also a namespace is mandatory. Let's name it "Avengers" for lack of a better word right now 🤷‍♂️. The namespace cannot be created from a client but needs to be configured in the server itself. By default, a namespace called "test" is created automatically. Let's create our namespace by modifying the docker command. We can pass in a NAMESPACE environment variable to the docker container and it'll create the namespace for us.

docker container run -e 'NAMESPACE=Avengers' --name 'Aero' -p 3000:3000 --rm aerospike:5.5.0.2

I've assigned a name to the docker container so that's it'll be easier to reference it later.

Close the previous container and run the above command to restart the Aerospike server.

3.1 Storing Records

For every record that we want to store, we need a key to identify that record; just like a primary key in RDBMS. Let's use the person's name as the key 🙃.

Then we need to create the record itself. Just so we have a reference, let's try to insert the person's age and the name.

# Reference SQL table
+-----+----------+
| age | name     |
+-----+----------+
| 25  | "Aditya" |
+-----+----------+

"age" and "name" are columns (bin). We can create them like this

binAge := as.NewBin("age", 25)
binName := as.NewBin("name", "Aditya")

Finally, to piece everything together here's the code to store the record.

package main

import (
	"log"

	as "github.com/aerospike/aerospike-client-go"
)

const Namespace = "Avengers"
const Set = "Person"

func main() {
	client, err := as.NewClient("localhost", 3000)
	if err != nil {
		log.Fatal(err)
	}
	defer client.Close()

	// Create key for a record
	key, err := as.NewKey(Namespace, Set, "aditya")
	if err != nil {
		log.Fatal(err)
	}

	binAge := as.NewBin("age", 25)
	binName := as.NewBin("name", "Aditya")

	client.PutBins(nil, key, binAge, binName)
}

Notice that I never explicitly create the Set "Person". It is created automatically on-demand.

The PutBins() method can take in any number of bins. If you supply two bins with the same name, then the latter will take precedence.

3.2 Different way to create and store bins

Instead of creating each bin one by one, we can instead create a BinMap.

binMap := as.BinMap{
  "age":  25,
  "name": "Aditya",
}

client.Put(nil, key, binMap)

To me, this looks neater and I think is more readable. Then we can use the .Put() method to store the bins.

3.3 PutBins() vs Put()

So we have two methods to insert bins. Which one should you prefer? Well internally, Put() uses PutBins() and the Official Best Practices recommends using PutBins over Put to avoid allocation for binmaps.

4. Reading records

We can read the records from Aerospike using the Key.

package main

import (
	"fmt"
	"log"

	as "github.com/aerospike/aerospike-client-go"
)

const Namespace = "Avengers"
const Set = "Person"

func main() {
	client, err := as.NewClient("localhost", 3000)
	if err != nil {
		log.Fatal(err)
	}
	defer client.Close()

	// Create key for a record
	key, err := as.NewKey(Namespace, Set, "aditya")
	if err != nil {
		log.Fatal(err)
	}

	binMap := as.BinMap{
		"age":  25,
		"name": "Aditya",
	}

	client.Put(nil, key, binMap)

	// Read a record
	record, err := client.Get(nil, key)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(record.Bins)
}

// OUTPUT
//s map[age:25 name:Aditya]

In addition to reading records with the key, Aerospike also offers a basic query language. It's similar to SQL but nowhere near as powerful. Here's an example query

SELECT * FROM Avengers.Person WHERE age BETWEEN 15 AND 30

I won't be discussing Aerospike Query language (AQL) in this article.

5. Deleting records

We can delete specific records by the key or drop the entire records in the Set.

package main

import (
	"fmt"
	"log"

	as "github.com/aerospike/aerospike-client-go"
)

const Namespace = "Avengers"
const Set = "Person"

func main() {
	client, err := as.NewClient("localhost", 3000)
	if err != nil {
		log.Fatal(err)
	}
	defer client.Close()

	// Create key for a record
	key, err := as.NewKey(Namespace, Set, "aditya")
	if err != nil {
		log.Fatal(err)
	}

	binMap := as.BinMap{
		"age":  25,
		"name": "Aditya",
	}

	client.Put(nil, key, binMap)

	// Read a record
	record, err := client.Get(nil, key)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(record.Bins)

	// Delete Record
	existed, err := client.Delete(nil, key)
	if err != nil {
		log.Fatal(err)
	}

	if existed {
		fmt.Println("Record deleted")
	}
}

Droping the entire records in the set

client.Truncate(nil, Namespace, Set, nil)

6. Policy

Did you notice that the first argument when inserting and reading records was set to nil? That parameter is the policy. A Policy helps us define the behavior of read and write operation and much more. For example: what happens when we insert a record with a key that already exists? We can define a policy to overwrite the previous data or ignore the one being inserted.

Let's create a write policy that'll tell Aerospike server to replace old records if new ones arrive

policy := as.NewWritePolicy(0, 0)
policy.RecordExistsAction = as.REPLACE

7. AQL

Aerospike offers a command-line utility - AQL. It comes preinstalled in the docker image we downloaded. Let's get an interactive shell on the container

docker container exec -it Aero aql

Try out these few queries

SHOW NAMESPACES;

SELECT * FROM Avengers.Person;

TRUNCATE Avengers.Person;

SHOW BINS;

References

Cursor-based pagination vs Offset-based pagination

Sat, 01 May 2021 18:45:00 GMT

A. Offset-based Pagination

This is the classic and the simplest way to achieve pagination. In essence, it's basically skipping X amount of data in front.

Example: You have a list of 100 flowers and you want to show 10 of them at a time. On page 1 you fetch 10 flowers and skip 0 of them, one page 2 you fetch 10 and skip 10, and so on ...

In SQL, a query would look something like this

SELECT * FROM flowers OFFSET ? LIMIT 10;

As simple as it is, it is still used heavily because of its sheer simplicity.

Offset-based pagination in google.com

Disadvantages

A big disadvantage, and really a deal-breaker for some use cases, of this method is that it is not at all vialbe for realtime data.

1. Duplicate data

Imagine in the above example a user is browsing on page 4, and 10 new flowers are added to the front. When the user visits page 5, he will see the same flowers again.

2. Skipping data

In contrast, if 10 flowers were removed from the front, then users will miss out on 10 flowers when navgating to page 5 from page 4.

3. Performance

The above SQL may appear innocent but it's not! An OFFSETed sql query will still scan through all the rows in front. So if you're offsetting by 100 rows, the database will still scan them.

Due to this reason, as the offset factor grows, pagination time increases. Imagine cases where you need to offset by millions of rows - that could take a minute.

Source: https://laptrinhx.com/

But to say that this is a disadvantage of offset-based pagination is not quite fair since this is a SQL problem. If we were using an index instead, then this wouldn't be a problem at all.

You can see how none of these disadvantages really affect Google because their search index do not change in real time and offset doesn't add overhead.

B. Cursor-based Pagination

In this method, you have a cursor instead of an offset value. To continue with our example above, a cursor based query would be "Give me 10 flowers after Anthurium". This method is extremely suitable where data changes frequently since you don't have the problem of duplicates or skipping over some data.

Disadvantages

1.Necessity for a unique cursor

Imagine in the example above you have two rows with the same flower. Suppose we are on page 5 and the first element on the next page is Anthurium. This means our cursor is set to "Anthurium". We start to navigate and reach page 8 and the cursor is set to "Anthurium" again because it's the first element on the 9th page. You can see how this can cause an infinite loop.

To solve this problem, we need to make sure the cursor is guaranteed to be unique for each data. We can change our cursor to be the flower name + row id.

2. Implementation Overhead

Implementing cursor-based pagination can sometimes become really difficult and might limit us from performing sorting on certain fields.

Megan Chan has done an excellent job in describing the implementation quirks which you can read here.

References

Dependency Injection In Go With Context API

Sun, 25 Apr 2021 14:00:00 GMT

1. Dependecy injection in Go with Context API

This week I found a really neat way to inject dependencies in Go using the context API. Here's the YouTube link of a brief demo by the author himself.

The classic way, that I know of, to inject dependencies is to make all the HTTP handlers a method of the main Server struct and the handlers can just access the necessary dependencies from the server.

type Server struct {
  db Database
}

func (s *Server) GetUsers(w http.ResponseWriter, r *http.Request) ([]string, errror) {
  users, _ := s.db.getUsers(ctx))
  ...
}

Nothing wrong with this approach really except maybe things will start to get a little messy as the number of handlers grows because all the handlers should reside inside the same package.

Instead, with Kayle's method, we can have handlers as standalone functions (and not methods on the server) and they can receive the dependencies from the context


type contextKeyType string

const (
	ContextKey contextKeyType = "dep-provider"
)

// This middleware will attach the dependencies in the request context on every request
_ = http.ListenAndServe(":8080", server.injectDB()(router))

// My standalone handler
func Getusers(w http.ResponseWriter, r *http.Request) {
  db := r.Context().Value(provider.ContextKey).(Database)
  user, _ := db.getUsers(ctx))
  ...
}

To learn more about it, you can check out the Github Repo

2. Go Type conversion vs Type assertion

I have been wrongly misusing the word type coercion not actually knowing the difference between type conversion and type assertion.

A. Type Assertion

The notation x.(T) is called a type assertion. With the absence of Generics in Go, interfaces and type assertions are used heavily. Even in the example above you can see that once we get the Database value from the context we perform type assertion to set the value of db to Database.

For a type assertion to succeed, the variable must satisfy all the interfaces of the type that's being asserted. In case of failure to satisfy the interface, a panic will occur. A graceful way to perform type assertion is to perform a check on the second value returned by the assertion

t, ok := i.(T)

If i holds a T, then t will be the underlying value, and ok will be true. If not, ok will be false and t will be the zero value of type T, and no panic occurs.

B. Type Conversion

A Conversion changes the type of an expression to the type specified by the conversion. A simple example would be converting an int to int32

var age int
age = 25

newAge := int32(age)

3. Time formats in Go

When converting a datetime object to a string, most programming languages offer different "verbs" to get the date in the format the way you want. For example: In python you can use datetime.now().strftime("%H:%M:%S") or in Java you have HH:mm:ss.

Go has a unique approach of formating date and time which I prefer more than any other ways out there. In Go, you can provide a human-readable format like this.

x := time.Now()
fmt.Println(x.Format("2006 Jan 03"))
fmt.Println(x.Format("3 01 06"))
fmt.Println(x.Format("Monday, January 2006"))
fmt.Println(x.Format("Monday, January 2006 03:15"))

// # Output
// 2009 Nov 11
// 11 11 09
// Tuesday, November 2009
// Tuesday, November 2009 11:23

The key, and the only, thing to remember is the reference date

Mon Jan 2 15:04:05 MST 2006

You see in the example above, I always used the month January and the year 2006. If I use any other month or year or date, Go will not understand it. So even though I can format the month anyway I like - Jan, January, 01 - it has to be the month January. The same is the case for date, day, year, hour, time, second, and timezone.

At first glance, this seems pretty weird. Why this specific date and not any other one would ask. There's a pretty clever reason behind this - it's that in POSIX date format it actually is 0 1 2 3 4 5 7 6

Mon Jan 2 15:04:05 -0700 MST 2006
0   1   2  3  4  5  7        6

But why is 7 in front of 6 ? It's because if we had to interchange them then it would be the year 2007 and Jan 2nd 2007 is not Monday whereas Jan 2nd 2006 is.

4. Declaring emtpy slices in Go

There are various ways to declare a slice in Go

var x []string
y := []string{}
z := make([]string, 0)

fmt.Printf("Len: %d\tCap: %d\tIsNil: %v\n", len(x), cap(x), x == nil)
fmt.Printf("Len: %d\tCap: %d\tIsNil: %v\n", len(y), cap(y), y == nil)
fmt.Printf("Len: %d\tCap: %d\tIsNil: %v\n", len(z), cap(z), z == nil)

If you run the code above, you'll see that they all pretty much do the same thing except that the first command declares a nil slice value. They are functionally equivalent as the len and cap are both zero. But the official way to declare a slice is the first one.

Do note that some libraries behave differently for nil slice value like the "encoding/json". A nil slice encodes to null, while []string{} encodes to the JSON array [].

References

Git tricks & Deterministic builds on NPM install

Sat, 10 Apr 2021 12:00:00 GMT

Find the files in a git repository that have changed the most

With a few CLI tools along with the git log command, you can see the files that have changed the most over a certain time period.

I've set up a ZSH function in my .zshrc file

gitchangehist() {
  git log --numstat |
    awk '/^[0-9-]+/{ print $NF}' |
    sort |
    uniq -c |
    sort -nr |
    head
}

I ran this on my website's codebase and here's the result

gitchangehist

37 src/components/Layout/layout.css
31 gatsby-config.js
23 package.json
20 src/templates/post.css
19 yarn.lock
19 src/pages/index.js
14 content/posts/twitter-bot-tutorial-nodejs/index.md
12 src/pages/portfolio.js
11 src/components/Portfolio/Project.css
10 content/posts/what-does-it-mean-to-own-a-bitcoin/index.md

Git provides a convenient way to limit commits with the --since option.

--since=<date>, --after=<date>
    Show commits more recent than a specific date.

You can use it like this to pass in a time frame

git log --since 1.year.ago
git log --since 1.month.ago

There are also other interesting options like --author and --grep.

List all the strings inside a git repository

This is probably my favorite git trick provided by the awesome @TomNomNom

gitstrings() {
  # Print all the strings in .git/objects/pack
  # Credits to tomnomnom - https://twitter.com/tomnomnom/status/1133345832688857095
  {
    find .git/objects/pack/ -name "*.idx" | while read i; do git show-index <"$i" | awk '{print $2}'; done
    find .git/objects/ -type f | grep -v '/pack/' | awk -F'/' '{print $(NF-1)$NF}'
  } | while read o; do git cat-file -p $o; done
}

This one-liner will find all the strings in a git repo and just spit it out to the stdout. It's super useful especially to bug-bounty hunters to look for any leaked credentials or API keys. Sometimes developers commit confidential keys and then make another commit to remove them but git stores the entire history and this one-liner will help you find them.

I used this trick to basically glide through the Recon Badge in PentesterLab.

You can use interesting grep patterns to find some juicy stuff.

Deterministic NPM install

NodeJS' package manager npm keeps track of the dependency graph (the dependencies of the main dependencies and their dependencies and so on) in a package-lock.json file. This file is essential to have consistent versions of the dependencies. The next time we need to install dependencies, this file should help us get the right versions.

Sounds good right? Nope. The command, npm install, that most of us use, does not use the lock file at all! You may have noticed that when you run npm install the lock file gets changed. If a dependency is not in the package-lock.json it will be added by npm install.

The command you want to use is npm ci. It's named after Continuous Integration. If any dependencies are missing or have incompatible versions, it will throw an error.

Source: https://stackoverflow.com/a/53325242/6199444

In my internship, I had to work on an Angular project but I just couldn't make it run. I probably should have used ci instead of install 😁.

Number of lines of code in my website

Sat, 03 Apr 2021 18:30:00 GMT

Swapfiles to the rescue

At my new job, I need to work on a massive project which is very demanding. I've been rocking an 8GB RAM since forever but the time had finally come to give it an upgrade. My system would just crash due to unavailable memory. I couldn't upgrade RAM as the stores were closed on weekends. My time was ticking, since I am to complete 40hrs of work a week, and I wasn't even able to run the project.

Fortunately for me, I could add swapfiles to act as virtual memory. I had completely forgotten about this and was waiting like an idiot for shops to open the next day.

A downside of using a swapfile on an SSD is that SSDs have a limited lifespan. Every write (but not read) cycle (or more accurately every erasure) wears a memory cell, and at some point it will stop working.

As always, the Arch wiki had all the instructions to create a swapfile. My system was still pushing the 6GB swapfile, I had created, to its limits. And since I'd be running multiple users at once, I decided to opt for 32GB of RAM. Quite an upgrade!

Number of lines in a git repo

I was curious to see how many lines of Go code our project had. There were lots of possible solutions but the one that I liked the most was using the CLoC - Count Lines of Code CLI tool.

What's nice about CLoC is that it can differentiate a real line of code from a comment or a blank line. It can also determine the lines of codes in a specific programming language. So if you have a project using multiple programming languages, then CLoC can help you get the LOC of just JavaScript code.

I ran CLoC on the code running this website and this was the result

❯ cloc --exclude-dir=node_modules,public,.cache .
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
JSON                             2              0              0          45005
Markdown                        20            769              0           1602
JavaScript                      21             97             33           1020
CSS                              9            116             27            604
YAML                             6             36              0            204
-------------------------------------------------------------------------------
SUM:                            58           1018             60          48435
-------------------------------------------------------------------------------

I'm excluding the node_modules directory for obvious reason and also the public directory which is the build files along with the .cache directory which is the build cache.

The high number of JSON code is due to the package-lock.json file.

Copy image from X clipboard to a file

I use Flameshot to instantly take screenshots and then copy them to my X clipboard. I have configured a key binding on SXHKD like this

super + shift + s
  flameshot gui -p ~/Pictures/Screenshots/

And with that, I can simply copy & paste the images to apps like Telegram or web apps on a Browser. This makes it really simple and fast as I don't need to save the image to a file and then browse the files to select the image.

But I want to copy those images, on the clipboard, to a file and the GUI file manager that I'm using, PCMan File Manager, doesn't support it.

I use xclip to manage my clipboards and it has a command to stream the image on the clipboard

xclip -selection clipboard -t image/png -o > image.png

# Shorter
xclip -sel clip -t image/png -o > image.png

Now, I can just add a shell alias and conveniently copy the image to a file

alias cpimg="xclip -selection clipboard -t image/png -o"

cpimg > myimg.png

Moreover, if you're using a TUI FM like Ranger or NNN you can easily setup key bindings for this.

References

Make sure to clean your Go build cache

Sat, 03 Apr 2021 13:00:00 GMT

Useless Anecdote

I realized that my storage was quickly getting filled out. I had recently added a new user to my system so I thought maybe that accounted for the sudden spike in storage usage. When I finally checked what was actually going on, I saw that my Go build cache directory was taking up about 35GB!

Locate Go build cache directory

The default location for build cache data is a directory named go-build in the standard cache directory. For me, on Linux, it is ~/.cache/go-build/. You can find out yours using this command

go env GOCACHE

Go also conveniently allows you to modify the cache location using the environment variable GOCACHE

Check disk usage of Go build cache

Here's a quick one-liner to see the disk usage of the build cache

du -hs $(go env GOCACHE)

Content of Go build cache

Contents of go-build cache using Diskonaut

Finally, to address the question of this article, you can run this command

go clean -cache

This command removes all the subdirectories inside go-build directory and leaves out just two files

README
trim.txt

The trim.txt file stores a Unix timestamp. I'm guessing it's the timestamp of the time you run the clean command.

Other caches

In addition to the build cache Go also has a few other caches.

-testcache expires all the test results cache inside the build cache
-fuzzcache remove files stored in the Go buildcache for fuzz testing.

Both of those flags clean caches that are inside the build cache directory.

-modcache remove the entire module download cache located at $GOPATH/pkg/

All of this information is available with go help clean

References

https://golang.org/pkg/cmd/go/#hdr-Build_and_test_caching

Security and Analytics

Sat, 06 Mar 2021 23:25:00 GMT

Security Stuffs

1. Using TLS as a mechanism to attack web-based systems

Most of the things in this talk went over my head. The researcher demonstrated a novel way to use TLS with DNS rebinding to perform SSRF attacks that were previously considered impractical to achieve.

TLS provides a custom field (session id - size upto 65KB) which can be controlled by the TLS client. The attacker injects the malicious payload as a session id.

2. Migrated away from Lastpass to Bitwarden

Lastpass recently announced that they are restricting the free subscription to just one device. I've been wanting to shift away from Lastpass for quite some time now mainly because of their bad UI in their chrome extension and also because security nerds don't speak highly of them.

Besides Lastpass, there were a couple of other good choices but I wanted one with a free version and a nice chrome extension. Bitwarden ticked all my requirements and it's also fully open source! The transition was pretty simple. All I had to do was export a CSV from Lastpass and import it on Bitwarden. I believe it took me about 5-10 minutes.

GoatCounter to SimpleAnalytics

Analytics are super fun! When I first deployed my website, I used to have google analytics running on my phone at all times; just staring at the stats checking if a new user visited my site. However, analytics add overhead to your website and privacy is another issue. A couple of months ago I ditched Google Analytics and started using Simple Analytics in favor of a simpler UI and better privacy policy. However, after some thought, I got rid of Simple Analytics. The main motivation was to make my website free from any tracker.

But recently, I tried GoatCounter - an open source web analytics platform. I used it for a week or so. It was simple and pretty fast but I wasn't a big fan of the UI. And now I'm back again with SimpleAnalaytics 😁. I remember them having a free version but apparently now they don't. I got a free 1-year subscription from my Github Students Benefit though.

Bypass ad-blockers with a custom subdomain

Generally, when you add a tracker, the tracker sends all the data to its own server. Adblockers and privacy extensions block any requests to these tracker servers. In order to circumvent this, Simple Analytics provides a nice feature to setup our own subdomain to collect the data. All we need to do is setup a CNAME DNS record that points to Simple Analytics' server. Easy Peasy!

A better way to embed external script in GatsbyJS

I'm using GatsbyJS for this site and guess what? There's actually a plugin to add an analytics tracking script! How ridiculous is that! All I need to do is add one line of script tag and there's an NPM module for that? I'm not sure what the plugin offers but I'm for sure not installing it. I just added the simple analytics script tag to my Layout component.

const Layout = ({ children }) => (
  <>
    <Navbar />
    <main className="page-content">
      <div className="wrapper">{children}</div>
    </main>
    <Footer />
    <script async defer src="https://stats.adityathebe.com/latest.js" />
  </>
);

However, this layout component was embedded inside a div and I wanted the script tag to be as the first level child of body like this.

<div id="___gatsby">...</div>
<script async="" defer="" src="..."></script>

Thankfully, gatsby provides a way to do this with gatsby-ssr.js.

import React from 'react';

export const onRenderBody = ({ setPostBodyComponents }) => {
  setPostBodyComponents([
    <script async defer src="https://stats.adityathebe.com/latest.js" />,
  ]);
};

I found the solution here.

Fixed a bug in Subdomain-Takeover scanner to reduce Azure cost

I've been running a scanner on my VPS for about a year now to hunt for subdomain takeover bugs. For that, Azure Free Student's subscription had been serving me well until I had to create Azure instances in regions that weren't available in the student's subscription. Due to that, I had to create a premium account. In the first 3 months of migrating, I racked a bill of $152! Although, I have to say it's not as bad as the guy who got a bill of $72k overnight.

Yesterday, I figured out the cause and fixed it. I was running an Azure App Service Plan 24x7! Now, the script creates one only when it's required and also uses a free version instead of the standard version. All I had to do was add two words --sku F1.

As a precaution, I've also added a new budget in Azure Cost Management so that I get an alert when I'm near the end of my budget.

Introducing Weekly Journals

Sun, 28 Feb 2021 12:30:00 GMT

I was a decent and an above-average student in college and I can say with some confidence that having clean and organized notes did help me a lot. I used to spend a tonne of time preparing notes and it's something I really take pride in.

However, after my college, there have been rarely a few times that I've picked a pen. I don't even have a pen on my desk right now. The habit of making notes has gone.

Most of my time is spent in front of my computer - reading blogs and watching videos; mostly tech-related. I've realized that I've been bombarding my brain with new information so much that I've difficulty retaining most of them. Sometimes, when I'm scanning through my browser history, I see an article and go - "ah ya, that was an interesting read. I totally forgot about it".

And some other times, I read a blog and google the things I don't understand, and quite often I'll fall into this rabbit hole eventually not knowing where I started and what I was initially trying to read.

In the end, all the time and effort spent learning will account for very little. This false sense of productivity is dragging me back as I'm not making the most of my time. In reality, I'm not learning much or rather not retaining much. For this reason, I've decided to start writing a journal every week in hopes that it'll help me retain the things that I learn. And also, this will motivate me to write every week which is a good thing I guess.

What is this weekly journal?

You can visit the weekly journal here. Every week I'll try to post an update on the things that I learned or built that week although certainly not limited to just these. It could be just about anything - I could maybe write about a podcast or a movie that I enjoyed, a new place I visited, and so and so.

I'm hoping this initiative will help me stay focused because at the back of my mind I'll always have this sense of responsibility to take a journal of everything that I learn. So, if I realize that something I'm reading or watching is too much for me to understand, then I'll just quit instead of trying to force myself to sit through it. Likewise, if I get diverted too much from the subject at hand, then maybe this will help me focus less on the less-important things and get back to the subject.

At the end of the week, I can also go through the journal and revise the things that I learned.

I believe simply reminding yourself of the things that you learned a while ago goes a long way.

Alternatives

I highly encourage you to embrace this practice of taking notes. You don't need to have a blog; a pen and a notebook will do just fine. If you need it on the web, then you can use some services like Whatgotdone.

References and Influences

GRPC, ProtoBuf and weekly journals

Sat, 27 Feb 2021 17:00:00 GMT

Weekly Journals

After having this idea of adding a new weekly journal section to my blog for literally more than a year, I have finally done it. It didn't take me as long as I had thought it would to implement this. GatsbyJS makes it super easy. I will soon write a blog post on what motivated me to add this new section to my blog.

GRPC and Protocol Buffers

I have used and created a lot of REST APIs and almost all of them used JSON to exchange data. I love JSON cuz its so simple, human-readable, and easy to add and modify data. However, they're not the most efficient way to exchange data in terms of bandwidth. The most "bang for bandwidth" way to do it would be to encode message in binary. That's basically what GRPC and ProtoBuf is about. This week I have started learning them in Go.

Resources

Go's Defer, Panic and Recover

I took the time to learn a bit more about some of the in-built functions in Go. The defered functions in Go are executed no matter what; even when a panic occurs. However, in case of os.Exit() the program shuts down immediately giving no chance for the defered function to execute.

One thing that really troubled me for quite some time was how the return value(s) of the defered function were handled. We are used to write defer file.Close() but this function call can return an error. It turns out the error or any other return value is simply ignored. To handle the return value, we need to wrap the function call inside an anonymous function.

defer func() {
  err := file.Close()
  if err != nil {
    log.Println("Ooops")
  }
}

Resources

SQL in Go

Anatomy of Go Slices

Wed, 10 Feb 2021 13:00:00 GMT

I've started to use a lot of Go recently. I just love the language! Seriously, I don't think I have felt this much love for any other language!

I'm ashamed to admit that it was up until yesterday that I didn't quite understand the behavior and implementations of Slices in Go. There are already so many great blog posts out there that I don't think I have anything to add. However, that shouldn't stop me from writing another blog post on the subject because with that mindset my blog wouldn't even exist. So in addition to the hundred articles out there on Go Slices, here's the 101th. 😁

Slices

Slices are flexible and they should be preferred over arrays whenever possible. The most important thing to note is that slices have Arrays underneath them. I like to think of slices as the frontend for arrays.

Visualizing a slice as a struct as shown below can be helpful

type sliceHeader struct {
    Length        int
    Capacity      int
    ZerothElement *byte
}

ZerothElement in sliceHeader points to the first* element of the array underneath. It doesn't necessarily have to point to the *first* element itself as we will see later.

Slice Capacity

One thing that just seemed a little off to me in the beginning was the need for capacity in slices. How is it any different from the length of the slice itself ? None of the languages that I was familiar with had this concept of length and capacity. Almost always, the length was sufficient to define the size of an array or a list or whatever you want to call it.

The capacity of a slice can be set when creating the slice with the make function. If no capacity is supplied then it defaults to the length of the slice.

make([]Type, length, capacity)
make([]Type, length)

To put it simply, the capacity of a slice is the size of its underlying array*. It's a little complicated than that. We'll cover it in a minute.

Let's start by making a new slice with a predefined capacity.

mySlice := make([]int, 2, 5)
// [0, 0]

This code will internally create an array of integers of size 5. However, the slice will have access to only the first two elements of the array since its length is 2. If we try to access the third element then we get a runtime error.

mySlice[2]
// panic: runtime error: index out of range [2] with length 2

Now consider that we need to add a third element to mySlice.

myNewSlice := append(mySlice, 10)
// [0, 0, 10]

Take a moment to think and tell me this - did the append function create a new array?

It didn't !

What actually happened here is we created a new slice with an incremented length of 3 and the new element 10 was inserted into the array. But this new slice still points to the same array. We have a situation where both mySlice and myNewSlice are pointing to the same array. I think that's pretty cool.

Multiple slices can point to the same array

Now, of course, this was only possible because the array underneath actually had room for growth. If initially, we had set the capacity of mySlice to 2 then the append function would have no choice other than to create a new array. As an exercise, you can try appending more than 10 items to the slice and observer the behavior.

Let's verify this in code

mySlice := make([]int, 2, 5)
fmt.Println(mySlice)
fmt.Printf("mySlice's Address: %p\n", mySlice) // 0xc000100030

myNewSlice := append(mySlice, 10)
fmt.Println(myNewSlice)
fmt.Printf("myNewSlice's Address: %p\n", mySlice) // 0xc000100030

You'll see that the first elements of the two slices still point to the same address.

A closer look at the Slice Capacity

Earlier when I said "The capacity of a slice is the size of its underlying array", I lied. It can be the case but it's not quite accurate. Let's take a look at the example below

We have that two slices sliceA and sliceB and both of them have a pointer to the same array but they represent different "pieces" of the array. sliceA represents the first 5 elements of the array while sliceB represents the 3 elements starting from the 12th index.

If you noticed, the capacity of sliceB is actually not 20 - it's 8 even though the underlying array has size 20. In order to measure the capacity of a slice, not only do we need to know the size of the underlying array we also need to know where in the array the slice begins. So a more accurate definition of capacity would be

The Capacity of a slice is equal to the length of the underlying array, minus the index in the array of the first element of the slice (zero in this case).

Let's verify this in code

package main

import "fmt"

func main() {
	parentSlice := []int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19}
	fmt.Printf("Parent Slice: %v\n", parentSlice)
	fmt.Printf("Parent Slice's Address: %p\n\n", parentSlice)

	sliceA := parentSlice[:5]
	fmt.Printf("sliceA: %v\n", sliceA)
	fmt.Printf("sliceA's Address: %p\n", sliceA)
	fmt.Printf("sliceA's Length %d\n", len(sliceA))
	fmt.Printf("sliceA's Capacity %d\n\n", cap(sliceA))

	sliceB := parentSlice[12:15]
	fmt.Printf("sliceB: %v\n", sliceB)
	fmt.Printf("sliceB's Address: %p\n", sliceB)
	fmt.Printf("sliceB's Length %d\n", len(sliceB))
	fmt.Printf("sliceB's Capacity %d", cap(sliceB))
}

We start by creating a parentSlice and then slicing it to create two new slices sliceA and sliceB. If you run the program above you can verify by checking the addresses that all the 3 slices are in fact pointing to the same array.

Parent Slice: [0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19]
Parent Slice's Address: 0xc000100000

sliceA: [0 1 2 3 4]
sliceA's Address: 0xc000100000
sliceA's Length 5
sliceA's Capacity 20

sliceB: [12 13 14]
sliceB's Address: 0xc000100060
sliceB's Length 3
sliceB's Capacity 8

But hold on, sliceB's address is different than the other two. That's because it's showing the address of the 12th element and not the first element. We can verify that by printing the address of the 12th element of the parentSlice

fmt.Printf("Parent Slice's 12th element address %p", &parentSlice[12])
// 0xc000100060

This address should match the address of sliceB's first element.

References

Pitfall in JavaScript's Array.prototype.sort()

Tue, 09 Feb 2021 20:45:00 GMT

I was recently going through the problems in LeetCode where I was faced with a problem that required sorting an array of numbers. Pretty easy, I immediately used the .sort() method

const nums = [1, 9, 4, 8];
nums.sort();
console.log(nums); // [1,4,8,9]

I ran the code and the test passed successfully. So I submitted the code but then I got an error. My code failed to pass all the tests 😧 ! If you've done LeetCode you're probably familiar with the rage I felt at the time.

This was the test case that my code failed

[74, 100, 60, 98, 8, 45, 6, 59, 69, 32, 93, 9]

Baffled with what was going on, I ran the code on my local machine and started using the Javascript developer's greatest asset - console.log().

[100, 32, 45, 59, 6, 60, 69, 74, 8, 9, 93, 98]

As soon as I saw this, I knew what was going on ! If no compare function is supplied to .sort() javascript coerces the array elements into strings and then sorts them in lexicographical manner. Damn it ! I knew this behavior and I still managed to fall for it.

The sort() method sorts the elements of an array in place and returns the sorted array. The default sort order is ascending, built upon converting the elements into strings, then comparing their sequences of UTF-16 code units values.

The Fix

I wrote a simple compare function to fix the bug.

nums.sort((a, b) => a - b);

console.log(nums);
// [6, 8, 9, 32, 45, 59, 60, 69, 74, 93, 98, 100]

What the hell is even a Digital Fingerprint ?

Fri, 15 Jan 2021 18:00:00 GMT

Digital fingerprint, as complex as it sounds, is simply just the digest(output) of a hash function. Example:

printf 'adityathebe' | sha256sum
62eb9a06b678df7d48eb59929ab07a9858ed940ddfd3d8596482458133babe67

There you go - we just created a digital fingerprint of the string 'adityathebe'. As simple as that!

The above command may not run on a Windows machine or on MacOS. In that case, you can try online tools to get the sha256 hash - https://emn178.github.io/online-tools/sha256.html

It's important to mention what hash function was used to generate the fingerprint as we can generate different fingerprints for the same data with other hash functions.

printf 'adityathebe' | sha1sum
7d6fa418a5cc416fc8e70551b4ea86fbbd9456d3

printf 'adityathebe' | md5sum
0673288a42e614ad56a7da35e5c88153

There are many more hash functions in addition to these.

Why create digital fingerprints ?

Digital fingerprints are created to ensure Intigrity. Let's imagine I have a will that says I want to donate all of my bitcoins to John Mayer. Here's the will.txt file where I state just that

I declare that I sign and execute this instrument as my last will
to grant all of my million bitcoins to John Mayer.

With that file in hand, I can create a digital fingerprint using sha256.

> sha256sum will.txt

7350c9d592a9b8a514efdf050c4ef658f21f497dab2ba4c1b99d8380da9bfcab

I submit my will along with the fingeprint to the executor.

Now comes sneaky Kanye. Kanye hacks into the executor's system and then changes the will.

I declare that I sign and execute this instrument as my last will
to grant all of my million bitcoins to Kanye West.

Kanye having the day of his life

But hold on, the executor can just create the fingerprint again and check if it matches the fingerprint I provided.

> sha256sum will.txt

a3571c15b7e5436d7483d26bf01383cf75d2227b298853818c0b8a540d5b2a0b

The two fingerprints do not match. And that's how the executor will know the Will was tampered !

Kanye in shambles

Now, of course, if Kanye was smart he would have changed the fingerprint as well. All he had to do was regenerate the sha256 hash of the modified will.txt file and then saved that as the fingerprint.txt file.

Authentication is not provided by digital fingerprints. Anyone can modify the data and the corresponding fingerprint. There's no way of ensuring that the fingerprint was created by the owner. Due to this reason, digital fingerprints should not be used for security purposes.

This is where digital signatures come in play. Digital signatures offer intrigrity along with authentication.

Read about Digital Signatures on my other blog post.

Digital Fingerprints of Public Keys

Digital fingerprints are extensively used in Public-key cryptography. Public keys in general are very length. You can find my PGP public key here which is around 3000 characters long.

How would you verify that you actually have my correct public keys? Well you can simply check each character one by one or instead compare the digital fingerprint of the public key which is far less tedious. Here's the sha256 digital signature of my public key 76070d7abecae498c02ed11d0f7c41bc9ca97a7108fea93edf3d98386d930e45

1. Github's SSL Fingerprint

You can visit github.com and then click on the lock screen in the address bar to see the certificates. There you can see the sha256 and sha1 digital fingerprints of Github's SSL public key.

2. Fingerprints in GPG

GPG, an encryption tool, also shows the digital fingerprints of the public keys instead of the entire public key.

3. Fingerprints in SSH

When you try to ssh into a new system, the host system's digital fingerprint will be displayed on the prompt.

How to follow redirections on a cross origin request with no CORS enabled

Fri, 15 Jan 2021 15:00:00 GMT

If you've ever tried to perform cross origin requests to a server that doesn't support CORS then you'll get a net::ERR_FAILED which is the expected behaviour. Sometimes, we just need the redirection to occur even though we cannot access the response. This article shows how you make the browser perform the redirections when CORS isn't supported by some or even all of the urls involved in the redirection ...

Same Origin Policy (SOP)

The image below demonstrates what happens when a cross origin request is made to localhost from example.com (two different origins).

Accessing localhost from example.com

A common misconception about Same Origin Policy is that it prevents making requests from one origin to another origin. That is a false assumption. SOP prevents accessing the response, it doesn't prevent making the request.

So, in the example shown in the image above a request was actually made to localhost, although the response wasn't accessible. If we check our server log, we'll in fact see the request.

Server Logs

Now, how do we make the resources on localhost available to example.com ? To circumvent this restriction by SOP, we can use CORS.

Let's add a new endpoint that supports CORS. It's as simple as adding the header Access-Control-Allow-Origin: http://www.example.com to the response.

Enable CORS with the Access-Control-Allow-Origin header

And now if we make a request to localhost from example.com we will be able to access the response.

Access response from the client with CORS enabled

Redirection with CORS support

Let's add a new endpoint /cors-redirect that redirects the user to /cors.

Endpoint that redirects to /cors

On the browser we can make a request to this new endpoint. We'll see that the redirection does occur and the response is in fact from cors.

Redirection successful

If we check our server log, we can see all the requests that were received.

Server Logs

So far so good ...

Redirection without CORS support

Let's add a new endpoint /no-cors-redirect that redirects the user to /cors.

func handleNoCorsRedirection(w http.ResponseWriter, req *http.Request) {
  fmt.Println("[/no-cors-redirect] Request received")
  fmt.Println("Referer:", req.Header.Get("Referer"))
  w.Header().Set("Location", "/cors")
  w.WriteHeader(http.StatusFound)
}

If we try to call this endpoint, the browser will throw an error.

Error on redirection with no CORS support

Even though the endpoint /cors supports CORS, the redirection does not occur because on the very first response itself the browser will terminate the request.

It doesn't matter whether the subsequent redirected urls support CORS or not if the ones before doesn't support it. For a redirection to occur, every urls on the redirection should support CORS.

We can verify that the redirection didn't occur by checking our server log.

Server Logs

no-cors mode

The fetch API provides a special option called mode. It takes 5 possible values "same-origin", "cors", "no-cors", "navigate", or "websocket" and the default value is "cors" although the specs suggest that "no-cors" is the default value 🤷‍♂️.

Fetch Mode Specification https://fetch.spec.whatwg.org/

Let's use the mode: 'no-cors' option this time and see if the redirection occurs.

Yay ! The redirection does occur. We can verify that by checking our server log.

Server Logs

But if you noticed we have an error on the browser console. The error occurs because this time SOP prevents us from accessing the response of /cors.

Remember, the endpoint /no-cors-redirect doesn't support CORS.

And if you look at the code we are calling the response.json() which tries to access the response body.

Let's modify our code a little bit.

Opaque response type

This time we're not trying to access the response body. We're simply logging the response to the console. This is an special type of response classified as - "Opaque" response. You can see in the log above.

There other types of request too. You can read about them here - https://developer.mozilla.org/en-US/docs/Web/API/Response/type

Let's take a step back and see the response type of the requests that we have made before.

Response types

Summary

SOP prevents accessing response from a different origin
CORS helps us access response from a different origin provided that the server supports it
By default, cross origin redirects are prohibited if the server doesn't support CORS
We used mode 'no-cors' to make the redirections

This blog post was inspired by

Find the server-side go code here

Crack weak JWT HS256 secrets with a wordlist in Golang

Sat, 09 Jan 2021 12:30:00 GMT

If you're reading this article you probably know what JWTs are. Let's jump straight into the details

Symmetric key and Asymmetric key Signatures

JWT uses cryptographic keys to generate signatures. It's possible to use one of either a symmetric key or an asymmetric key. There are various factors one should consider when choosing one of these two approaches but that's a discussion for another article.

With a symmetric key, a single key is used to sign and validate the signature. Technically, with symmetric keys we are encrypting the data and producing MAC (Message Authentication Token) and not Digital Signatures. And with the asymmetric keys two different keys are used to sign and validate the signature respectively. This article focuses on symmetric key signing.

Symmetric key algorithms

One of the most popular algorithms for JWT is the HS256 algorithm. There are other variations to this algorithm like HS384 & HS512 which are more secure.

The HS256 algorithm takes in two inputs:

the message to encrypt (JWT header + JWT payload)
the secret key used to encrypt the message

Cracking JWT secrets

Since the JWT is just a base64URL encoded data we can simply decode it to see what the header and the payload are. Here's a pretty handy webapp that let's you play around with JWTs.

With that in our hand we can simply try to generate signature by bruteforcing every single possible characters and see if the generated signature matches the one on the JWT. If we find a match then we have successfully cracked the secret key.

In Golang, you can use this function to generate the HS256 signature. Find the full code here

func hs256(data, secret string) string {
	mac := hmac.New(sha256.New, []byte(secret))
	mac.Write([]byte(data))
	return base64.RawURLEncoding.EncodeToString(mac.Sum(nil))
}

If the secret key is pretty weak, bruteforcing can be fruitful. However more often than not the secret keys are pretty lengthy for this attack to work.

A dictionary attack can give us better results.

Here's a public Github repository that has scraped JWT secrets from other public github repositories. You can find it here - https://github.com/wallarm/jwt-secrets

You can try using the code I provided above to perform the dictionary attack. But there are more mature tools that are designed exactly for this purpose

John The Ripper

john token.txt --wordlist=wordlist.txt --format=HMAC-SHA256

Hashcat

hashcat -a 0 -m 16500 token.txt wordlist.txt

hashcat has better support for GPU and can perform signficantly better than john.

Recommended practice for generating JWT secrets

It's also important to note than we can perform this attack completely offline. We do not need to make any requests to the web server since we have all the data we need in the JWT itself.

With a pretty powerful GPU we can test billions of hashes per second. This is why the length of the shared secret for HS256 is of the utmost importance. JSON Web Algorithms defines the minimum key length to be equal to the size in bits of the hash function used along with the HMAC algorithm which in case of HS256 would be 256bits.

References

Basic introduction to HTTP requests with TCP Sockets in NodeJs

Sun, 22 Mar 2020 22:00:00 GMT

I love programming in NodeJs; mainly due to the speed in which I can quickly get an application up and running. Need to make an http request ? npm install axios and a few lines of code will do the trick. Need to create an http server? npm install express and with few lines you've an http server running. Now take a look at this StackOverflow answer on how to make http request in Java. There's just so much of code to write ...

But this convenience comes at a cost. All these fancy easy-to-use libraries abstract the nitty gritties that's happening under the hood.

I learned about sockets years after learning to make http requests in NodeJs. The convenience provided by RequestJs and ExpressJs never made the need for me to learn sockets. In fact I didn't even know that there's such thing as sockets just a while ago and I am sure many new devs don't either.

What are sockets ?

You've probably heard of TCP. It is one of the protocols that governs the exchange of data over network connections.

HTTP works over TCP (Any other protocol like UDP is fine as well but TCP is the standard). This means whenever we need to make an http request to a server - via whatever means like nodejs, python, curl or web browsers - we first need to establish a TCP connection with the server. Once the TCP connection is established, http requests and responses can be exchanged to-and-fro over the connection.

So how do we establish a TCP connection? That's where sockets come in. A socket is an interface provided by the operating system that enables us to read and write to a network. There are 3 types of sockes

Stream Sockets (TCP)
Datagrap Sockets (UDP)
Raw Sockets

Since HTTP is most commonly used with TCP, we are only concerned with TCP sockets in this article.

Most programming languages provide a socket interface. NodeJs provides the net library, python provides the socket library and likewise there's java.net.Socket in Java.

Zines by Julia Evans

Establish a TCP connection in NodeJs with sockets

To establish a TCP connection we need the IP Address and the port number of the two communicating devices. As an example, if I want to establish a TCP connection with google's server then I'll need

Google Server's IP Address
Google Server's Port Number
My IP Address (The OS takes care of this)
My Port Number (The OS takes care of this one too)

There's a webserver running on adityathebe.com on port 80. Let's try to establish a TCP connection with that server.

const net = require('net');

const port = 80;
const host = 'adityathebe.com';

// Create a new socket
const socket = new net.Socket();

// Establish a TCP connection
socket.connect(port, host);

socket.on('connect', () => {
  console.log(`Established a TCP connection with ${host}:${port}`);
  socket.destroy();
});

If you run the code above, you'll establish a TCP connection and then also kill it right away. TCP connections remain active for as long as days or even months or, theoretically, forever as long as there's no network connection issues. That's why it's important to kill the connection after there's no need for it.

Send an HTTP request over the socket

Now let's try to send some data over the TCP connection. The data could be anything from a simple text to a large media file. In our case, the data will be an HTTP messages.

HTTP is a really simple protocol - it's literally just plain texts! You can even craft http requests & responses by hand. There are standards & structures to HTTP messages.

HTTP Structure

As a fun demo, let's try to make an http request to example.com. We can send a very bare minimal http request like this that only consists of http header.

GET / HTTP/1.1
Host: example.com

Notice the two blank lines at the end? That's important. The blank line separates http header from the http body. If you leave out the blank line, then example.com's server will respond with an error because the http request is invalid. Give it a try... It's also important to note that the blank line is actually a CRLF \r\n and not the Line Feed \n character. Although servers these days accept the LineFeed \n character.

const net = require('net');

const port = 80;
const host = 'example.com';

// Instead of \n as a line break we use \r\n
// Because that's how it's defined in the http specification
// https://stackoverflow.com/a/5757349/6199444
const rawHttpRequest = 'GET / HTTP/1.1\r\nHost: example.com\r\n\r\n';

const socket = new net.Socket();
socket.connect(port, host);

socket.on('connect', () => {
  console.log(`Connected to ${host}:${port}`);
  console.log(`Local port ${socket.localPort}\n`);

  socket.write(rawHttpRequest);
});

socket.on('data', data => {
  // data is an array buffer. We need to transform it to string
  console.log(data.toString());

  // Close the connection
  socket.destroy();
});

HTTP Response from example.com

HTTP/1.1 200 OK
Age: 267333
Cache-Control: max-age=604800
Content-Type: text/html; charset=UTF-8
Date: Sun, 22 Mar 2020 17:23:08 GMT
Etag: "3147526947+ident"
Expires: Sun, 29 Mar 2020 17:23:08 GMT
Last-Modified: Thu, 17 Oct 2019 07:18:26 GMT
Server: ECS (nyb/1D2A)
Vary: Accept-Encoding
X-Cache: HIT
Content-Length: 1256

<!doctype html>
<html>
<head>
  ...
<body>
<div>
  <h1>Example Domain</h1>
  ...
</div>
</body>
</html>

I hope this was fun and helpful. Under the hood, the libraries like axios, request, express, etc are doing all of these and much more ...

A handy way to know your public IP Address with DNS servers

Mon, 10 Feb 2020 08:45:00 GMT

There are plenty websites that show you your public IP address when you visit them. The simplest way would be to just ask google. These sites get the job done but when you want to constantly monitor your public ip it becomes tedious to keep visiting them. You need to have a browser running and the website itself takes 3-4 seconds to load.

There are some handy api services like ipify API. If you visit https://api.ipify.org/ you'll get a plain text http response. This method is slightly faster than asking google due to the smaller response size. It's also handy as you can use it in creating scripts.

Ipify Api Service

But there's a faster method. Much much faster! It's crazy how fast it is! It takes no more than a 100-200 milliseconds. This method involves quering a DNS server rather than an HTTP server.

I use the dig command to make DNS queries. Dig is a DNS lookup utility that comes preinstalled in most linux distros.

You can use dig to find the ip address of a website like this.

# This will spill out a bunch of metadata
dig adityathebe.com

# This will simply return the ip address
dig +short adityathebe.com

But we're here to find the ip address of our router and not some website's. In order to get the ip address of our router, we need to query a special url myip.opendns.com to the DNS resolver resolver1.opendns.com.

This DNS resolver is programmed to return the requesters ip address when a request is made to that special url. If you simply query myip.opendns.com with any other DNS resolvers then you'll get NXDOMAIN response; which basically means that the domain does not exist.

# To know you public ipv4 address
dig +short myip.opendns.com @resolver1.opendns.com -4

# To know you public ipv6 address
dig +short AAAA myip.opendns.com @resolver1.opendns.com

Besides OpenDNS, Google and Akamai also provide a similar feature

dig @ns1-1.akamaitech.net ANY whoami.akamai.net +short

dig @ns1.google.com TXT o-o.myaddr.l.google.com +short

Akamai's service was a bit slow for me.

Alias

I have added an alias to my zshrc file so that I can query my public ip address from the terminal by simply typing publicip.

alias publicip='dig +short myip.opendns.com @resolver1.opendns.com -4'

Polybar

# Polybar module

[module/publicip]
type = custom/script
exec = dig @resolver1.opendns.com A myip.opendns.com +short -4
interval = 100
label =  %output%

Reference

What does it mean to own bitcoins

Sun, 02 Feb 2020 00:05:00 GMT

What does it mean to "have" bitcoins? Where exactly are they stored? Where can one sign up for a bitcoin account? These were some of the questions that I had when I first started to study Bitcoin.

The notion of owning bitcoins can be difficult to grasp to anyone who has dealt with cash all their lives. There's nothing physical to hold and everything is digital.

When Bob says he has $5 we can understand that he either has a $5 bill or $5 in his bank account. But what does it mean when Alice says she owns 5 BTC? Well, that's what we are going to find out ...

Bitcoin can mean both the currency and the protocol that governs the currency. The Bitcoin protocol differs fundamentally from our traditional financial system. And so to understand it, it's important to learn the fundamental concepts and the underlying building blocks.

Blockchain

The blockchain is a double-entry bookkeeping ledger. The name is derived from the fact that it is literally a linear chain of blocks. A block consists of numerous transactions. Every single bitcoin transaction that has ever happened has an entry in one of the blocks in the blockchain.

Structure of a blockchain. Source: https://blockspoint.com

The blockchain is public, and anyone can download it and look through the transactions that's not just theirs but anyone else's. If I have your bitcoin address then I can see all the transactions going in and out from your address. Despite the popular belief that Bitcoin is untraceable, it is actually the opposite. In fact, one would need to take extreme measures to stay anonymous in the Bitcoin network.

The important thing to understand, for now, is just that the blockchain is a database of transactions.

Every single bitcoin transaction that has ever happened has an entry in one of the blocks in the blockchain.

Transactions

A bitcoin transaction is just an entry in the blockchain that indicates transfer of bitcoins. Transaction of bitcoins is quite different from a banking transaction. In a banking system, you have an account where you store all your money. To make a transaction you'd transfer money from your account to the receiver's account. In Bitcoin, there's no such thing as an Account.

Recall, the blockchain is just a list of transactions. There is no such entry in the blockchain that states that Bob's account has 5 BTC or even that there's an account that belongs to Bob. To put it precisely, there's no concept of balance in Bitcoin.

So how and where does one store bitcoins? To answer that we need to understand transactions.

A transaction comprises of a bunch of metadata which are shown below.

Structure of a bitcoin transaction. Source: Mastering Bitcoin by Andreas Antonopolous

The fields that are of interest to us are inputs and outputs. The input to a transaction is the source of funds, and the output is the destination. If there are no accounts in Bitcoin, then what is the source of funds and where is the destination?

Transaction Outputs

The output of a transaction is called UTXO (Unspent Transaction Outputs). A UTXO consists of a locking script & the amount that is to be transferred. The locking script locks the transaction amount. For someone to use the transaction amount, an unlocking script is required.

Structure of a Bitcoin UTXO. Source: Mastering Bitcoin by Andreas Antonopolous

Think of UTXO as a safe-deposit box that locks some amount of bitcoin. Anyone that has the key to that safe can spend that bitcoin.

So to make a transaction, Bob asks Alice for her bitcoin address. He then creates a UTXO with an amount of 5 BTC and a locking script using Alice's bitcoin address such that only Alice can create an unlocking script using her private key.

Sending someone bitcoin is basically creating an unspent transaction output (UTXO) that is cryptographically locked and can only be accessed by the receiver's private key.

Transaction Inputs

The input to a transaction is actually a pointer to another transaction that's in the blockchain. Precisely, the input is a pointer to a UTXO.

Structure of a transaction input

So essentially, when Bob wants to make the 5 BTC transaction, he needs to provide the following input.

A transaction hash which is a unique id that identifies the transaction.
An output index that points to a UTXO (fund source) in that transaction referenced by the transaction hash (there can be more than one UTXO in a single transaction).
An unlocking script indicating authority over the UTXO.

This transaction will be placed on the blockchain as a new transaction with a unique transaction hash. In future, Alice can refer to this transaction and the UTXO to use the fund locked in the UTXO.

Bitcoin UTXO transaction model. Source: https://medium.com/coinmonks

Calculating Bitcoin Balance

You might have seen some bitcoin wallet applications that show the balance of a bitcoin address.

wallet application

If there are no accounts then how is the bitcoin balance calculated? The concept of a user’s bitcoin balance is a derived construct created by the wallet application.

The Bitcoin network actually keeps track of all the available unspent UTXO. A wallet application calculates the user’s balance by scanning the blockchain and aggregating all UTXO belonging to that user.

A user's bitcoin balance is the sum of all unspent UTXO that can be unlocked by the user.

How are Bitcoins generated?

Hold on... so Alice got her 5 BTC from Bob. Bob must have gotten them from someone else and that someone must have gotten from some other person. But where is the point of origination in this chain of ownership? How are bitcoins added into the blockchain? Who adds them?

Bitcoins are generated by miners. They are the ones who introduce new bitcoins into the Bitcoin network.

Did you not find it odd that the transactions are stored in blocks? Why is it not just a single block but rather a chain of blocks? The answer to that certainly deserves another article but just understand that when you make a transaction it does not get added into the blockchain immediately.

When you press "send" in your wallet application you are basically announcing the network that you want to make a transaction. Your wallet application sends this transaction information to all the nodes they are connected to and then those nodes in turn send the transaction metadata to the nodes they are connected to. This way eventually the transaction metadata will reach every single node connected to the Bitcoin network. Some of the nodes are special nodes called Mining Nodes.

The mining nodes receive lots of transaction information and they store them in their computer's memory. The task of a miner is to put the transactions in the blockchain in a new block. This process of placing a block into the blockchain is called Mining.

Mining bitcoins requires making a lot of autmoated calculations and only those miners with very powerful mining devices have a chance at successfully mining a block. Miners compete against each other to put a block into the blockchain and when one of them successfully mines a block, he/she gets some amount of BTC as a reward called a Block Reward. Unlike a normal transaction, a block reward does not require an input. The money is generated out of thin air. This is the only way bitcoins are added into the blockchain.

The block reward gets halved every 210,000 blocks or approximately every 4 years. When Bitcoin started in Jan 2009, the block reward was 50 BTC. On November 28, 2012 it dropped to 25 BTC per block and on July 9, 2016, it droped again to 12.5 BTC per block. It is estimated that on May 2020 the block reward will again get halved to 6.25 BTC per block.

The block reward will keep on decreasing exponentially. It is projected that in the year 2140, the last halvening will take place and there'll be no more block rewards. This essentially means that there is a limit to the production of bitcoins. It's 21 million and more than 18 million of them have already been mined.

Tldr

There are no bitcoin accounts.
Bitcoins are locked in UTXO in the blockchain. In order to use the UTXO, a private key is required. Thus, a user's bitcoin may be scattered as UTXO amongst hundreds of transactions and hundreds of blocks.
The UTXO consumed by a transaction are called transaction inputs, while the UTXO created by a transaction are called transaction outputs
Miners add the transactions in blocks into the blockchain. They get rewarded some amount of BTC for successfully adding a block into the blockchain. This is how new bitcoins are generated and it's the only way.

Most of the things that I've written in this blog post, I've learned it from the book "Mastering Bitcoin" by Andreas Antonopolous. It's open sourced !!

A beginner's guide to installing softwares in a Linux system

Mon, 27 Jan 2020 12:00:00 GMT

One of the main reasons I migrated to Linux from Windows was the ease of software management. I could install, update and remove "any" software with just a few commands from the command line.

Life was fun and easy until it wasn't when I had to install a software which wasn't in the distro's software repository. A distro's software repository is basically a database of application installation packages and upgrade packages. If the software you want to install is not listed in the software repository then you'll have to look for other ways to install.

In Windows, almost every software provides an installation wizard but that's rarely the case in Linux.

So, what are some other ways to install a software when it's not available to install via the package manager? That's what you will be learning in this article.

1. Distro's Package Manager

This is my (and most peoples') preferred way of installing softwares because the distro's package manager makes it extremely easy, to not only just install, but also to update, remove and manage paths & the launchers of a software.

Ubuntu uses apt for package management and so does Debian and Linux Mint. Different distros have different package managers. Arch based distros use pacman, Red hat uses rpm, CentOS uses yum ...

Every distro has its own set of software repositories. What may be available to install with apt in Ubuntu may not be available to install in Debian even though both distros use apt.

The package manager a distro provides can be a deciding factor in choosing a distro for many people. To give you an example: Debian is known for its solid stability and hence the softwares available in its official repositories may not be a latest release but a stable one. So Debian may not be a good choice for people who want the latest bleeding edge softwares.

The package manager will not always be able to help you because well obviously the software repository cannot include every software in existence. Sometimes although the software is available, the required version isn't.

Apart from the official software repositories, you can also add third-party software repositories. You may have noticed Google Chrome is not listed in any of the Ubuntu software repositories and so running sudo apt update google-chrome will not work. There are various other ways to install it and one of them is to add the Google Chrome's sofware repository. Once you add the repository, it's the same basic routine to install any software from that repository.

Read: How to install Google Chrome in Ubuntu

If in case the software is not available through third-party repositories then you'll have to look for other options mentioned below.

2. A programming language's package manager

Some softwares are available to install through a programming language's package manager. Python has pip, NodeJs has npm (or yarn), GoLang provides the convenient go get command and there are a lot others.

Some of the softwares I install via a programming language's package managers are

localtunnel, tldr, http-server, typescript (via NPM)
wpm, sublist3r, httpie (via pip)
amass (via go get)

It goes without saying that you need to have the programming language and the package manager installed. As far as I know, the package manager does come along when you install the programming language.

3. Compile the source code

The above two methods are fairly straightforward. Now comes the time to get our hands dirty.

Some softwares provide the source code, which could be in C, C++, Go, ... you name it, and you're supposed to compile them. Being provided with the source code enables a user to compile the the source code specifically tailored and optimized to his/her system.

I have to say this is not an easy thing to do although I've never done it myself.

I am skipping this section for now

4. Download the compiled binary

Conveniently, you can also find the compiled binaries of a software. It may be provided by the official software developers or by some third party. The compiled binary is usually archived in a single tar file. In some rare cases you may find a zip or a rar archive. All you need to do to install the software simply download the tar file, extract it and then run the binary file inside the extracted directory.

As for an example: Jetbrains provide the compiled binary for its IDEs. Check out the image below. Look! There's the tar file I told you about.

There are handful of softwares that provide the compiled binaries in a tar file. Here are some to name a few:

Once you download the tar file you should extract it with the tar command.

# Extract an archive into a target directory:
# tar xf source.tar -C directory

# Extract Intellij Idea to /opt directory
sudo tar xf ideaIC-2019.3.2.tar.gz -C /opt

If you look at the extracted directory you'll see a binary file. You may need to look a bit further if you do not find it in the main directory. Some softwares place the binary file in a /bin directory while some place them in the main directory itself. This directory structuring is unique to each software, but, more often than not, you'll find the binary file in the bin directory.

Here's what the extracted directory of the Intellij tar file looks like

ls -al idea-IC-193.6015.39

total 68K
drwxr-xr-x  8 root root 4.0K Jan 27 20:50 .
drwxr-xr-x  4 root root 4.0K Jan 27 20:50 ..
drwxr-xr-x  2 root root 4.0K Jan 27 20:50 bin (binary file in here ??)
drwxr-xr-x  7 root root 4.0K Jan 27 20:50 jbr
drwxr-xr-x  4 root root  12K Jan 27 20:50 lib
drwxr-xr-x  2 root root 4.0K Jan 27 20:50 license
drwxr-xr-x 44 root root 4.0K Jan 27 20:50 plugins
drwxr-xr-x  2 root root 4.0K Jan 27 20:50 redist
-rw-r--r--  1 root root   14 Jan 21 12:01 build.txt
-rw-r--r--  1 root root 1.9K Jan 21 12:01 Install-Linux-tar.txt
-rw-r--r--  1 root root  12K Jan 21 12:01 LICENSE.txt
-rw-r--r--  1 root root  128 Jan 21 12:01 NOTICE.txt
-rw-r--r--  1 root root  370 Jan 21 12:04 product-info.json

You can run the software by simply running the binary file. To run, open the terminal in the extracted directory and simply write the name of the binary file prepending it with ./. In case of Intellij, there's an idea.sh file inside the /bin directory.

# To Run Intellij
./idea.sh

Where to place the extracted directory ?

Practically you could place it anywhere you want; be it the /home directory or even the /Downloads directory itself but, it's a standard practice to place it in one of these two directories

/usr/local
/opt

Did you notice that I extracted the IntelliJ tar file inside the /opt directory?

You might want to checkout these discussions in stackoverflow to get a better idea on where to place the software directory

How to add path to PATH variable

Isn't it a bit tiresome to first open the terminal, then navigate to the software directory and only then being able to run the program? If you want to be able to run the program from anywhere in the terminal then you need to add the software's binary path to the Linux's PATH variable.

To do so, first figure out where the binary file resides. In my Intellij example, it resides in the /opt/idea-IC-193.6015.39/bin directory.

Now add this line to the ~/.profile file.

# export PATH=$PATH:<new-path>

# Add Intellij binary to path
export PATH=$PATH:/opt/idea-IC-193.6015.39/bin

You won't be able to run the idea command just yet because the ~/.profile is read only once when the computer first starts up. You can restart your system and everything will work fine. However, you could instead add line to ~/.bash_profile file instead and the command will run immediately. The ~/.bash_profile is run everytime you open your terminal. But, it's considered a good practice to export PATH variables in the .profile file.

This post is just much as a guide to others as it is to me. I'll keep updating this post as I learn ...

Configure ssh keys on Github

Fri, 24 Jan 2020 23:30:00 GMT

The reason I like to use ssh over https is because I hate to provide my username and password on every remote actions like git clone, git pull, or git push. Sure, I could use a credential manager but why would I do that when I can simply configure ssh keys ?

On the contrary, Github actually suggest using https over ssh mainly because it is the easiest to set up for novice users and https is almost never blocked by firewalls.

1. Generate ssh keys

First thing first - make sure you have a ~/.ssh directory.

mkdir ~/.ssh

This is the file in which we'll store our keys and configuration for ssh.

Create an ssh key pair with ssh-keygen command. It'll ask you for the key name - I'd suggest using id_rsa_github just to stay consistent with this post.

cd ~/.ssh && ssh-keygen -t rsa -b 4096 -C "<your_email>"

Generating public/private rsa key pair.
Enter file in which to save the key (/home/gunners/.ssh/id_rsa): id_rsa_github
Enter passphrase (empty for no passphrase):
Enter same passphrase again:

This command generates two files: id_rsa_github and id_rsa_github.pub in your ~/.ssh directory. One is a public key which you should upload on Github and the other is a private key that you should keep on your local computer.

2. Upload the public key to Github.

Go to https://github.com/settings/keys and click on New SSH Key.

Add a suitable title and copy all the content from the id_rsa_github.pub file to the key section. The pub file is just a simple plain text file. You can open it with any text editor like notepad, sublime text, gedit, etc...

3. Add ssh-keys

If you try to perform some remote actions, like cloning a repo, you'll get an error like this.

git clone git@github.com:adityathebe/dotfiles.git

Cloning into 'dotfiles'...
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

This is because git does not know where to look for the ssh key (Remember the id_rsa_github key we generated in ~/.ssh directory?)

So to let git know that you do indeed have an ssh key use these two commands

eval "$(ssh-agent -s)"

ssh-add ~/.ssh/id_rsa_github

Now you can perform any git remote actions.

However, if you close the current terminal session and try to clone again, you'll get the same error.

You could add the keys manually again with the ssh-add command but I'll show you a much elegant way.

Permanently add the ssh-keys

Now make sure you have the ~/.ssh/config file. If it's not there you can simply create it

touch ~/.ssh/config

It's important the config file has it's permission set to 600.

chmod 600 ~/.ssh/config

This config file is read by the ssh command on every execution.

Add the following lines to the ~/.ssh/config file.

Host github.com
  User git
  IdentityFile ~/.ssh/id_rsa_github

And that's it !

Beware !!

You might still be prompted for your password. This happens if you initially cloned the repository with an https url. So make sure the remote origin url is an ssh url.

1. Check your existing remote's urls

git remote -v

origin  https://github.com/adityathebe/www.adityathebe.com.git (fetch)
origin  https://github.com/adityathebe/www.adityathebe.com.git (push)

2. Change your remote's URL from HTTPS to SSH

Grab the ssh clone url from Github

git remote set-url origin git@github.com:adityathebe/www.adityathebe.com.git

3. Verify

git remote -v

origin  git@github.com:adityathebe/www.adityathebe.com.git (fetch)
origin  git@github.com:adityathebe/www.adityathebe.com.git (push)

Be sure to clone with ssh from now on :)

Setup mpd on your Linux system as a User service

Fri, 24 Jan 2020 00:00:00 GMT

In this tutorial we'll go through the process of setting up mpd as a user service. This is just a bare minimum installation setup. If you want customizations then you can look up dotfiles on the internet - lots of people share it. Here's mine.

First thing first - download the mpd package along with the mpd clients mpc and ncmpcpp.

# On Arch
sudo pacman -S mpd ncmpcpp mpc

# On Ubuntu
sudo apt install mpd ncmpcpp mpc

You can choose other mpd clients instead of ncmpcpp of course.

Setup the config files

It's a good practice to keep all the mpd configs in the ~/.config/mpd directory. If you don't have one create it.

Just to get started, you can copy the example config file that mpd provides.

cp /usr/share/doc/mpd/mpdconf.example ~/.config/mpd/mpd.conf

In order to be able to run mpd, you'll need to set up at least these parameters in the mpd.conf file

music_directory       "~/Music"
playlist_directory    "~/.config/mpd/playlists"
db_file               "~/.config/mpd/database"
log_file              "~/.config/mpd/database"
pid_file              "~/.config/mpd/pid"
state_file            "~/.config/mpd/state"

It's not necessary to create any of these files as mpd will itself create them. However, you'll need to create a new "playlists" directory to create new playlists.

Start the service as a user

systemctl --user enable mpd
systemctl --user start mpd

Don't forget the --user flag !

Update the music database

mpc update

For some reason this command alone does not work me. So I run an additional command

mpc ls | mpc add

That's it ! You should now be able to run ncmpcpp and see all your songs.

Twitter Bot Tutorial for Beginners In NodeJS

Wed, 07 Nov 2018 00:00:00 GMT

Chat bots are growing in power with the increase in social media users and advancement in technology. They will be the app of the future and heck they probably already are. They can be useful in many aspects of the customer experience, including providing customer service, presenting product recommendations and engaging customers through targeted marketing campaigns. Although chat bots have still a long way to go in terms of AI, they're still powerful.

We'll code our Twitter Bot in JavaScript - NodeJS. I have tried my best to be as detailed as possible and I hope you won't have any trouble following along.

Create A Twitter App
Set up and Install NodeJs
Project Setup
Install Twit Module
Code
Deploy on Heroku

1. CREATING A TWITTER APP

You can't create a bot without a twitter account. Sign up for a new account here.

Great! Now we'll need something called API keys. API keys are the 'username and password' for your bot. Your Bot will require them to make API calls, or in simple words, interact with the data on twitter's server. To get your API keys, you need to first create a Twitter app. Go to apps.twitter.com to get them.

Create new app on Twitter

Click on "Create New app" and fill up all the necessary details on the following page. If you don't have a website just put a placeholder in the "Website field". You can leave out the Callback URL field. Tick the Developer Agreement and then finally click on 'Create Your Twitter application' at the bottom. We now have a Twitter application. Let's get the API keys.

Create new app on Twitter

On the next page go to the "Key and access Tokens" tab. We'll need four different key from here out of which 2 (Consumer Key and Consumer Secret) have already been created for us. To get the other 2 click on "Create my access tokens" at the bottom of the page, .

Grab the api keys

If you did everything right, you shall now have the following keys:

Consumer Key
Consumer Secret
Access Token
Access Token Secret

NEVER EVER SHARE YOUR KEYS

Fire up your text editor and create a new file to hold all the configurations. Let's name it config.js for convenience. Copy the code below to config.js and fill up the necessary details.

module.exports = {
  consumer_key: '<YOUR_CONSUMER_KEY_HERE>',
  consumer_secret: '<YOUR_CONSUMER_SECRET_KEY_HERE',
  access_token: 'YOUR_ACCESS_TOKEN_HERE',
  access_token_secret: 'YOUR_ACCESS_TOKEN_SECRET_HERE',
};

Do not modify the parameters. For example : Do not change consumer_key to Consumer Key. The format should be exactly as shown above.

2. Setup and Install NodeJS

Now, let's install NodeJS. The instructions will vary a bit depending on your operating system, so just follow the official guide - it's pretty simple.

Once you have installed NodeJS fire up your command line and type

node --version

If you get something like v6.10.3 then your installation was successful. The number you get back is the version of Node. If you get something like 'not recognized', there was some error in the installation. Try reinstalling.

NodeJS comes with NPM (Node Package Manger) with which you can download NPM packages. They are simply JavaScript libraries. Once again in your command line, type

npm --version

You should get back a version number. Great! We now have all the requirements fulfilled.

3. Project Setup

Hold your horses folks. Let's set up our project first. Few more minutes and then we'll get into coding - I promise !

Create a folder to store all your bot files. Inside that folder you should have the config.js file we created earlier. Create another file called bot.js - this is the file where we'll be writing all of logic.

Well, every NodeJS project requires a package.json file. This file will contain all the metadata of our project like the project name, project version, description, license and most importantly the dependencies. Dependencies are the node modules that we use in our program. You can manually create this file but there's a much better way. On your command line navigate to the project folder and then type the following command

npm init -f

This command creates the package.json file. You folder should have 3 files by now

config.js
bot.js
package.json

4. Install Twit Module

To communicate with the twitter server we'll use a NPM package called Twit. This is the only package we'll need. On your terminal run the following command

npm install --save twit

This will install the Twit package. If you check your package.json file you should see "twit" in your dependencies.

{
  "name": "your-project-name",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "dependencies": {
    "twit": "^2.2.9" // twit module
  }
}

Every time you install a package it'll automatically be added to your package.json file.

It's very important that you are on the project folder on your command line while running these commands

5. Code

It's time to write some code! Open the bot.js file and copy these lines.

// Import NPM package
const Twit = require('twit');

// Import Config File
const config = require('./config');

The first line of our code imports the Twit package and assigns it to the Twit variable. Similarly the second line of code imports the config.js file on our Folder. Make sure you have added all the required consumer keys and access tokens on the config.js file.

We now create a Twit instance.

// Create a Twit Instance
const Bot = new Twit(config);

The variable Bot holds the Twit instance. Bot has a function called post() which enables us to make a tweet. This post function takes 3 different arguments

method
input
callback function

// tweet 'hello world!'
Bot.post('statuses/update', { status: 'Hello World' }, (err, data) => {
  if (err) return console.log(err);
  console.log('Successfully Tweeted');
});

Let's breakdown and understand this code.

a. The first argument: –

'statuses/update' means we want to update the status. There are other various methods like 'media/upload' to upload a media, 'favorites/create' to favorite a tweet, 'direct_messages/new' to send direct messages, and many more. Read more about Twitter API here.

b. The second argument: –

{ status: 'hello world!' } is a JavaScript object with a property 'status' and a value of 'hello world!'. This means we want to make a tweet 'hello world'. To make any other tweets just edit the 'hello world!' part like this { status: 'Best bot ever' } , { status: 'I love Twitter Bots' } …

c. The Third Argument:-

This argument is a function or more specifically a callback function.

So now what our function does is updates the status 'hello world!' right? As soon as the post function makes the tweet or if there occurs any error while tweeting, this callback function is invoked!

Inside this function we have three variables 'error', 'data' and 'response'. If any error occurs while tweeting, this error variable will contain the information of the error, if everything goes right then we'll get a response.

Now your bot.js file should look something like this

const Twit = require('twit');
const config = require('./config');

let Bot = new Twit(config);

Bot.post('statuses/update', { status: 'Hello World' }, (err, data) => {
  if (err) return console.log(err);
  console.log('Successfully Tweeted');
});

Great! With these lines of code we can already make a tweet. Open up your command line and navigate to your project folder. Now run the command

node bot.js

This should make a tweet! Go check your bot's twitter account.

It's tedious to write that long string of function to make a tweet. Let's wrap that Twit.post function in our own custom function.

const Twit = require('twit');
const config = require('./config');

let Bot = new Twit(config);

const makeTweet = tweet => {
  Bot.post('statuses/update', { status: tweet }, (err, data) => {
    if (err) return console.log(err);
    console.log('Tweet Made Successfully');
  });
};

makeTweet("I am tweeting from my command line and it's awesome");

Now everytime we need to make a tweet we can just call the makeTweet() function and pass in the tweet like this

makeTweet("I am tweeting from my command line and it's awesome");

This is great but what's the fun if we need to run a command everytime we need to make a tweet? Let's automate the process so that our bot will make a tweet in certain time interval. We can achieve this by using the setInterval() function in JavaScript that runs a task in a certain set of intervals. Add this line of code at the bottom

setInterval(makeTweet, 60000, Math.random() * 20);

This code runs the makeTweet() function every 60 seconds ( 60,000 milliseconds). If you run the bot from your command line the bot will tweet a random number every minute.

Do not spam and post every few seconds or so. Twitter will block your app!

Streaming Tweets

Alright so we have made a bot that can post tweets. But that's not enough. We will step up our game by streaming realtime tweets! For example, we can track any words like 'Donald Trump' so that whenever someone makes a tweet with the word 'Donald Trump' anywhere in the tweet, our bot will grab that tweet in realtime. We can track twitter accounts, words, hashtags, etc with this method.

Lets create a trackTweet function. This function takes a parameter, that we name keyword, which stores the word we want to track. Once someone makes a tweet with that keyword, we'll program our bot to print the tweet in the command line. Let's track 'trump'.

const Twit = require('twit');
const config = require('./config');

let Bot = new Twit(config);

const makeTweet = tweet => {
  Bot.post('statuses/update', { status: tweet }, (err, data) => {
    if (err) return console.log(err);
    console.log('Tweet Made Successfully');
  });
};

// Lets not call this function right now
// setInterval(makeTweet, 60 * 1000, Math.random() * 20)

const trackTweet = keyword => {
  let stream = Bot.stream('statuses/filter', { track: keyword });
  stream.on('tweet', function(tweet) {
    console.log(tweet.text);
  });
};

trackTweet('trump');

The last line calls the function and the bot will start tracking every tweets with 'trump'

Track tweets and favorite the tweet

We're not done yet! Currently the trackTweet function track tweets and print it on the command line. Let's modify it so that we can not only track tweets but also favorite the ones we like.

We'll create a favoriteTweet function that will favorite tweets we pass to it. Then inside our trackTweet function we'll use this function to favorite tweets we like.

const Twit = require('twit');
const config = require('./config');

let Bot = new Twit(config);

const makeTweet = tweet => {
  Bot.post('statuses/update', { status: tweet }, (err, data) => {
    if (err) return console.log(err);
    console.log('Tweet Made Successfully');
  });
};

const favoriteTweet = tweet_id => {
  Bot.post('favorites/create', { id: tweet_id }, (err, data) => {
    if (err) return console.log(err);
    console.log('Successfully favorited Tweet!');
  });
};

const trackTweet = keyword => {
  let stream = Bot.stream('statuses/filter', { track: keyword });
  stream.on('tweet', function(tweet) {
    console.log(tweet.text);
    favoriteTweet(tweet.id_str);
  });
};

trackTweet('nepal');
// setInterval(makeTweet, 60 * 1000, Math.random() * 20)

Do you see some resemblance in the favoriteTweet() function we just created? It is almost identical to the makeTweet() function. The only difference is the parameter passed to them.

Go to the Twitter API Documentation and try to make a function that follows a twitter user. Please take your time and try to figure this on your own because once you learn to read the documentation you can add plenty more features that twitter provides.

The code must look something like this.

// Pass it the username and your bot will follow the user.
const followUser = username => {
  Bot.post('friendships/create', { screen_name: username }, (err, data) => {
    if (err) return console.log("Couldn't Follow!");
    console.log('Followed!', username);
  });
};

We have created a twitter bot that tracks a keyword and favorites the tweet if it finds one. I know it's not the most exciting bot. I tried to focus this tutorial mainly for beginners. To summarize this tutorial, We have learnt to

Track realtime tweets
Make tweets
Follow users
Favorite tweets

There's still so much more to explore! Visit the Twitter API Documentation and try adding more features.

6. Deploy on Heroku

I know you must be thinking "Wait a minute, do I have to keep my computer running for this bot to function?". I have a good news for you - NO, you can actually run your bot on the cloud. One cloud platform that I can vouch for is Heroku (mainly because it is very easy to setup and there's a free package that doesn't require credit card). The free package, of course, has limitations. A very limited compute-time is provided to you but that should be more than enough for this tutorial.

A. Create a Heroku account

Create an account here. If you create a free account then you can host up to 5 different applications. Once you create a new account create a new app. Now there's a whole bunch of things we need to do here. Basically there are 3 different ways we can add our code on heroku

Through Heroku Command Line (Preffered and the programmer's way)
Github (Easy way)
Dropbox (Easy way)

I would recommend the first way although it's a bit harder to get at first. However you can just follow along by using the syntax I provide here.

B. Heroku Command Line Interface (CLI) and GIT:

Download Heroku Command Line from here. We need to download one more thing and that's git – a CLI for GitHub. Download git here. Run those two installation files. If you have any trouble installing look for tutorials online, there are plenty.

Deploy on Heroku

By now you should have Heroku CLI and GIT installed. To check if you have successfully installed those two application open your command line and type

heroku --verion

If you get a reply something like this

heroku-cli/6.14.22-f598c4a (windows-x64) node-v8.4.0

then the app was installed successfully. Likewise type

git --version

and you should see the git version. Please proceed to rest of the tutorial only if you have installed Heroku CLI & Git.

C. Creating a Procfile

Heroku requires a special file called Procfile. If you're hosting your bot on any other hosting service then this file is not require. This file does not have any extension like .js or .txt, but only a name 'Procfile'. So in your Project folder, create a new file and name it as Procfile. Inside this file, write

worker: node bot.js

and save it. That's all we need to do with Procfile.

This Procfile tells heroku to run node bot.js command just like you did in your command line. The worker is a type of a thing called dyno in heroku which I better not talk about right now.

D. Creating a New App in Heroku

On your terminal, type heroku login. Type your Heroku email and password and log in. Now, to create a new app type the following command

heroku create <your_app_name>

Replace <you_app_name> with your app name. The app name should be in lowercase. We're done. That's how easy it is to create an app in heroku. All we need to do now is upload our code to our Heroku app.

Login to your heroku account and you'll see a list of all your apps. Click on the app you created just now and then go to the Deploy tab section. If you scroll down you'll see this :

As I mentioned earlier, there are various methods to deploy your code on heroku. We are using the Heroku CLI because its cooler than using GUI 😀 .

E. Creating a Git Repository and Uploading our Twitter Bot code:

We first need to create a Git Repository of our bot and only then we can upload (push) it to Heroku. Open command line and navigate to your project file and type

git init

This command initializes an empty git repository. Now we must add all the files in our project to the repository. To do so use the following two commands

git add .
git commit -m "first commit"

Great! Now before we upload our code, we first need to tell where to actually upload the code. We can do that by using the command below

heroku remote:set -a <your app name>

Now the final step is to upload(push) the code. Use the following command

git push heroku master

F. Final Step – Turning on Worker Dyno

There's one last thing we need to do. Go to your heroku app page on your browser. On the 'Resources' tab you should see your dynos.

Heroku Dynos

We need to turn on the worker dyno to start our bot. Click on the edit button and disable the web dyno and turn on worker dyno. This is where the Procfile comes into play! Without the Procfile, we wouldn't have this option to turn on worker dyno.

If you've read this far I hope you enjoyed it :)

Seive's Algorithm To Calculate The Sum Of First Million Prime Numbers

Fri, 16 Mar 2018 00:00:00 GMT

If you're a programmer there's a good chance that you have, at least once, written a prime detection function. This problem seems easy enough for anyone who knows what a prime number is and in fact it is ! But many of us approach it the wrong way. The most obvious way would be to make a function that classifies a number as prime or composite and then another function that loops through all the numbers say upto a million and then checks each number with the classifier function. There's nothing wrong with this but depending on the prime classifier function, calculating sum of all prime numbers upto 1 million can take anywhere from 1 second to 10 minutes ! Let me write a prime classifier function first ...

const isPrimeNumber = num => {
  if (num < 2) return false;
  for (let i = 2; i < num; i += 1) {
    if (num % i === 0) return false;
  }
  return true;
};

Now to get the sum we loop over all numbers upto 1 million

let sum = 0;
for (let i = 2; i < 1000000; i += 1) {
  if (isPrimeNumber(i)) sum += i;
}

Lets see what's wrong with this approach. To classify a prime number say 11, we divide it by every numbers starting from 2 upto 10. To classify 13 we repeat the same process, divide it by every numbers starting from 2 to 12. You can see where this is going. As the number increases, each classification will take more time than the previous one. In CS term the Big O of this approach is O(n²) which is really really bad for an algorithm.

Optimization

We can optimize this by not classifying even numbers, except 2, since they are for sure not prime. By increasing the counter up by 2 in the two for loops we can ignore all the even numbers. That would decrease the processing time by half. But that's still way off from being a good algorithm.

We can further optimize by taking divisors upto the square root of num. In our for loop in checkPrime function the loop should only continue up to √num. We have now significantly decreased the amount of divisors but this is still not good for large value of N.

Primality Test

There's another much better way to classify a number as prime - the "Primality Test Algorithm". This algorithm does not give prime factors but only state whether the input number is prime or not. Hence there's much less computation to do.

const primality = num => {
  if (num <= 1) return false;
  if (num <= 3) return true;
  if (num % 2 === 0 || num % 3 === 0) return false;

  let i = 5;
  while (i * i <= num) {
    if (num % i === 0 || num % (i + 2) === 0) {
      return false;
    }
    i = i + 6;
  }
  return true;
};

I used both the algorithms to classify the 50 millionth prime number 982,451,653 and the result was:

# first Algorithm: 2952.744ms
# primality: 0.527ms

Similary, I used those two algorithms to get the sum of all primes below 1 million and this was the result

console.time('runtime');
for (let i = 0; i < 1000000; i++) {
  if (isPrimeNumber(i)) {
    sum += i;
  }
}
console.timeEnd('runtime');

# First algorithm : 705403.287ms // About 11 minutes
# Using Primality algorithm : 117.574ms

Wot !! That's a massive performance gain if you ask me. Primality Test is in fact considered to be the fastest prime detection algorithm. But wait there's another even better algorithm for this type of prime calculations. This next algorithm gets the same result in just 50ms.

Sieve's Algorithm

The Sieve of Eratosthenes is a simple, ancient algorithm to find all the prime numbers that are less than a given number. This algorithm cannot directly classify a number X as prime or composite but will generate list of all prime numbers that are less than X.

Sieve's Algorithm

Here's how it works. Let's say we need to get all the prime numbers upto a number N.

Step 1: We generate an array of numbers starting from 2 to N-1. So now we have N-2 elements in the array.

Step 2: Then all the elements of the array are set to boolean value TRUE. True means they are prime (yet). But as we proceed we'll have many False values. It's analogous to how we filter out unwanted particles using a sieve. At first all of them are in the sieve machine (set to True) and then at last only the required particles (prime numbers) are present.

Source: www.tigtagcarolina.com

Step 3: Now starting from the 2nd index of the array to the last, we remove the multiples of the number. But first we need to check if the current number ( or index of array ) is TRUE of false. If the value is TRUE, we calculate its multiple and if it's false we don't.

So what do we do once we find the multiples? We set the boolean value of that index to FALSE. Don't worry if this sounds confusing right now ( I'm not very good at explaining ) but this example will clear up everything - I promise!

Example: We start with 2nd index of the array (everytime). Its value in the array is TRUE. Hence we proceed to calculate its multiples i.e. 4, 6, 8, 10, 12, ... in the array and set them to false. Then we get to 3. Again, the value in the 3rd index is TRUE and so we proceed in a similar manner setting all the multiple of 3 i.e. 9, 12, 15, 18, ... to false. Now comes 4. But remember the 4th index is already set to false since it was one of the multiples of 2. So we do not need to find its multiples. We keep proceeding in this manner until the end.

Notice that I didn't include 6 in the multiples of 3. It so happens that, for a number n we can start counting its multiple from n^2. Example: for n = 5, we do not need to look for 10, 15, 20. These numbers are already cut out by previous numbers and excluding them will save a tiny bit of processing time.

Step 4: Now we have an array with mixed values of True and False. True indicates that the number is a prime and false indicates the number isn't prime. We loop through the array and add the index value whose value is set to True. And finally we will have the sum of all required Primes.

// Eratosthenes algorithm to find all primes under n
const eratosthenes = n => {
  let array = [];
  let output = [];
  let upperLimit = Math.sqrt(n);

  // We need to make an array from 2 to (n - 1)
  // For easy indexing we add extra two elements 0, 1 hence the length = n
  for (let i = 0; i < n; i++) {
    array.push(true);
  }

  // Remove multiples of primes starting from 2, 3, 5,...
  for (let i = 2; i <= upperLimit; i++) {
    if (array[i]) {
      for (let j = i * i; j < n; j += i) {
        array[j] = false;
      }
    }
  }

  // All array[i] set to true are primes
  sum = 0;
  for (let i = 2; i < n; i++) {
    if (array[i]) {
      sum += i;
    }
  }

  return sum;
};

Downside of Sieve's Algorithm

You must have probably guessed the problem with this algorithm. Yes, it's the memory consumption! For N = 1million we need an array of size 1million! In Node for large values of N, we see the classic javascript error - " heap out of memory ". But Node can easily handle upto N = 50 million.

<--- JS stacktrace --->

==== JS stack trace =========================================

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

If you've read this far I hope you enjoyed it :)

The Code behind @KuNotifier - A Twitter bot for Kathmandu University

Tue, 07 Nov 2017 00:00:00 GMT

The last time Kathmandu University updated its official website's layout was almost around 5 years ago. The old-fashioned website doesn't facilitate email subscription due to which students tend to miss out on important news and notices.

This time of year is a start to a new semester and every students (not the freshmans) in the college are waiting for their end semester results. The only way they can get notified about their results is by visiting the website or through friends. I have seen people who refresh the exam section page atleast a couple of dozen times a day! This is a very inconvinient and inefficient way since it takes time and if you're on mobile - an unnecessary data cost.

As a programmer I couldn't bear this and so I decided to build a bot that would notify me on every new notice published on the site.

Disclaimer: I didn't intend to make this bot for the public but soley for my own purpose. I am not responsible if the bot fails to deliver any news.

Deciding on how to get notified

I use a gmail chrome extension that notifies me instantly whenever I get new emails and since I am always on my browser I decided that the bot notifies me through email. This didn't go as planned due to Google's security system. It was because the bot was running on a server in United states which raised suspicion everytime the bot tried to connect with gmail's smtp server. This wasn't a huge issue to solve but I didn't bother to look for the solution.

So the obvious next way to notify was through social medias - Twitter and Facebook. In this way, not only me, but any student can take the benefit of the bot.

The bot is currently running on Twitter @KuNotifier and on Facebok @botlimbu. I have been running the Facebook Messenger bot for few months for learning purposes. It doesn't make facebook posts but message users directly who are subscribed to it. To subscribe, go to https://m.me/botlimbu and on the persistence menu click on the "Subscribe" option.

Summarizing the process

KU fortunately provides RSS feeds for its /exam and /news subdomains. The bot checks these two RSS feed every 20 minutes.
The result from the RSS is checked against a firebase database. If any new notice is found, tweet it and update the datbase

This is basically what the bot does ...

Code

The bot is written in JavaScript and runs on a NodeJS environment.

The bot bascially has two main functions. One to get the RSS feed and the other to check for new notices in the feed.

1. Reading the RSS Feed

In order to read and parse the RSS feeds, I used the NPM module called - "rss-parser". The function that gets the rss feed is named getResult. I prefer using ES6 and hence you'll find the arrow functions.

const parser = require('rss-parser');
const examFeed = 'http://www.ku.edu.np/exam/?feed=rss2';
const newsFeed = 'http://www.ku.edu.np/news/rss.php?blogId=1&profile=rss20';

const getResult = callback => {
  parser.parseURL(examFeed, (err, examData) => {
    parser.parseURL(newsFeed, (err, newsData) => {
      let items = examData.feed.entries.concat(newsData.feed.entries);
    });
  });
};

The data we get back is an object with all the feeds in an array in its feed.entries property. The two arrays from the exam and news feed is concatenated using the concat function in JavaScript. The concatenated array is stored in a variable named "items". Now the elements in these array must be sorted according to the date from latest to oldest.

// Sorting the array based on isoDate of each element
items = items.sort((a, b) => new Date(b.isoDate) - new Date(a.isoDate));

Great! Now just to keep things pretty I have also modified the pubDate format and added a new category property to each element of the array. This is the final code for our getResult function.

const getResult = callback => {
  parser.parseURL(examFeed, (err, examData) => {
    parser.parseURL(newsFeed, (err, newsData) => {
      let items = examData.feed.entries.concat(newsData.feed.entries);

      items = items
        .sort((a, b) => new Date(b.isoDate) - new Date(a.isoDate))
        .map(item => {
          item.pubDate = item.pubDate
            .split(' ')
            .slice(0, 4)
            .join(' ');
          return item;
        })
        .map(item => {
          item.category = item.link
            .split('')
            .splice(21, 4)
            .join('')
            .toUpperCase();
          return item;
        })
        .slice(0, 7);

      callback(items, null);
    });
  });
};

2. Checking against the database to look for new notices

Next I have a checkResult function that fetches data from firebase database, and compares it with the latest data fetched by getResult function. If theres any new notice, this function triggers a notification.

Let's break it down.

const checkResult = () => {
  console.log('Checking New Notice');

  getResult((res, err) => {
    if (err) return console.log(err);

    db.once('value', notice => {
      // Two Arrays, one from RSS and other from Database
      let DB_Results = objToArray(notice);
      let feed_Results = res.map(el => el.title);
    });
  });
};

The above function uses the getResult function to get latest feed from the RSS source. After that, it fetches data from the firebase database using the db.on method. The data received from the datbase is stored in a notice variable. This data is an object. To convert this object into array I create a new function objToArray.

function objToArray(obj) {
  let arr = [];
  for (var key in obj.val()) {
    arr.push(obj.val()[key]);
  }
  return arr;
}

Using this objToArray function I convert the notice ojbect to arrays;

// Two Arrays, one from RSS and other from Database
let DB_Results = objToArray(notice);

To make things simpler I created a new array fed_Results to store only the title of each post.

let feed_Results = res.map(el => el.title);

Great! Now we have two arrays that are identical in structure. Because of this we can easily compare them and see if any element of the array (post) differs. If there is any difference then we trigger a notification for that particular post.

Here's the full code for the checkResult function.

const checkResult = () => {
  console.log('Checking New Notice');

  getResult((res, err) => {
    if (err) return console.log(err);

    db.once('value', notice => {
      // Two Arrays, one from RSS and other from Database
      let DB_Results = objToArray(notice);
      let feed_Results = res.map(el => el.title);

      // Compare Those Two Arrays
      if (JSON.stringify(DB_Results) !== JSON.stringify(feed_Results)) {
        // Find all New Indices that don't match
        res.forEach((elm, index) => {
          if (DB_Results.indexOf(elm.title) === -1) {
            // Tweet the new indices
            shortenUrl(elm.link).then(url => {
              let status = statusMaker(elm, url);
              makeTweet(Bot, status);
              sendFBMsg(status);
            });
          }
        });

        // Update Database
        clearAndStore(feed_Results);
      }
    });
  });
};

This is it! Now I use a setTimeout method to run the checkResult function every 20 minutes

setInterval(checkResult, 1000 * 60 * 20);

Source Code

Read the full source code here

Aditya Thebe Personal Blog RSS Feed

Bypassing DRM protection to download a movie

🕰️ Now

Go v1.21+ Auto-Downloads Toolchains and Verifies Them Using a Transparency Log

Who Audits Go’s Transparency Log?

Verifying Reproducibility vs Consistency

Why is Golang's atomic.Bool 32 bits (not 8 bits)

Backup your GitHub repositories now!

Backup with gickup

Setup

Restoration

Alternative Tools

Delete all files in a Cloudflare R2 bucket using rclone

TLDR

Get the Access key

Setup rclone

Emptying the bucket

Fix invalid version: unknown revision from go get

The Solution

References

Git Objects

Git Command Categories

Git Objects

Commit

Tree:

Blob:

Tags:

References

Running AI agents in parallel

Coding agents roles and commands

Roles

Reviewing PR comments

DoltgresSQL

Containers on MacOS !!

Notifications on Claude Code

Moving to Claude Max Plan

Trying out Codex CLI

ARP - The DNS for IP addresses

Caching

ARP inside the Linux kernel

Demo

Monitor live ARP message

Things we didn't cover

References

IP Routing in Linux

Network Interface

Virtual interfaces

Analogy to understand network interfaces

Multiple interfaces

Routing decision

ip tool

View all interfaces on your system

View IP addresses of all network interfaces

IP destination types

A. Route

ip route

Default Gateway

Query the route for a destination

Adding a new route

B. Routing Table

local routing table

main routing table

default routing table

C. Rule

Routing Policy Database (RPDB)

Things we didn't cover

References:

Golang Footgun: Why is this interface not nil?

Explanation

How to Check the Underlying Value for Nil

Type assertion

Reflection

Git Hygiene: How to Automatically Delete Stale Local Branches

Primer on Git branches

Delete remote branches locally

List out all the stale branches

Setup an alias in Git config

Why I switched from Arch Linux to MacOS

Pain of using a desktop machine

Don't think with your fingers

`ip` tool

`ip route`

What's new in `SQL:2023`

Create `launch.json` file