Skip to main content

Saturday, April 6, 2024

· One min read

Globally ignore .DS_Store from all the git repos in macOS

# remove any existing files from the repo, skipping over ones not in repo
find . -name .DS_Store -print0 | xargs -0 git rm --ignore-unmatch
# specify a global exclusion list
git config --global core.excludesfile ~/.gitignore
# adding .DS_Store to that list
echo .DS_Store >> ~/.gitignore

macos - How can I Remove .DS_Store files from a Git repository? - Stack Overflow

List Git remote branches

git branch -r

Friday, April 5, 2024

· 3 min read

What is the relation between filesystem, volume, partition and disk? What are they?

Filesystem, volume, partition, and disk are fundamental components within the realm of computer storage, particularly in the context of hard disk drives (HDDs) and solid-state drives (SSDs). Here's an explanation of each and their relationship:

  1. Disk: A disk, in the context of computing, refers to the physical storage device, such as a hard disk drive (HDD) or a solid-state drive (SSD). It consists of one or more platters (in HDDs) or memory chips (in SSDs) on which data is stored.

  2. Partition: A partition is a logical division of a physical disk. It involves dividing the disk into separate sections that the operating system treats as independent units. Each partition can be formatted with a particular filesystem and can contain its own set of files and directories. Partitions are useful for various purposes, such as organizing data, separating operating system files from user files, or hosting multiple operating systems on a single disk.

  3. Volume: A volume is a logical storage unit that can span one or more partitions. It represents a formatted section of a disk that is accessible to the operating system and applications as a single entity. Volumes are typically associated with a specific filesystem, which determines how data is organized and stored on the volume. In simpler terms, a volume is what you see and interact with when you access a storage device (e.g., drive letters like C:, D:, etc., in Windows, or mount points in Unix-like systems).

  4. Filesystem: A filesystem is a method of organizing and storing files and directories on a storage device. It provides a structure for accessing, reading, and writing data stored on the disk. Filesystems define how data is stored, named, and accessed, including aspects such as directory structure, file naming conventions, and metadata storage. Examples of filesystems include NTFS (used in Windows), ext4 (used in many Linux distributions), and APFS (used in macOS).

Relationship:

  • A disk is the physical storage medium.
  • A disk can be divided into one or more partitions.
  • Each partition can be formatted with a filesystem.
  • Each formatted partition represents a volume, which is the accessible storage unit seen by the operating system and users.

In summary, partitions divide disks into manageable sections, volumes represent accessible storage units within those partitions, and filesystems dictate how data is organized and stored within volumes. Together, they form the foundation of storage management in computer systems.

What is the distinction between "drive" and "disk"?

  1. Drive: Refers to the entire storage unit, including both the physical disk (in the case of HDDs) or memory chips (in the case of SSDs), as well as any associated hardware components responsible for reading from and writing to the storage medium. This includes not only the memory or storage components themselves but also components such as controllers, interfaces, and connectors.

  2. Disk: Often refers specifically to the physical storage medium itself, whether it's the spinning platters inside a hard disk drive (HDD) or the memory chips inside a solid-state drive (SSD). It typically refers to the component where data is stored magnetically (in HDDs) or electronically (in SSDs), excluding other components such as controllers or interfaces.

In common usage, the terms Drive and Disk are often used interchangeable.

Monday, March 25, 2024

· 2 min read

Solid State Drive (SSD) Form Factors

SSD components:

  • Controller chips
  • Memory chips
    • NAND Flash particles
      • SLC (1 bit/cell)
      • MLC (2 bits/cell)
      • TLC (3 bits/cell)
      • QLC (4 bits/cell)
    • Manufactures
      • Samsung
      • SanDisk
      • Micron
      • SK Hynix
      • Toshiba
  • Interfaces
    • SATA (Serial Advanced Technology Attachment)
      • Size
        • M.2
      • Protocols
        • AHCI
    • PCIe (PCIe 5.0, 4.0, 3.0)
      • Size
        • M.2
      • Protocols
        • AHCI (rarely!)
        • NVMe

NOTE Data Transfer Protocols:

  • AHCI
  • NVMe

NOTE M.2 is a size format: about the size and shape of a stick of gum.

Popular SSD: PCIe 4.0/M.2/NVMe over 7000MB/s

Portal SSD:

  • SSD (mentioned above)
  • Protective case
    • Durable
    • Waterproofing
    • Drop resistance
    • Heat dissipation
    • Compact and portable
  • Bridge chips
  • USB-C interface (external) to PCIe/M.2/NVMe interface (internal)
  • USB-C interface (external) to SATA/M.2/AHCI interface (internal)

NVMe vs M.2: Bus, Interface, and Protocol - MiniTool Partition Wizard

3 SSD Terminologies You Need to Know when Buying SSD Drive - MiniTool Partition Wizard

USB Flash Drive

Both USB Flash Drive and SSD use NAND flash memory for data storage. The main difference is from other form factors, such as SSD own more advanced and complicate controller chips to read/write data.

Friday, August 4, 2023

· One min read
  • Authentication and Authorization in Microservices
  1. Authentication and Authorization in each service
  2. Authentication in a centralized service, and Authorization in each service
  3. Authentication and Authorization in a centralized service
  • Auth Service and User (Profile) Service

Never write a UserService again. Or when to use external Microservices

Microservices Authentication Best Strategy | Aspecto

Authentication and Authorization Concepts for MicroServices · GitHub

design - Microservice Architecture - using Auth Server as a User Resource server

How to Run Your Own Decentralized Authentication Service Using AuthN

  • Implement event-driven architecture microservices using Redis

Using Redis as an Event Store for Communication Between Microservices

Tuesday, July 25, 2023

· 2 min read
  • Authentication and Authorization in Microservices

Authentication in microservices involves two main occasions:

  1. authentication required when end users communicate with services.
  2. authentication happens between internal services.
  3. authentication needed when external services enter internal services.

OAuth 2.0 provides the industry-standard protocol for authorizing users in distributed systems. The OAuth framework reduces the burden on developers, eliminating duplications to build their own authentication mechanism in each microservice.

Authentication & Authorization in Microservices Architecture - Part I

https://softwareengineering.stackexchange.com/questions/366815/microservice-architecture-using-auth-server-as-a-user-resource-server

https://auth0.com/docs/get-started

  • User registration flow in microservice

  • Communication between microservices

Share user data between micro services

User service and Comment service

populate user data into Comment service, save user data in comment service, update user data in comment service

https://stackoverflow.com/questions/67543408/microservices-storing-user-data-in-separate-database

Ideally, the client communicates with the each service directly, and no interaction between the services!

However, there is the need for communication between these services.

For example, o what happens if a user deletes his account? What if you delete a TV show? You probably want to trigger some events that will update the data in your comment microservice. In the long run you want to keep everything "eventually consistent".

The event-driven architecture comes up!

  • Data retrieved from two or more services

For example, you send a request from UI saying "give me comments with usernames",

GraphQL interface then first issues a request to comments service, then to user service and finally sends one response with combined data

NOTE: issue a number of requests to various micro-services to collect all the data and return it in only 1 response

Rest needs to send many.

https://softwareengineering.stackexchange.com/questions/418153/design-a-correct-microservices-architecture-with-data-relations

Event-Driven Data Management for Microservices - NGINX

Friday, July 21, 2023

· One min read
  • Mysql, redis, or other db connections pool vs as single connection in Nodejs

Since Node.js and Redis are both effectively single threaded there is no need to use multiple client instances or any pooling mechanism save for a few exceptions;

Thursday, July 20, 2023

· 2 min read
  • HapiJS

Hapi.js — Project Structure and Best Practices (Part 2)

Optimizing HapiJS for Benchmarks. In the past year or so, our team… | by Joel Chen | Walmart Global Tech Blog | Medium

  • The confused saying in Microservices: "each service should own its own database and no two services should share a database"

No golden rule, no fast rules, no best practices suitable for all businesses, only tradeoff

Q: Need separate database per service? A: Creating a separate database for each service helps to enforce domain boundaries.

The Hardest Part About Microservices: Your Data – Software Blog

  • Nodejs development practices

Set default configs: author name, author email, author url, the license, and the version.

npm set init.author.name "Your name"
npm set init.author.email "your@email.com"
npm set init.author.url "https://your-url.com"
npm set init.license "MIT"
npm set init.version "1.0.0"
function node-project {
git init
npx license $(npm get init.license) -o "$(npm get init.author.name)" > LICENSE
npx gitignore node
npx covgen "$(npm get init.author.email)"
npm init -y
git add -A
git commit -m "Initial commit"
}

Setting up efficient workflows with ESLint, Prettier and TypeScript in vscode.

Setting up efficient workflows with ESLint, Prettier and TypeScript - JavaScript inDepth

Wednesday, July 19, 2023

· 2 min read
  • I still prefer os.path over Pathlib, as follows
  1. Consistency, I'm used to use path string as an argument between functions and I think Pathlib is not flexible enough to handle arguments
  2. Pure and Function, Although Pathlib brings many useful features like glob, stem, and so on. I still like the concept of simplicity that don't put all things together!
  • Trim $ for clipboard copy in Docusaurus in code block bash.

Ignore $ for clipboard copy · Issue #1745 · facebook/docusaurus · GitHub

  • Some common issues I often hit when using git

Configure username/password for different repos or remotes

Global configuration

git config --global --list
git config --local --list

GIT two popular authentication methods:

  • ssh key

How to Authenticate Your Git to GitHub with SSH Keys

  • git credentials

Store username/password instead of ssh for multiple remotes

To enable git credentials

# local
git config credential.helper store
# global
git config --global credential.helper store

Each credential is stored in ~/.git-credentials file on its own line as a URL like:

https://<USERNAME>:<PASSWORD>@github.com

Configure credentials,

# Global
git config --global credential.https://github.com.username <your_username>

# Or
git config --local user.name <your_username>
git config --local user.email <your_useremail>
# Then git pull or git push to let it cache your username/password after it prompt you to input password in the first time

Alternatively, we can directly edit our global Git config file ~/.gitconfig,

[credential "https://github.com"]
username = <username>

Git - Config Username & Password - Store Credentials - ShellHacks

Configuring Git Credentials

  • Programming Algorithms

Top Algorithms Every Programmer Should Know

What is Algorithm | Introduction to Algorithms - GeeksforGeeks