# Introduction

[My knowledge wiki](https://wiki.omar.engineer). You can also find [a repository](https://github.com/obahareth/knowledge) mirroring this wiki with Markdown files on GitHub.

This is the beginning of a Gitbook containing everything I know, and it's heavily inspired from <https://github.com/nikitavoloboev/knowledge>.

You can search using the top right corner or you can simply explore the tree view to the left.

You can [ask me anything](https://github.com/obahareth/ama) you want at anytime.

## Other things I wrote and shared

I also share my knowledge by writing articles [on Medium](https://medium.com/@obahareth), on [dev.to](https://dev.to/obahareth), and my [personal website](https://omar.engineer).

I also love writing code that solves various problems I have and I share it all [on GitHub](https://github.com/obahareth).

I share a number of pages, boards, and lists on my [public Notion page](https://www.notion.so/obahareth/Public-04ad2eb582a448b1ae834249d5ada9b9).

## Make your own wiki

You can view other similar to this, continuously updated wikis, [here](https://github.com/RichardLitt/meta-knowledge#readme).

Don't be afraid to create one of your own and share what you know with the world.

If you found a mistake anywhere in this wiki, I would appreciate your help. You can quickly find any entry you wish to edit by [searching for the topic](https://github.com/obahareth/knowledge/find/master) and then making the changes.

I also appreciate any [ideas you have](https://github.com/obahareth/knowledge/issues/new) on how I can improve this wiki.


# My Stack

I currently use a 2020 13" Macbook Pro with an M1 Chip and my phone's a iPhone 12 Pro Max.&#x20;

I also share my [Mac setup](https://github.com/obahareth/my-mac-os), (old) [Linux setup](https://github.com/obahareth/my-linux) and my (old)[ Android setup](https://github.com/obahareth/my-android) on GitHub.

The tools and languages I've used to solve problems across my career as a programmer are [available on StackShare](https://stackshare.io/obahareth/obahareth).

{% embed url="<https://stackshare.io/obahareth/obahareth>" %}


# Sharing

## **Code on** [**GitHub**](https://github.com/obahareth)

I love solving problems with code. [Here](https://github.com/obahareth?tab=repositories\&type=source) are all the repositories I made.

## **Thoughts on** [**Twitter**](https://twitter.com/o_bahareth)

I share a lot of cool tech posts I come by, sometimes about random life tidbits, and I tweet in both Arabic and English.

## **Blog on** [**Medium**](https://medium.com/@obahareth)

I write mostly about programming, tools, and management.

## **Experiences on** [**Instagram**](https://www.instagram.com/obahareth/)

I mostly share cat pics or pics from my travels.

## **Ideas in** [**Notion**](https://www.notion.so/obahareth/Public-04ad2eb582a448b1ae834249d5ada9b9)

My time is limited and I can't often do everything I wish. But I have many ideas and lists so I put all of them [in the open](https://www.notion.so/03b7e9880c26428596228dfb14ee6e47).

## Snippets on [GitHub Gists](https://gist.github.com/obahareth)​ <a href="#snippets-on-github-gists" id="snippets-on-github-gists"></a>

I things I'm tinkering with, code that I want to embed in articles, and configurations for various tools on GitHub Gist.

## **Knowledge in** [**this wiki**](https://wiki.omar.engineer/)

I am trying to fill everything I know in this wiki. It can be easily queried with the search at the top right.

## **Everything else**

I share most everything else here:

* [Reddit](https://www.reddit.com/user/obahareth/)
* [Hacker News](https://news.ycombinator.com/user?id=obahareth)
* [Stack Overflow](https://stackoverflow.com/users/1544440/omar-bahareth)
* [Quora](https://www.quora.com/profile/Omar-Bahareth)
* [Product Hunt](https://www.producthunt.com/@o_bahareth)
* [Indie Hackers](https://www.indiehackers.com/user/obahareth)


# Work


# How I Want to Work

Being able to work at a place or project that has all of these things is a rare opportunity, and if I get it I would definitely give everything I have to it.

## I want to do work I'm proud of

I've often had to do work that I believed had bad performance, believed will cause bugs, and is just troublesome for other developers to deal with due to time constraints. I don't want to do that in my career anymore, I want to add value to all people who interact with my work, whether it's users or developers, and reaching a point where I've been doing this for seven years, it just hurts and it has to stop. I know I can do awesome stuff but I am held back, and I no longer want to be. I know that sometimes you have to sacrifice these things to reach a certain goal, but it's always been the sacrifice without the goal so far.

I want to make software that'll make both users and developers (performance, structure, readability, you name it) smile.

## I want to create something meaningful

Throughout my career I've built too much software for clients where almost no users use the software, and those that do suffer from lack of maintenance; and I've also built too much software that has never seen the light of day. I want to build something that at least makes someone smile or saves some time in their day. I am extremely inspired by initiatives like[ IBM's Call for Code](https://developer.ibm.com/callforcode/), a chance to save people from natural disasters using what I know? That's like a dream come true!

## I want to grow as an engineer

I've been stuck doing the same kind of work over and over, building webapps and mobile apps that generally just do the same exact thing. I want to do something challenging, something new. I want to work on something that has to deal with scaling issues, I want to work on something like Docker or cloud services. I'm seeing a lot of cool stuff being done with Go, Rust, and Elixir lately and I want to be a part of those communities.

## I want to have a mentor

I've never really had a mentor in my career and I've generally had to always figure things out on my own or build up teams. I want someone to build me up.

## I want my time back, and I want to see the sun

Traditional work hours just drain my energy and leave me with a Vitamin-D deficiency, I honestly don't think that's healthy in any way imaginable, and it's unfair to all my loved ones. I want more time to give back to open source, I want to start a technical podcast, and do so much more. I would love to be in a workplace that enables me to do these things.

## I want to work remotely

I believe remote work is definitely the next extension of our digital careers. I have distantly watched GitLab grow with that mission and am immensely impressed by what they've done. One of the things they noticed that one of the very first things people do when they join GitLab is move back to be close to their friends and family. It feels morally wrong for me to have to put friends and family second to my career (both due to lack of time, and lack of jobs where I live).

## I need to be able to communicate with my colleagues

I would like to utilize the languages and skills I know to communicate with my colleagues. Most of my career has been in Saudi Arabia and so I have to be good with both Arabic and English. I'm pretty decent with written and spoken English, and I'm pretty decent with spoken Arabic too, but with written Arabic it's just hell for me because I believe we as Arabs still haven't decided how we want to use its written form, especially when discussing technical topics.

**On Arabic**

One example is that our language uses diacritics for vowels but we stopped using them in our written communication (probably because they make writing take so much longer) and it makes reading things involve a lot of guessing, and there are so many different dialects (without any grammar rules, any dictionary, or written resource you can look up), and that results in all sorts of communication issues and even bugs. I've often had to work with people who I struggle to communicate with because they dislike written communication (which makes things like pull request reviews quite difficult), and when they're ok with it, it's in an Arabic dialect that's different from mine. I would love to use [Moden Standard Arabic](https://www.wikiwand.com/en/Modern_Standard_Arabic) but many Arabs (including me) don't know it that well, and for some it feels too formal (some even tell me, it feels like you're giving a speech rather than having a casual conversation), although there are certain places where that form is the norm, and I really love and respect that. I don't really have a problem with Arabic, but I have a problem with how we choose to use it, how we treat our dialects as a "lesser form" that should go away and thus refuse to standardize it... Since we are both speaking and writing in these dialects, we should have references, proper spellings, and dictionaries for it. If I want to do something as simple as searching an issue tracking system for a bug that used to happen when users selected a specific city, I can't write the city's actual name, I have to try and write it the same dialect as the person who reported it (and I may not necessarily know that dialect, and that person may not be on our team anymore, and not everyone will even expect this scenario or know how to deal with it). This also of course makes things like Natural Language Processing incredibly more difficult, and I perhaps should move this topic to its own page in the future.


# Hiring Process

### Process

#### Detailed Job Posting

It's very important to have a detailed job posting, this will help candidates know what to expect of your workplace and help you to filter out unsuitable candidates as they will not even apply if the job doesn't suit them. A great example in detailed listings is Basecamp's listings, take a look at their [Head of Marketing](https://m.signalvnoise.com/basecamp-is-hiring-a-head-of-marketing/) example.

#### Simple Coding Challenge

*After some screening for communication alignment and HR screening*

This would be a simple coding challenge based on a real problem we face at our workplace; preferably solved in a language/framework we use but not mandatory, it should ideally be doable within 2 hours but I'd give you a whole week because sometimes we as humans get stuck on the silliest matters no matter how long we've been doing something, perhaps you'd like some time to learn the basics of the framework/language we use and try to solve it in it, and because you probably have other priorities.

#### Interview Process

Hiring is something that's hard to get right and I think most of the methodologies we currently use are quire flawed. We look too much at what someone has done in the past and almost never look at what they want to do now. I believe [Microsoft's new process](https://blog.usejournal.com/rethinking-how-we-interview-in-microsofts-developer-division-8f404cfd075a) is a huge step towards the right direction, give someone a chance to work with you using the same information you have. Solve a real problem together, do actual work. The only way to know if someone is a fit or not is to work with them and have all the different teams work with them.

I realize that's expensive and few places can afford that, but that's my ideal scenario, I've never gotten to do this before but I think it would yield much better results.

### Points to Consider

#### Degrees

Degrees are great, but I wouldn't discredit someone because they don't have one. The gap between what you learn at university and what the market needs is growing a lot, especially in developing countries. As I said above the only solution is to work together for a bit.

#### Communication

As I described in [How I Want to Work](https://wiki.omar.engineer/work/what-i-want-in-a-workplace), being able to communicate in the same language and in the same style is extremely important. I once tried to build a culture of asynchronous communication based primarily on written communication, but I hired people whose stronger communication points aren't aligned with that and it didn't work out so well. In future hiring I must make sure that our style of communication works well for both parties.

#### Clear Expectations

[HackerRank's 2019 Developer Skills Report](https://research.hackerrank.com/developer-skills/2019) shows that the number one reason to lose candidates is unclear roles, and I've often been put in that situation as well. You join in as a backend developer and you're suddenly asked to do everything (DevOps, sysadmin, dba, native iOS and Android development, Angular, React, Laravel, Kubernetes, etc.), this is all too common in developing countries where specialization is still rare. I've also sadly hired people with unclear expectations in the past and it's something I do not wish on anyone. I would do everything in my power to explain everything a certain role requires and make sure it remains so.

#### Compensation for Work Done During Process

I have been seeing a lot of people saying that candidates should be paid for the work they do during hiring processes, and I think that is a fair point, especially with the idea of letting you work with us on a real problem. If the place I'm hiring for can afford it, this would happen. If not, I would do my best to try and make it happen.


# Apps & Tools


# Raycast

Raycast is an awesome launcher with many extensions that make it go so much more beyond that. Extensions can present a variety of interfaces, it has an extension store, and it's so easy to develop extensions for.


# Extensions

Raycast extensions that I frequently use

## Bundled with Raycast

* Clipboard History
* Define Word
* File Search
* Search Emoji

## Available on Store

* [Change Case](https://www.raycast.com/es183923/change-case)
* [Linear](https://www.raycast.com/raycast/linear)
* [Ruby Evaluate](https://www.raycast.com/obahareth/ruby-evaluate)


# Visual Studio Code


# Theme

Color Theme:&#x20;

* [Polar](https://marketplace.visualstudio.com/items?itemName=merithayan.polar) (day, via Workbench: Preferred Light Color Theme).
* [Nord](https://marketplace.visualstudio.com/items?itemName=arcticicestudio.nord-visual-studio-code) (night, via Workbench: Preferred Dark Color Theme).
* I use Window: Auto Detect Color Scheme to auto switch themes based on time of day.

Icon Theme: [Material Icon Theme](https://marketplace.visualstudio.com/items?itemName=PKief.material-icon-theme).

Font: [Pragmata Pro](https://www.fsd.it/shop/fonts/pragmatapro/).

![](/files/-L_gw9u6h7ZvPKdQa6Xw)


# Extensions

### Editing

>

#### Syntax Highlighters / Programming Languages

* [Apache Conf](https://marketplace.visualstudio.com/items?itemName=mrmlnc.vscode-apache) - Syntax highlighter for Apache configuration files.
* [C#](https://marketplace.visualstudio.com/items?itemName=ms-vscode.csharp) - C# for Visual Studio Code (powered by OmniSharp).
* [Crystal](https://marketplace.visualstudio.com/items?itemName=g3ortega.crystal) - Provides Crystal language support for Visual Studio Code.
* [DotENV](https://marketplace.visualstudio.com/items?itemName=mikestead.dotenv) - Support for dotenv file syntax.
* [Go](https://marketplace.visualstudio.com/items?itemName=ms-vscode.go) - Rich Go language support for Visual Studio Code.
* [Handlebars](https://marketplace.visualstudio.com/items?itemName=andrejunges.handlebars) - Handlebars Visual Studio Code.
* [Mustache](https://marketplace.visualstudio.com/items?itemName=dawhite.mustache) - Syntax highlighting for Mustache.
* [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python) - Linting, Debugging (multi-threaded, remote), Intellisense, code formatting, refactoring, unit tests, snippets, and more.
* [Ruby](https://marketplace.visualstudio.com/items?itemName=rebornix.ruby) - Provides Ruby language and debugging support for Visual Studio Code.
* [Rust](https://marketplace.visualstudio.com/items?itemName=kalitaalexey.vscode-rust) - Rust language integration for VSCode.
* [vscode-elixir](https://marketplace.visualstudio.com/items?itemName=mjmcloug.vscode-elixir) - Elixir support for VSCode.
* [YAML](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml) - Great YAML support, including Kubernetes syntax, and lets you find out where you exactly are in the YAML if you have [breadcrumbs](https://code.visualstudio.com/updates/v1_26#_breadcrumbs) enabled.

#### General Editing

* [Babel ES6/ES7](https://marketplace.visualstudio.com/items?itemName=dzannotti.vscode-babel-coloring) - Adds JS Babel es6/es7 syntax coloring.
* [Beautify](https://marketplace.visualstudio.com/items?itemName=HookyQR.beautify) - Beautify code in place for VS Code.
* [Better Align](https://marketplace.visualstudio.com/items?itemName=wwm.better-align) - Align code without selecting them first.
* [Bracket Pair Colorizer 2](https://marketplace.visualstudio.com/items?itemName=CoenraadS.bracket-pair-colorizer-2) - A customizable extension for colorizing matching brackets.
* [Code Spell Checker](https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker) - Spelling checker for source code.
* [indent-rainbow](https://marketplace.visualstudio.com/items?itemName=oderwat.indent-rainbow) - Makes indentation easier to read.
* [Vim](https://marketplace.visualstudio.com/items?itemName=vscodevim.vim) - Vim emulation for Visual Studio Code.

### Collaboration

* [VS Live Share](https://marketplace.visualstudio.com/items?itemName=MS-vsliveshare.vsliveshare) - Real-time collaborative development from the comfort of your favorite tools.
* [GitHub Pull Requests](https://marketplace.visualstudio.com/items?itemName=GitHub.vscode-pull-request-github) - Review and manage GitHub pull requests in Visual Studio Code.

### Git

* [Git History](https://marketplace.visualstudio.com/items?itemName=donjayamanne.githistory) - View git log, file history, compare branches or commits.
* [GitLens](https://marketplace.visualstudio.com/items?itemName=eamodio.gitlens) - Supercharge the Git capabilities built into Visual Studio Code.

### Devops

* [Docker](https://marketplace.visualstudio.com/items?itemName=peterjausovec.vscode-docker) - Adds syntax highlighting, commands, hover tips, and linting for Dockerfile and docker-compose files.

### Debugging

* [Debugger for Chrome](https://marketplace.visualstudio.com/items?itemName=msjsdiag.debugger-for-chrome) - Debug your JavaScript code in the Chrome browser, or any other target that supports the Chrome Debugger protocol.
* [Debugger for Unity](https://marketplace.visualstudio.com/items?itemName=unity.unity-debug) - Unity debugger extension.
* [PHP Debug](https://marketplace.visualstudio.com/items?itemName=felixfbecker.php-debug) - Debug support for PHP with XDebug.

### Snippets

* [Ruby on Rails](https://marketplace.visualstudio.com/items?itemName=hridoy.rails-snippets).
* [ES7 React/Redux/GraphQL/React-Native snippets](https://marketplace.visualstudio.com/items?itemName=dsznajder.es7-react-js-snippets).

### Utility

* [Browser Preview](https://github.com/auchenberg/vscode-browser-preview) - A real browser preview inside your editor that you can debug.
* [advanced-new-file](https://marketplace.visualstudio.com/items?itemName=patbenatar.advanced-new-file) - adds the ability to create files anywhere in your workspace.
* [Code Runner](https://marketplace.visualstudio.com/items?itemName=formulahendry.code-runner) - Run code in VS Code.
* [EditorConfig for VS Code](https://marketplace.visualstudio.com/items?itemName=editorconfig.editorconfig) - EditorConfig Support for Visual Studio Code.
* [ESLint](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint) - Integrates ESLint JavaScript into VS Code.
* [GhostText](https://marketplace.visualstudio.com/items?itemName=tokoph.ghosttext) - Use VS Code to write in your browser.
* [hexdump for VSCode](https://marketplace.visualstudio.com/items?itemName=slevesque.vscode-hexdump) - Display a specified file in hexadecimal.
* [Insert Numbers](https://marketplace.visualstudio.com/items?itemName=asuka.insertnumbers) - Insert increasing numbers.
* [IntelliSense for CSS class names](https://marketplace.visualstudio.com/items?itemName=Zignd.html-css-class-completion) - CSS class name completion for the HTML class attribute based on the definitions found in your workspace.
* [Live Server](https://marketplace.visualstudio.com/items?itemName=ritwickdey.liveserver) - Launch a development local Server with live reload feature for static & dynamic pages.
* [Path Intellisense](https://marketplace.visualstudio.com/items?itemName=christian-kohler.path-intellisense) - Filename autocompletion.
* [phpcs](https://marketplace.visualstudio.com/items?itemName=ikappas.phpcs) - PHP CodeSniffer for Visual Studio Code.
* [Quit Control for VSCode](https://marketplace.visualstudio.com/items?itemName=artdiniz.quitcontrol-vscode) - Stop mistyping keyboard shortctus that close/quit VSCode unintentionally.
* [React Native Tools](https://marketplace.visualstudio.com/items?itemName=vsmobile.vscode-react-native) - Code-hinting, debugging and integrated commands for React Native.
* [Ruby Solargraph](https://marketplace.visualstudio.com/items?itemName=castwide.solargraph) - Code completion and inline documentation for Ruby.
* [Settings Sync](https://marketplace.visualstudio.com/items?itemName=shan.code-settings-sync) - Synchronize Settings, Snippets, Themes, File Icons, Launch, Keybindings, Workspaces and Extensions Across Multiple Machines Using GitHub Gist.
* [Sort lines](https://marketplace.visualstudio.com/items?itemName=tyriar.sort-lines) - Sort lines of text.
* [TODO Highlight](https://marketplace.visualstudio.com/items?itemName=wayou.vscode-todo-highlight) - Highlight TODOs, FIXMEs, and any keywords, annotations.
* [WakaTime](https://marketplace.visualstudio.com/items?itemName=wakatime.vscode-wakatime) - Metrics, insights, and time tracking automatically generated from your programming activity.


# Google Chrome


# Theme

### [Material Nord](https://chrome.google.com/webstore/detail/material-nord/cnfjnjfppmpabbbdeijhimfijipmmanj?hl=en)

![The dark background for the webpage is from the Dark Reader extension](/files/-LWujWAJMCUrGNjwCmMI)


# Extensions

These are the Google Chrome extensions I use (RIP my RAM).

### General

* [1Password X - Password Manager](https://chrome.google.com/webstore/detail/1password-x-%E2%80%93-password-ma/aeblfdkhhhdcdjpifhhbdiojplfjncoa).
* [Buffer](https://chrome.google.com/webstore/detail/buffer/noojglkidnpfjbincgijbaiedldjfbhh).
* [Midnight Lizard](https://chrome.google.com/webstore/detail/midnight-lizard/pbnndmlekkboofhnbonilimejonapojg) with [Nord Scheme](https://midnight-lizard.org/schemes/index/full/641417e1-5773-4e95-b707-75358ed5286c) to make all websites use the Nord theme.
* [Enhanced Steam](https://chrome.google.com/webstore/detail/enhanced-steam/okadibdjfemgnhjiembecghcbfknbfhg).
* [GhostText](https://chrome.google.com/webstore/detail/ghosttext/godiecgffnchndlihlpaajjcplehddca).
* [Hunter](https://chrome.google.com/webstore/detail/hunter/hgmhmanijnjhaffoampdlllchpolkdnj).
* [Markdown Here](https://chrome.google.com/webstore/detail/markdown-here/elifhakcjgalahccnjkneoccemfahfoa).
* [Notion Web Clipper](https://chrome.google.com/webstore/detail/notion-web-clipper/knheggckgoiihginacbkhaalnibhilkk?hl=en).
* [OneTab](https://chrome.google.com/webstore/detail/onetab/chphlpgkkbolifaimnlloiipkdnihall).
* [Refined Twitter](https://chrome.google.com/webstore/detail/refined-twitter/nlfgmdembofgodcemomfeimamihoknip).
* [Save to Pocket](https://chrome.google.com/webstore/detail/save-to-pocket/niloccemoadcdkdjlinkgdfekeahmflj).
* [Reddit Enhancement Suite](https://chrome.google.com/webstore/detail/reddit-enhancement-suite/kbmfpngjjgdllneeigpgjifpgocmfgmb).
* [Shodan](https://chrome.google.com/webstore/detail/shodan/jjalcfnidlmpjhdfepjhjbhnhkbgleap).
* [Tampermonkey](https://chrome.google.com/webstore/detail/tampermonkey/dhdgffkkebhmkfjojejmpbldmpobfkfo).
* [The Great Suspender](https://chrome.google.com/webstore/detail/the-great-suspender/klbibkeccnjlkjkiokjodocebajanakg).
* [Todoist](https://chrome.google.com/webstore/detail/todoist-to-do-list-and-ta/jldhpllghnbhlbpcmnajkpdmadaolakh).
* [uBlock Origin](https://lh3.googleusercontent.com/gE-W0qCedE7UhXvN_VvyUn1axwmEV5fc2XybuKY3OGyvB54Ci6IMBGLAemQUkejaB2bDxVrmWQ=w440-h280-e365).
* [Vimium](https://chrome.google.com/webstore/detail/vimium/dbepggeogbaibhgnhhndojpepiihcmeb).
* [Wappalyzer](https://chrome.google.com/webstore/detail/wappalyzer/gppongmhjkpfnbhagpmjfkannfbllamg).
* [WhatRuns](https://chrome.google.com/webstore/detail/whatruns/cmkdbmfndkfgebldhnkbfhlneefdaaip).
* [Wikiwand](https://chrome.google.com/webstore/detail/wikiwand-wikipedia-modern/emffkefkbkpkgpdeeooapgaicgmcbolj).

### Github Enhancements

* [Awesome Autocomplete for GitHub](https://chrome.google.com/webstore/detail/awesome-autocomplete-for/djkfdjpoelphhdclfjhnffmnlnoknfnd).
* [Isometric Contributions](https://chrome.google.com/webstore/detail/isometric-contributions/mjoedlfflcchnleknnceiplgaeoegien).
* [OctoLinker](https://chrome.google.com/webstore/detail/octolinker/jlmafbaeoofdegohdhinkhilhclaklkp).
* [Refined GitHub](https://chrome.google.com/webstore/detail/refined-github/hlepfoohegkhhmjieoechaddaejaokhf).
* [Notifier for GitHub](https://github.com/sindresorhus/notifier-for-github).

### Development

* [EditThisCookie](https://chrome.google.com/webstore/detail/editthiscookie/fngmhnnpilhplaeedifhccceomclgfbg?utm_source=inline-install-disabled).
* [RailsPanel](https://chrome.google.com/webstore/detail/railspanel/gjpfobpafnhjhbajcjgccbbdofdckggg).
* [React Developer Tools](https://chrome.google.com/webstore/detail/react-developer-tools/fmkadmapgofadopljbjfkapdkoienihi).
* [Redux DevTools](https://chrome.google.com/webstore/detail/redux-devtools/lmhkpmbekcpmknklioeibfkpmmfibljd).


# iTerm 2


# Theme

I use [Nord](https://github.com/arcticicestudio/nord-iterm2) as my Hyper theme, along with the [starship prompt](https://github.com/starship/starship).

![](/files/-Mh-PU4VDkxeo_I8zAAK)


# Fish


# Plugins

I don't have a lot of plugins because Fish is awesome and comes with great defaults.

I use [Fisher](https://github.com/jorgebucaran/fisher) to manage [my plugins](https://github.com/obahareth/dotfiles).

### Utility <a href="#utility" id="utility"></a>

* ​[autojump](https://github.com/wting/autojump) - A cd command that learns - easily navigate directories from the command line.
* [asdf](https://github.com/asdf-vm/asdf) - One version manager for all my programming languages (no need to have rvm/nvm/etc.).
* [bass](https://github.com/edc/bass) - Make Bash utilities usable in Fish shell.

### Prompt <a href="#prompt" id="prompt"></a>

* [Starship](https://github.com/starship/starship) - Pretty, minimal and fast cross-shell prompt.


# Docker


# Volumes

### Table of Contents

* [Mounting](/apps-and-tools/docker/volumes-1/volumes).


# Mounting

### Mounting

Mounting Docker volumes via Docker for Mac can result in performance bottlenecks as seen in [this issue](https://github.com/docker/for-mac/issues/77). This also heavily impacts the performance of my [Elixir base16 builder](https://github.com/obahareth/base16-builder-elixir/issues/2).

On macOS, you can [configure mount consistency](https://docs.docker.com/storage/bind-mounts/#configure-mount-consistency-for-macos) to get better performance on your use case. These are the three available consistencies:

* `consistent` (the default) - This is the slowest setting and it aims for full consistency between the container and host
* `delegated` - This setting treats the container view as the source of truth, changes in the container can take a while to appear on the host. This setting is better used when the container is making the majority of the changes (especially if they're a lot). In my Elixir base16 builder, this is the setting I chose because A LOT of files are written in the container that should then be reflected back to the host, and I don't really care about the host's version of the files.
* `cached` - This setting treats the host as the source of truth, changes in the host may take a while to appear in the container. This setting is better used for development where you always want the latest changes on your host to show up on the container, and you aren't really going to make changes inside of the container only.


# Web Bundlers


# Webpack

### Tips

#### Be very careful with importing large libraries

Trying to compile large libraries (like react-plotly or PDF libraries) can take your Webpack compile from seconds to 10 minutes+. If a package is slowing down your compile, consider using a CDN version. We simply used script tags, but there are Webpack plugins that can help with that too:

* [webpack-cdn-plugin](https://www.npmjs.com/package/webpack-cdn-plugin).
* [dynamic-cdn-webpack-plugin](https://www.npmjs.com/package/dynamic-cdn-webpack-plugin).

#### Try to find a webpack plugin for your dependencies

Just importing packages like [moment.js](https://momentjs.com/) or [lodash](https://lodash.com/) bring in a lot of bloat that you probably don’t need. Try to import what you need only, or better yet find a webpack plugin that removes the unused things from your bundle, because [selective imports don’t always work](https://github.com/react-bootstrap/react-bootstrap/issues/2683). As one example, there’s [a webpack plugin](https://github.com/iamakulov/moment-locales-webpack-plugin) that removes a lot of the unnecessary bloat added by Moment.js.

Google actually has a [nice repository](https://github.com/GoogleChromeLabs/webpack-libs-optimizations) listing some common problematic dependencies.

#### Inspect your bundle with Webpack bundle analyzer

![](https://cdn-images-1.medium.com/max/800/0*TewqripGyXujWGJs.png)

[Webpack Bundle Analyzer](https://github.com/webpack-contrib/webpack-bundle-analyzer) is extremely helpful to see what exactly is going into your bundle. In the screenshot above, you’ll notice that moment.js has lots of localization files that your app probably doesn’t need. Webpack Bundle Analyzer can help you easily spot these issues.

#### Add es-check to your CI pipeline early on

[es-check](https://github.com/dollarshaveclub/es-check) will help you find out which ES version your bundle is using, it’s super useful to find out if you’re somehow suddenly not producing ES5 anymore. Even if you’re using Babel and browserslist, you might be importing a node module that’s not even meant to be used in browsers, or even a package that’s not being distributed as ES5. Add es-check to your continuous integration pipeline early on and it should help you find out if your bundle ever stops working with ES5, and that’ll help you find which package is the culprit so you can then transpile it.

#### Transpiling a node\_module

We had imported a very simple package called [hex-rgb](https://github.com/sindresorhus/hex-rgb) that’s not even meant for browsers and this tiny package made our bundle not ES5-compatible anymore. Such packages should go through Babel and be transpiled.

In your webpack config, your babel loader’s exclude field probably looks like this: `/node_modules/` . We need to make a regex that excludes node\_modules except the specific ones that should be transpiled:

```
// Exclude all node modules except hex-rgb and another-package
/node_modules\/(?![hex\-rgb|another\-package])/
```

And once again, this might not be a good solution for large packages as it can drastically slow your build time and you might want to switch to a CDN version instead.

Follow [this issue](https://github.com/babel/babel-loader/issues/171) from the babel-loader repo to stay up to date on how to handle cases like this.

#### Use Browserslist to specify your target browsers

[Browserslist](https://github.com/browserslist/browserslist) lets you specify which browsers to transpile for.

```
> 1%
ie >= 8
```

This simple configuration targets browsers with usage more than 1% global usage, and IE versions 8 and above.

#### Use babel.config.js over .babelrc (for Babel ≥ 7.0)

Favor using `babel.config.js` to [configure Babel](https://babeljs.io/docs/en/configuration) over `.babelrc` . If you want to transpile `node_modules` (which is now becoming a very common case with webapps), then you should use `babel.config.js` .

`.babelrc` can be overridden by another `.babelrc` belonging to a node\_module that you’re transpiling and that can lead to all sorts of weird issues.

#### Make your webpack-dev-server logging output friendlier

1. Change your [webpack-dev-server config](https://webpack.js.org/configuration/dev-server/) to this

```
devServer: {
  noInfo: true,
  stats: 'minimal'
}
```

2\. Add [WebpackBar](https://github.com/nuxt/webpackbar) to get much less-verbose, friendlier, and more concise output.

![](https://cdn-images-1.medium.com/max/800/1*3CqzwcXgMpT-42e22OMu7Q.png)

Note: The first configuration is meant to be combined with Webpack Bundle Analyzer, as it suppresses console output for things related to your bundle that Webpack Bundle Analyzer already shows. If you’re not using Webpack Bundle Analyzer, don’t apply the first step.

#### <br>


# DevOps


# Databases


# Estimating Connections

## Links

* [Heroku PostgreSQL Connection Calculator](https://railsautoscale.com/heroku-postgresql-connection-calculator/)


# Kubernetes

*See* [*Terminology*](https://github.com/obahareth/knowledge/tree/333c0901236edb40cfc5774172c5b200f5f2fe58/devops/kubernetes/terminology.md) *for a reference*

Kubernetes (K8s) is an open-source orchestration system for automating deployment, scaling, and management of containerized applications.

\[TOC]

## Architecture

![](https://www.aquasec.com/wiki/download/attachments/2855092/kubertes.png?version=1\&modificationDate=1520363380138\&api=v2)

​ (Image taken from <https://x-team.com/blog/introduction-kubernetes-architecture/>)

* Everything must be in a namespace, a **default** namespace is created when you create a cluster.

## Default objects

### Master Node (one is always present in a cluster)

* **API Server** - Allows interaction with kubernetes API, it's the frontend for the control plane.
* **kube-scheduler** - Assigns **pods** to **nodes** at runtime, and checks resources, quality of service, policies, and specifications before scheduling
* **Controller Manager** - Runs **controllers** (background threads that run tasks in a cluster). It's compiled into a single binary and has a number of roles:
  * **Node Controller** - Responsible for worker states.
  * **Replication Controller** - Responsible for maintining correct number of **pods** for replicated controllers.
  * **Endpoint Controller** - Responsible for joining **services** and **pods** togethers
  * **Service Account Contoller and Token Controller** - Responisble for handling access management to **API Server**.

### etcd

Database used by Kubernetes to store all cluster data (job scheduling info, pod details, stage information, etc.).

### kubectl

Commandline interface used to interact with the **master node**. It can be prunounced as Kube Cuttle. Kubectl has a config file which has server information and authentication information to access the API server.

## Nodes

A Node requires the following to be running:

* **Supervisord** - Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems
* **Container tooling** (e.g. Docker)

### kubelet

The main service on a node, regularly taking in new or modified pod specifications (primarily through the kube-apiserver) and ensuring that pods and their containers are healthy and running in the desired state. This component also reports to the master on the health of the host where it is running. It executed pod containers via the container engine and mounts pod volume and secrets. **Kubelet must be running on all nodes**, and if it isn't working correctly, you're going to have issues.

It takes a set of **Podspec**s (Yaml files provided by API Server) that describe a pod.

Kubelet only manages containers that were created by the API server, not any container running on the node.

### kube-proxy

A proxy service that runs on each worker node to deal with individual host subnetting and expose services to the external world. It performs request forwarding to the correct pods/containers across the various isolated networks in a cluster. **kube-proxy must be running in each node**.

* It can do simple network stream or round-robin forwarding across a set of backends.
* Services definged against the API server: kube-proxy watches the API server for the addition and removal of services.
* For each new service, kube-proxy opens a **randomly chosen port** on the local node.
* Connections made to the chosen port are proxied to one of the corresponding back-end pods.

Modes of kube-proxy:

* User space mode (most common).
* Iptables mode.
* Ipvs mode.

## Pods

A pod is one or more containers that should be controlled as a single application. It encapsulates application containers, storage resources, a unique network ID and other configuration on how to run the containers.

Pods are ephemeral/disposable, they never self-heal and are not restarted by the scheduler, never create them by themselves in production. Instead use higher level constructs like **controllers**.

**States**

* **Pending** - Accepted by Kubernetes system but a container hasn't been created yet.
* **Running** - A pod has been scheduled on a node, all containers have been created, and at least one container is in the **running** state.
* **Succeeded** - All containers in the pod have exited with a status of 0 (successful execution and will not be restarted)
* **Failed** - All containers have exited and at least one has failed (returned non-zero exit status)
* **CrashLoopBackOff** - A container fails to start and Kubernetes keeps trying to restart it

## Controllers

In Kubernetes, controllers are control loops that watch the state of your **cluster**, then make or request changes where needed. Each **controller** tries to move the current **cluster state** closer to the desired state. They help with application reliability, scaling, and load balancing.

### Controller Types

#### ReplicaSet

Ensures that the specified number of **replicas** for a **pod** are **running at all times**.

#### Deployment

Most applications are packaged as Deployments. Deployments provide declarative updates for **pods** and **ReplicaSets**. Used to describe desired state in a YAML file and the Deployment controller will align the actual state to match. Can be used to create new **ReplicaSets** or replace them with new ones. A **Deployment** manages the **ReplicaSet**, and the **ReplicaSet** manages the **Pod**. This architecture allows deployments to support a rollback mechanism.

A new **ReplicaSet** is created each time a new Deployment config is deployed but the old ReplicaSet is still kept to allow for easier rollbacks.

**Use-cases**

* **Pod management** - Running a ReplicaSet allows us to deploy a number of pods and check their status as a single unit.
* Scaling a ReplicaSet scales out the pods and allows for a deployment to handle more traffic.
* **Pause and Resume** (traffic will still get passed to existing replica set)
  * Used with larger changesets.
  * Pause, make changes, and resume deployment.

#### DaemonSet

Ensures all nodes run a copy of a specific pod. As nodes are added or removed from the cluster, a DaemonSet will add or remove the required pods.

**Use-cases**

* Run a single log aggregator.
* Monitoring agent.

#### Job

Supervisor process for pods carrying out batch jobs. Used to run individual processes that need to run once and complete successfully.

**Use-cases**

* Cron job to run nightly report or database backup.

#### Service

Allows the comunication between one set of deployments with another. When a service is created it is assigned a unique IP address that never changes throughout the lifetime of the service. Pods are then configured to talk to the service and can rely on the service IP on any requests that might be sent to the pod. Services are an important concept because they allow one set of pods to communicate with another set of pods in an easy way.

**Example**

Frontend deployment needs to talk to backend deployment (which consists of multiple pods), a backend service can provide a single IP for the frontend pod to talk with.

**Kinds of Services**

* **Internal** - IP is only reachable within the cluster
* **External** - Endpoint available through node IP (known as **NodePort**)
* **Load balancer** - Exposes application to the internet with a load balancer (available with a cloud provider)

## Labels and Selectors

Covered in the [terminology page](https://github.com/obahareth/knowledge/tree/333c0901236edb40cfc5774172c5b200f5f2fe58/devops/kubernetes/terminology.md).

## Namespaces

* Allows teams to access resources, with accountability.
* Great way to divide cluster resources between users.
* Names for resources **must be unique within a namespace**.
* A **default** namespace is created when kubernetes is launched.
* Newer applications install their resources in a different namespace so they don't interfere with an existing cluster.

## Maintenance

* Features are backward-compatible and APIs are versioned.
* Host can be turned off/on during maintenance.

## Logging and Monitoring

**Application Monitoring**

* Built-in TCP/HTTP/container-execution health checks.

**Node health check**

* Failures are monitored by node controller.

**Kubernetes status**

* Can be monitored through addons like [metrics-server](https://github.com/kubernetes-sigs/metrics-server) or the [Prometheus Operator](https://github.com/coreos/prometheus-operator).

## Limitations

*As of 1.16, see* [*official Limitations page*](https://kubernetes.io/docs/setup/best-practices/cluster-large/)*.*

* No more than 5000 nodes
* No more than 150000 total pods
* No more than 300000 total containers
* No more than 100 pods per node

## References

* [Kubernetes Architecture 101](https://www.aquasec.com/wiki/display/containers/Kubernetes+Architecture+101)
* [Learning Kubernetes](https://www.linkedin.com/learning/learning-kubernetes)


# Terminology

*Mostly taken from the* [*Kubernetes glossary*](https://kubernetes.io/docs/reference/glossary/?fundamental=true) *with added changes to help me understand better*

* **Cluster** - A set of machines, called **nodes**, that run containerized applications managed by Kubernetes. A cluster has at least one worker node and at least one master node.
* **Container** - A lightweight and portable executable image that contains software and all of its dependencies.
* **Container Runtime** - The container runtime is the software that is responsible for running **containers**.
* **Container runtime interface (CRI)** - The container runtime interface (CRI) is an API for container runtimes to integrate with **kubelet** on a **node**.
* **Control Plane** - The container orchestration layer that exposes the API and interfaces to define, deploy, and manage the lifecycle of **containers**.
* **Controller** - In Kubernetes, controllers are control loops that watch the state of your **cluster**, then make or request changes where needed. Each **controller** tries to move the current cluster state closer to the desired state.
* **DaemonSet** - Ensures a copy of a **Pod** is running across a set of **nodes** in a **cluster**.
* **Deployment** - An API object that manages a replicated application.
* **Ephemeral Container** - A **Container** type that you can temporarily run inside a **Pod**.
* **Image** - Stored instance of a **container** that holds a set of software needed to run an application.
* **Init Container** - One or more initialization containers that must run to completion before any app containers run.
* **Job** - A finite or batch task that runs to completion.
* **kube-controller-manager** Component on the master that runs **controllers**.
* **kube-proxy** - [kube-proxy](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/) is a network proxy that runs on **each node** in **your cluster**, implementing part of the Kubernetes [Service](https://kubernetes.io/docs/concepts/services-networking/service/) concept.
* **Kubectl** - A command line tool for communicating with a [Kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/) server.
* **Kubelet** An agent that runs on **each node** in **the cluster**. It makes sure that **containers** are running in a **pod**. **Each node must have a Kubelet**.
* **Kubernetes API** - The application that serves Kubernetes functionality through a RESTful interface and stores the state of the cluster.
* **Label** - Tags objects with identifying attributes that are meaningful and relevant to users (e.g. `"environment" : "production"`).
* **Namespace** - An abstraction used by Kubernetes to support **multiple virtual clusters on the same physical** [**cluster**](https://kubernetes.io/docs/reference/glossary/?all=true#term-cluster).
* **Node** - A node is a worker machine in Kubernetes. A node has **Pods** running inside it.
* **NodePort** - A service type used to expose the Service on each Node’s IP at a static port (the `NodePort`). A `ClusterIP` Service, to which the `NodePort` Service routes, is automatically created. You’ll be able to contact the `NodePort` Service, from outside the cluster, by requesting `<NodeIP>:<NodePort>`.
* **Pod** - The smallest and simplest Kubernetes object. A Pod represents a set of running [containers](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/#why-containers) on your cluster. An example of a pod is a Rails app backend, it can scale up/down horizontally through **ReplicaSets** which replicates pods.
* **ReplicaSet** - A ReplicaSet (aims to) maintain a set of replica Pods running at any given time.
* **Pod Lifecycle** - A high-level summary of what phase the **Pod** is in within its lifecyle.
* **Pod Security Policy** - Enables fine-grained authorization of **Pod** creation and updates.
* [**Selector**](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors) - Allows users to filter a list of resources based on labels. Examples:
  * `environment = production`
  * `tier != frontend`
  * `environment in (production, qa)`
  * `tier notin (frontend, backend)`
* **Service** - An abstract way to expose an application running on a set of [Pods](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/) as a network service.
* **Service Account** - Provides an identity for processes that run in a [Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/).
* **StatefulSet** - Manages the deployment and scaling of a set of [Pods](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/), *and provides guarantees about the ordering and uniqueness* of these Pods.
* **Static Pod** - A [pod](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/) managed directly by the **kubelet daemon** on a specific node.
* **Volume** - A directory containing data, accessible to the containers in a [pod](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/).


# kubectl Cheatsheet

\[TOC]

## References

* [cheatsheet-kubernetes-A4](https://github.com/dennyzhang/cheatsheet-kubernetes-A4) ([Internet Archive link](https://web.archive.org/web/20191112135614/https://github.com/dennyzhang/cheatsheet-kubernetes-A4)). This page is taken from it.

## 1.1 Common Commands

| Name                                                                              | Command                                                                                                                                                                                                                                         |
| --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Run curl test temporarily                                                         | `kubectl run --rm mytest --image=yauritux/busybox-curl -it`                                                                                                                                                                                     |
| Run wget test temporarily                                                         | `kubectl run --rm mytest --image=busybox -it`                                                                                                                                                                                                   |
| Run nginx deployment with 2 replicas                                              | `kubectl run my-nginx --image=nginx --replicas=2 --port=80`                                                                                                                                                                                     |
| Run nginx pod and expose it                                                       | `kubectl run my-nginx --restart=Never --image=nginx --port=80 --expose`                                                                                                                                                                         |
| Run nginx deployment and expose it                                                | `kubectl run my-nginx --image=nginx --port=80 --expose`                                                                                                                                                                                         |
| Set namespace preference                                                          | `kubectl config set-context  --namespace=`                                                                                                                                                                                                      |
| List pods with nodes info                                                         | `kubectl get pod -o wide`                                                                                                                                                                                                                       |
| List everything                                                                   | `kubectl get all --all-namespaces`                                                                                                                                                                                                              |
| Get all services                                                                  | `kubectl get service --all-namespaces`                                                                                                                                                                                                          |
| Get all deployments                                                               | `kubectl get deployments --all-namespaces`                                                                                                                                                                                                      |
| Show nodes with labels                                                            | `kubectl get nodes --show-labels`                                                                                                                                                                                                               |
| Get resources with json output                                                    | `kubectl get pods --all-namespaces -o json`                                                                                                                                                                                                     |
| Validate yaml file with dry run                                                   | `kubectl create --dry-run --validate -f pod-dummy.yaml`                                                                                                                                                                                         |
| Start a temporary pod for testing                                                 | `kubectl run --rm -i -t --image=alpine test-$RANDOM -- sh`                                                                                                                                                                                      |
| kubectl run shell command                                                         | `kubectl exec -it mytest -- ls -l /etc/hosts`                                                                                                                                                                                                   |
| Get system conf via configmap                                                     | `kubectl -n kube-system get cm kubeadm-config -o yaml`                                                                                                                                                                                          |
| Get deployment yaml                                                               | `kubectl -n denny-websites get deployment mysql -o yaml`                                                                                                                                                                                        |
| Explain resource                                                                  | `kubectl explain pods`, `kubectl explain svc`                                                                                                                                                                                                   |
| Watch pods                                                                        | `kubectl get pods -n wordpress --watch`                                                                                                                                                                                                         |
| Query healthcheck endpoint                                                        | `curl -L http://127.0.0.1:10250/healthz`                                                                                                                                                                                                        |
| Open a bash terminal in a pod                                                     | `kubectl exec -it storage sh`                                                                                                                                                                                                                   |
| Check pod environment variables                                                   | `kubectl exec redis-master-ft9ex env`                                                                                                                                                                                                           |
| Enable kubectl shell autocompletion                                               | `echo "source <(kubectl completion bash)" >>~/.bashrc`, and reload                                                                                                                                                                              |
| Use minikube dockerd in your laptop                                               | `eval $(minikube docker-env)`, No need to push docker hub any more                                                                                                                                                                              |
| Kubectl apply a folder of yaml files                                              | `kubectl apply -R -f .`                                                                                                                                                                                                                         |
| Get services sorted by name                                                       | kubectl get services –sort-by=.metadata.name                                                                                                                                                                                                    |
| Get pods sorted by restart count                                                  | kubectl get pods –sort-by=’.status.containerStatuses\[0].restartCount’                                                                                                                                                                          |
| List pods and images                                                              | kubectl get pods -o=’custom-columns=PODS:.metadata.name,Images:.spec.containers\[\*].image’                                                                                                                                                     |
| List all container images                                                         | [list-all-images.sh](https://github.com/dennyzhang/cheatsheet-kubernetes-A4/blob/master/list-all-images.sh#L14-L17)                                                                                                                             |
| kubeconfig skip tls verification                                                  | [skip-tls-verify.md](https://github.com/dennyzhang/cheatsheet-kubernetes-A4/blob/master/skip-tls-verify.md)                                                                                                                                     |
| [Ubuntu install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) | =”deb [apt.kubernetes.io](https://apt.kubernetes.io/) kubernetes-xenial main”=                                                                                                                                                                  |
| Reference                                                                         | [GitHub: kubernetes releases](https://github.com/kubernetes/kubernetes/tags)                                                                                                                                                                    |
| Reference                                                                         | [minikube cheatsheet](https://cheatsheet.dennyzhang.com/cheatsheet-minikube-A4), [docker cheatsheet](https://cheatsheet.dennyzhang.com/cheatsheet-docker-A4), [OpenShift CheatSheet](https://cheatsheet.dennyzhang.com/cheatsheet-openshift-A4) |

## 1.2 Check Performance

| Name                                         | Command                                              |
| -------------------------------------------- | ---------------------------------------------------- |
| Get node resource usage                      | `kubectl top node`                                   |
| Get pod resource usage                       | `kubectl top pod`                                    |
| Get resource usage for a given pod           | `kubectl top  --containers`                          |
| List resource utilization for all containers | `kubectl top pod --all-namespaces --containers=true` |

## 1.3 Resources Deletion

| Name                                    | Command                                        |
| --------------------------------------- | ---------------------------------------------- |
| Delete pod                              | `kubectl delete pod/ -n`                       |
| Delete pod by force                     | `kubectl delete pod/ --grace-period=0 --force` |
| Delete pods by labels                   | `kubectl delete pod -l env=test`               |
| Delete deployments by labels            | `kubectl delete deployment -l app=wordpress`   |
| Delete all resources filtered by labels | `kubectl delete pods,services -l name=myLabel` |
| Delete resources under a namespace      | `kubectl -n my-ns delete po,svc --all`         |
| Delete persist volumes by labels        | `kubectl delete pvc -l app=wordpress`          |
| Delete state fulset only (not pods)     | `kubectl delete sts/ --cascade=false`          |

## 1.4 Log & Conf Files

| Name                      | Comment                                                                   |
| ------------------------- | ------------------------------------------------------------------------- |
| Config folder             | `/etc/kubernetes/`                                                        |
| Certificate files         | `/etc/kubernetes/pki/`                                                    |
| Credentials to API server | `/etc/kubernetes/kubelet.conf`                                            |
| Superuser credentials     | `/etc/kubernetes/admin.conf`                                              |
| kubectl config file       | `~/.kube/config`                                                          |
| Kubernets working dir     | `/var/lib/kubelet/`                                                       |
| Docker working dir        | `/var/lib/docker/`, `/var/log/containers/`                                |
| Etcd working dir          | `/var/lib/etcd/`                                                          |
| Network cni               | `/etc/cni/net.d/`                                                         |
| Log files                 | `/var/log/pods/`                                                          |
| log in worker node        | `/var/log/kubelet.log`, `/var/log/kube-proxy.log`                         |
| log in master node        | `kube-apiserver.log`, `kube-scheduler.log`, `kube-controller-manager.log` |
| Env                       | `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf`                   |
| Env                       | export KUBECONFIG=/etc/kubernetes/admin.conf                              |

## 1.5 Pod

| Name                                                                             | Command                                                                                        |
| -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| List all pods                                                                    | `kubectl get pods`                                                                             |
| List pods for all namespace                                                      | `kubectl get pods -all-namespaces`                                                             |
| List all critical pods                                                           | `kubectl get -n kube-system pods -a`                                                           |
| List pods with more info                                                         | `kubectl get pod -o wide`, `kubectl get pod/ -o yaml`                                          |
| Get pod info                                                                     | `kubectl describe pod/srv-mysql-server`                                                        |
| List all pods with labels                                                        | `kubectl get pods --show-labels`                                                               |
| [List all unhealthy pods](https://github.com/kubernetes/kubernetes/issues/49387) | kubectl get pods –field-selector=status.phase!=Running –all-namespaces                         |
| List running pods                                                                | kubectl get pods –field-selector=status.phase=Running                                          |
| Get Pod initContainer status                                                     | `kubectl get pod --template '{{.status.initContainerStatuses}}'`                               |
| kubectl run command                                                              | kubectl exec -it -n “$ns” “$podname” – sh -c “echo $msg >>/dev/err.log”                        |
| Watch pods                                                                       | `kubectl get pods -n wordpress --watch`                                                        |
| Get pod by selector                                                              | kubectl get pods –selector=”app=syslog” -o jsonpath=’{.items\[\*].metadata.name}’              |
| List pods and images                                                             | kubectl get pods -o=’custom-columns=PODS:.metadata.name,Images:.spec.containers\[\*].image’    |
| List pods and containers                                                         | -o=’custom-columns=PODS:.metadata.name,CONTAINERS:.spec.containers\[\*].name’                  |
| Reference                                                                        | [Link: kubernetes yaml templates](https://cheatsheet.dennyzhang.com/kubernetes-yaml-templates) |

## 1.6 Label & Annontation

| Name                             | Command                                                           |
| -------------------------------- | ----------------------------------------------------------------- |
| Filter pods by label             | `kubectl get pods -l owner=denny`                                 |
| Manually add label to a pod      | `kubectl label pods dummy-input owner=denny`                      |
| Remove label                     | `kubectl label pods dummy-input owner-`                           |
| Manually add annonation to a pod | `kubectl annotate pods dummy-input my-url=https://dennyzhang.com` |

## 1.7 Deployment & Scale

| Name                         | Command                                                                                                                                                                                                                                            |
| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Scale out                    | `kubectl scale --replicas=3 deployment/nginx-app`                                                                                                                                                                                                  |
| online rolling upgrade       | `kubectl rollout app-v1 app-v2 --image=img:v2`                                                                                                                                                                                                     |
| Roll backup                  | `kubectl rollout app-v1 app-v2 --rollback`                                                                                                                                                                                                         |
| List rollout                 | `kubectl get rs`                                                                                                                                                                                                                                   |
| Check update status          | `kubectl rollout status deployment/nginx-app`                                                                                                                                                                                                      |
| Check update history         | `kubectl rollout history deployment/nginx-app`                                                                                                                                                                                                     |
| Pause/Resume                 | `kubectl rollout pause deployment/nginx-deployment`, `resume`                                                                                                                                                                                      |
| Rollback to previous version | `kubectl rollout undo deployment/nginx-deployment`                                                                                                                                                                                                 |
| Reference                    | [Link: kubernetes yaml templates](https://cheatsheet.dennyzhang.com/kubernetes-yaml-templates), [Link: Pausing and Resuming a Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#pausing-and-resuming-a-deployment) |

## 1.8 Quota & Limits & Resource

| Name                          | Command                                                                                        |
| ----------------------------- | ---------------------------------------------------------------------------------------------- |
| List Resource Quota           | `kubectl get resourcequota`                                                                    |
| List Limit Range              | `kubectl get limitrange`                                                                       |
| Customize resource definition | `kubectl set resources deployment nginx -c=nginx --limits=cpu=200m`                            |
| Customize resource definition | `kubectl set resources deployment nginx -c=nginx --limits=memory=512Mi`                        |
| Reference                     | [Link: kubernetes yaml templates](https://cheatsheet.dennyzhang.com/kubernetes-yaml-templates) |

## 1.9 Service

| Name                            | Command                                                                                        |
| ------------------------------- | ---------------------------------------------------------------------------------------------- |
| List all services               | `kubectl get services`                                                                         |
| List service endpoints          | `kubectl get endpoints`                                                                        |
| Get service detail              | `kubectl get service nginx-service -o yaml`                                                    |
| Get service cluster ip          | kubectl get service nginx-service -o go-template=’’                                            |
| Get service cluster port        | kubectl get service nginx-service -o go-template=’’                                            |
| Expose deployment as lb service | `kubectl expose deployment/my-app --type=LoadBalancer --name=my-service`                       |
| Expose service as lb service    | `kubectl expose service/wordpress-1-svc --type=LoadBalancer --name=ns1`                        |
| Reference                       | [Link: kubernetes yaml templates](https://cheatsheet.dennyzhang.com/kubernetes-yaml-templates) |

## 1.10 Secrets

| Name                             | Command                                                                                                                                                                    |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| List secrets                     | `kubectl get secrets --all-namespaces`                                                                                                                                     |
| Generate secret                  | `echo -n 'mypasswd'=, then redirect to =base64 --decode`                                                                                                                   |
| Get secret                       | `kubectl get secret denny-cluster-kubeconfig`                                                                                                                              |
| Get a specific field of a secret | kubectl get secret denny-cluster-kubeconfig -o jsonpath=”{.data.value}”                                                                                                    |
| Create secret from cfg file      | kubectl create secret generic db-user-pass –from-file=./username.txt                                                                                                       |
| Reference                        | [Link: kubernetes yaml templates](https://cheatsheet.dennyzhang.com/kubernetes-yaml-templates), [Link: Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) |

## 1.11 StatefulSet

| Name                               | Command                                                                                        |
| ---------------------------------- | ---------------------------------------------------------------------------------------------- |
| List statefulset                   | `kubectl get sts`                                                                              |
| Delete statefulset only (not pods) | `kubectl delete sts/ --cascade=false`                                                          |
| Scale statefulset                  | `kubectl scale sts/ --replicas=5`                                                              |
| Reference                          | [Link: kubernetes yaml templates](https://cheatsheet.dennyzhang.com/kubernetes-yaml-templates) |

## 1.12 Volumes & Volume Claims

| Name                      | Command                                                                                        |
| ------------------------- | ---------------------------------------------------------------------------------------------- |
| List storage class        | `kubectl get storageclass`                                                                     |
| Check the mounted volumes | `kubectl exec storage ls /data`                                                                |
| Check persist volume      | `kubectl describe pv/pv0001`                                                                   |
| Copy local file to pod    | `kubectl cp /tmp/my /:/tmp/server`                                                             |
| Copy pod file to local    | `kubectl cp /:/tmp/server /tmp/my`                                                             |
| Reference                 | [Link: kubernetes yaml templates](https://cheatsheet.dennyzhang.com/kubernetes-yaml-templates) |

## 1.13 Events & Metrics

| Name                            | Command                                                 |
| ------------------------------- | ------------------------------------------------------- |
| View all events                 | `kubectl get events --all-namespaces`                   |
| List Events sorted by timestamp | kubectl get events –sort-by=.metadata.creationTimestamp |

## 1.14 Node Maintenance

| Name                                      | Command                       |
| ----------------------------------------- | ----------------------------- |
| Mark node as unschedulable                | `kubectl cordon $NDOE_NAME`   |
| Mark node as schedulable                  | `kubectl uncordon $NDOE_NAME` |
| Drain node in preparation for maintenance | `kubectl drain $NODE_NAME`    |

## 1.15 Namespace & Security

| Name                                                                                    | Command                                                                                            |
| --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| List authenticated contexts                                                             | `kubectl config get-contexts`, `~/.kube/config`                                                    |
| Set namespace preference                                                                | `kubectl config set-context  --namespace=`                                                         |
| Load context from config file                                                           | `kubectl get cs --kubeconfig kube_config.yml`                                                      |
| Switch context                                                                          | `kubectl config use-context`                                                                       |
| Delete the specified context                                                            | `kubectl config delete-context`                                                                    |
| List all namespaces defined                                                             | `kubectl get namespaces`                                                                           |
| List certificates                                                                       | `kubectl get csr`                                                                                  |
| [Check user privilege](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) | kubectl –as=system:serviceaccount:ns-denny:test-privileged-sa -n ns-denny auth can-i use pods/list |
| [Check user privilege](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) | `kubectl auth can-i use pods/list`                                                                 |
| Reference                                                                               | [Link: kubernetes yaml templates](https://cheatsheet.dennyzhang.com/kubernetes-yaml-templates)     |

## 1.16 Network

| Name                              | Command                                                  |
| --------------------------------- | -------------------------------------------------------- |
| Temporarily add a port-forwarding | `kubectl port-forward redis-134 6379:6379`               |
| Add port-forwaring for deployment | `kubectl port-forward deployment/redis-master 6379:6379` |
| Add port-forwaring for replicaset | `kubectl port-forward rs/redis-master 6379:6379`         |
| Add port-forwaring for service    | `kubectl port-forward svc/redis-master 6379:6379`        |
| Get network policy                | `kubectl get NetworkPolicy`                              |

## 1.17 Patch

| Name                          | Summary                                                                |
| ----------------------------- | ---------------------------------------------------------------------- |
| Patch service to loadbalancer | =kubectl patch svc $svc\_name -p ‘{“spec”: {“type”: “LoadBalancer”}}’= |

## 1.18 Extenstions

| Name                                    | Summary                    |
| --------------------------------------- | -------------------------- |
| Enumerates the resource types available | `kubectl api-resources`    |
| List api group                          | `kubectl api-versions`     |
| List all CRD                            | `kubectl get crd`          |
| List storageclass                       | `kubectl get storageclass` |

## 1.19 Components & Services

### 1.19.1 Services on Master Nodes

| Name                                                                                                        | Summary                                                                                                |
| ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| [kube-apiserver](https://github.com/kubernetes/kubernetes/tree/master/cmd/kube-apiserver)                   | exposes the Kubernetes API from master nodes                                                           |
| [etcd](https://coreos.com/etcd/)                                                                            | reliable data store for all k8s cluster data                                                           |
| [kube-scheduler](https://github.com/kubernetes/kubernetes/tree/master/cmd/kube-scheduler)                   | schedule pods to run on selected nodes                                                                 |
| [kube-controller-manager](https://github.com/kubernetes/kubernetes/tree/master/cmd/kube-controller-manager) | node controller, replication controller, endpoints controller, and service account & token controllers |

### 1.19.2 Services on Worker Nodes

| Name                                                                              | Summary                                                                                                                                     |
| --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| [kubelet](https://github.com/kubernetes/kubernetes/tree/master/cmd/kubelet)       | makes sure that containers are running in a pod                                                                                             |
| [kube-proxy](https://github.com/kubernetes/kubernetes/tree/master/cmd/kube-proxy) | perform connection forwarding                                                                                                               |
| [Container Runtime](https://github.com/docker/engine)                             | Kubernetes supported runtimes: Docker, rkt, runc and any [OCI runtime-spec](https://github.com/opencontainers/runtime-spec) implementation. |

### 1.19.3 Addons: pods and services that implement cluster features

| Name                          | Summary                                                                   |
| ----------------------------- | ------------------------------------------------------------------------- |
| DNS                           | serves DNS records for Kubernetes services                                |
| Web UI                        | a general purpose, web-based UI for Kubernetes clusters                   |
| Container Resource Monitoring | collect, store and serve container metrics                                |
| Cluster-level Logging         | save container logs to a central log store with search/browsing interface |

### 1.19.4 Tools

| Name                                                                         | Summary                                                                                 |
| ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| [kubectl](https://github.com/kubernetes/kubernetes/tree/master/cmd/kubectl)  | the command line util to talk to k8s cluster                                            |
| [kubeadm](https://github.com/kubernetes/kubernetes/tree/master/cmd/kubeadm)  | the command to bootstrap the cluster                                                    |
| [kubefed](https://kubernetes.io/docs/reference/setup-tools/kubefed/kubefed/) | the command line to control a Kubernetes Cluster Federation                             |
| Kubernetes Components                                                        | [Link: Kubernetes Components](https://kubernetes.io/docs/concepts/overview/components/) |


# Best Practices

## Links

* [Kubernetes in Production: Readiness Checklist and Best Practices for Resource Management](https://www.replex.io/blog/kubernetes-in-production-readiness-checklist-and-best-practices-for-resource-management)
* [Configuration Best Practices](https://kubernetes.io/docs/concepts/configuration/overview/)
* [Setup Best Practices](https://kubernetes.io/docs/setup/best-practices/)
* [A Practical Guide to Setting Kubernetes Requests and Limits](http://blog.kubecost.com/blog/requests-and-limits/)


# Application Health Checks

## Readiness Probe

Used to know when a **container is ready to accept traffic**. It can be configured to wait for a number of seconds after pod startup before applying health checking, and with a number of seconds to time out.

Types of readiness probes:

* HTTP probe.

## Liveness Probe

Periodic check used be Kubernetes to make sure the **container is still healthy**.


# Upgrades

We can upgrade by using `kubectl set image deployment/deployment_name helloworld=obahareth/new_hello_world`.

When we redeploy with a new image we get a new replicaset that matches the new set of images.

## Rollouts

We can view the rollout history by running `kubectl rollout history deployment/deployment_name`, we can use the revision numbers here to roll back to a specific version as well.

### Undoing

`kubectl rollout undo deployment/deployment_name`

This will roll back to the previous version

### Rolling Back to a Specific Version

`kubectl rollout undo deployment/deployment_name --to-revision=3`

This will roll back to revision 3.


# Troubleshooting

* Describe the deployment, `kubectl describe deployment bad_deployment`.
* Decribe a pod, `kubectl describe pod bad_pod`.
* Get the logs for a pod `kubectl logs pod_name`.
* Exec into the pod, `kubectl exec -it pod_name /bin/bash` (If you have multiple containers in the pod use `-c container_name` to select the desired container)


# Ruby on Rails

\[TOC]

## References

* [kubernetes-rails.com](http://kubernetes-rails.com/) by [Marco Colli](http://collimarco.com/) is a goldmine of information on running Rails on Kubernetes. I've archived it using the Wayback Machine just in case [here](https://web.archive.org/web/20191112135115/http://kubernetes-rails.com/). This page is taken from it.

## Deploying a Rails application to Kubernetes

There are many ways to deploy a Ruby on Rails application: one of them is using Docker containers and Kubernetes for orchestration. This guide shows some of the advantages of Kubernetes compared to other solutions and explains how to deploy a Rails application in production using Kubernetes. We focus on the usage of containers for production, rather than development, and we value simple solutions. This guide covers all the common aspects required for running a Rails application in production, including the deployment and continuous delivery of the web application, the configuration of a load balancer and domain, the environment variables and secrets, the compilation of assets, the database migrations, logging and monitoring, the background workers and cron jobs, and how to run maintenance tasks and updates.

## History and alternatives to Kubernetes

The easiest way for deploying a Rails application is probably using a PaaS, like Heroku, which makes the deployment and scaling extremely simple and lets you forget about servers. However:

* as the application scales, the cost may become prohibitive for your kind of business;
* you don't have full control of your application, which is managed by others, and this may raise concerns about uptime;
* you have constraints imposed by the platform;
* your application may become bounded to a specific platform, raising portability concerns.

A cheaper alternative is using a IaaS, like DigitalOcean. You can start with a single server, but soon you will need to scale horizontally on multiple servers. Usually you have at least one load balancer with HAProxy, some web servers with nginx and Puma, a database (probably Postgresql and Redis with replicas) and maybe some separate servers for background processing (e.g. Sidekiq). When you need to scale the application you just create a snapshot of a server and you replicate it. You can also manage or update multiple servers with pssh or using configuration management tools like Chef and the application can be easily deployed with Capistrano. It is not very hard to create and configure a bunch of servers. However:

* the initial setup requires some time and knowledge;
* applying changes to many servers may become painful;
* running the wrong command on a fleet of servers may be difficult to revert;
* you must make sure to keep all the servers updated with the same configuration;
* scaling requires a lot of manual work.

Kubernetes offers the advantages of a PaaS at the cost of a IaaS, so it is a good compromise that you should consider. It is also an open source technology and most cloud providers already offer it as a managed service.

Let's see how to deploy a Rails application in production using Kubernetes.

## Prerequisites

This guide assumes that you already have general knowledge about web development.

We also expect that you already have a development machine with all the necessary applciations installed, including Ruby (e.g. using rbenv), Ruby on Rails, Git, Docker, etc.

You also need to have an account on Docker Hub and DigitalOcean in order to try Kubernetes (or you can use your favorite alternatives).

## The Rails application

You can use an existing Rails application or you can create an example Rails application with this command:

```bash
rails new kubernetes-rails-example
```

Then add a simple page to the example application:

*config/routes.rb\*\*app/controllers/pages\_controller.rb*

```ruby
class PagesController < ApplicationController
  def home
  end
end
```

*app/views/pages/home.html.erb*

## The Git repository

Let's save the changes in the local Git repository, which was already initialized by Rails:

```
git add .
git commit -m "Initial commit"
```

Then we need to create a Git repository online. Go to Github and create a new repository, then connect the local repository to the remote one and publish the changes:

```
git remote add origin https://github.com/username/kubernetes-rails-example.git
git push -u origin master
```

Although a Git repository is not strictly required for Docker and Kubernetes, I mention it here because most CI/CD tools, including Docker Hub, can be connected to your Git repository in order to build the Docker image automatically whenever you push some changes.

## The Docker image

First step for containerization is to create the Docker image. A Docker image is simply a package which contains our application, together with all the dependencies and system libraries needed to run it.

Add this file in the root folder of your Rails application:

*Dockerfile*

First we use `FROM` to tell Docker to download a public image, which is then used as the base for our custom image. In particular we use an image which contains a specific version of Ruby.

Then we use `RUN` to execute a command inside the image that we are building. In particular we use `apt-get` to install some libraries. Note that the libraries available on the default Ubuntu repositories are usually quite old: if you want to get the latest versions you need to update the repository list and tell APT to download the libraries directly from the maintainer's repository. In particular, just before `apt-get`, we can add the following commands to update the Node.js and Yarn repositories:

```
RUN curl https://deb.nodesource.com/setup_12.x | bash
RUN curl https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
```

In the next block of code, we copy our Rails application to the image and we install all the required gems using Bundler. The Gemfile is copied before the other application code because Docker can use caching to build the image faster in case there aren't any changes to the Gemfile.

We also run a task to precompile the assets (stylesheets, scripts, etc.).

Finally we configure a default command to be executed on the image.

Before building the image we want to make sure that some files are not copied to it: this is important to exclude *secrets*, for security, and useless directories, like `tmp` or `.git`, which would be a waste of resources. For this we need to create a `.dockerignore` file in the Rails root: you can usually take inspiration from your `.gitignore`, since the aim and syntax are very similar.

It's time to build the image:

```
docker build -t username/kubernetes-rails-example:latest .
```

The `-t` option, followed by its argument, is optional: we use it to assign a *name* and a *tag* to the new image. This makes it easier to find the image later. We can also use a *repository* name as the image name, in order to allow push to that repository later. The image name is the part before the colon, while the tag is the part after the colon. Note that we could also omit the tag `latest` since it is the default value used in case the tag is omitted. The last dot is a required argument and indicates the path to the Dockerfile.

Then you can see that the image is actually available on your machine:

```
docker image ls
```

You can also use the image ID or its name to run the image (a running image is called a *container*):

```
docker run -p 3000:3000 username/kubernetes-rails-example:latest
```

Note that we map the host port 3000 (on the left) to the container port 3000 (on the right). You can also use other ports if you prefer: however, if you change the container port, you also need to update the image to make sure that the Rails server listens on the correct port and you also need to open that port using `EXPOSE`.

You can now see your webiste by visiting `http://localhost:3000`.

Finally we can push the image to the online repository. First you need to sign up to Docker Hub, or to another *registry* and create a *reposotory* there for your image. Then you can push the local image to the remote repository:

```
docker push username/kubernetes-rails-example:latest
```

## The Kubernetes cluster

It's time to create the Kubernetes cluster for production. Go to your favorite Kubernetes provider and create a cluster using the dashboard: we will use DigitalOcean for this tutorial.

Once the cluster is created you need to downlooad the credential and cluster configuration to your machine, so that you can connect to the cluster. For example you can move the configuration file to `~/.kube/kubernetes-rails-example-kubeconfig.yaml`. Then you need to pass a `--kubeconfig` option to `kubektl` whenever you invoke a command, or you can set an environment variable:

```
export KUBECONFIG=~/.kube/kubernetes-rails-example-kubeconfig.yaml
```

Then you need to make sure that you have Kubernetes installed and that it can connect to the remote cluster. Run this command:

```
kubectl version
```

You should see the version of your command line tools and the version of the cluster.

You can also play around with this command:

```
kubectl get nodes
```

A *node* is simply a server managed by Kubernetes. Then Kubernetes creates some virtual machines (in a broad sense) called *pods* on each node, based on our configuration. Pods are distributed automatically by Kubernetes on the available nodes and, in case a node fails, Kubernetes will move the pod to a different node. A pod usually contains a single *container*, but it can also have multiple related containers that need to share some resources.

The next step is to run some pods with our Docker image. Since our Docker image is probably hosted in a private repository, we need to give the Docker credentials to Kubernetes, so that it can download the image. Run these commands:

```
kubectl create secret docker-registry my-docker-secret --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL
kubectl edit serviceaccounts default
```

And add this to the end after `Secrets`:

```yaml
imagePullSecrets:
- name: my-docker-secret
```

Then you can define the Kubernetes configuration: create a `config/kube` directory in your Rails application.

We start by defining a deploy for the Rails application:

*config/kube/deployment.yml*

The above is a minimal deployment:

* `apiVersion` sets the API version for the configuration file;
* `kind` sets the type of configuration file;
* `metadata` is used to assign a name to the deployment;
* `replicas` tells Kubernetes to spin up a given number of pods;
* `selector` tells Kubernetes what template to use to generate the pods;
* `template` defines the template for a pod;
* `spec` sets the Docker image that we want to run inside the pods and other configurations, like the container port that must be exposed.

Now we also need to forward and distribute the HTTP requests to the pods. Let's create a load balancer:

*config/kube/load\_balancer.yml*

Basically we tell the load balancer:

* listen on the default port 80;
* forward the requests on port 3000 of the pods that have the label `rails-app`.

Now we can apply the configuration to Kubernetes, using a *declarative* management:

```
kubectl apply -f config/kube
```

It't time to verify that the pods are running properly:

1. `kubectl get pods` shows the pods and their status;
2. the status for all pods should be `Running`;
3. if you see the error `ImagePullBackOff` probably you have not configured properly the secret to download the image from your private repository;
4. you can get more details about any errors by running `kubectl describe pod pod-name`;
5. you can fix the configurations and then regenerate the pods by running `kubectl delete --all pods`.

Now you can get the load balancer IP and other information by running the folowing commands:

```
kubectl get services
kubectl describe service service-name
```

In particular you must find the `LoadBalancer Ingress` or `EXTERNAL-IP` and type it in your browser address bar: our website is up and running!

## Domain name and SSL

Probably your users won't access your website using the IP address, so you need to configure a domain name. Add the following record to your DNS:

```
example.com.   A   192.0.2.0
```

Obviously you need to replace the domain name with your domain and the IP address with the *external IP* of the load balancer (you can get it using `kubectl get services`).

An SSL certificate can be added to your website using proprietary YAML configurations, but the easiest way, if you are using DigitalOcean, is to use their dashboard to configure the SSL certificate (i.e. go to the load balancer settings).

## Environment variables

There are different solutions for storing environment variables:

* you can define the env variables inside your Rails configuration;
* you can define the env variables inside your Dockerfile;
* you can define the env variables using Kubernetes.

I suggest that you use Kubernetes for managing the environment variables for production: that makes it easy to change them without having to build a new image each time. Also Kubernetes has more information available about your environment and it can set some variables dinamically for you (e.g. it can set a variable with the pod name or IP). Finally it offers more granularity when you have multiple deployments (e.g. Puma and Sidekiq), since you can set different values for each deployment.

Add the following attribute to your *container* definition (e.g. after the `image` attribute) inside *config/kube/deployment.yml*:

```yaml
env:
- name: EXAMPLE
  value: This env variable is defined by Kubernetes.
```

Then you can update the Kubernetes cluster by running this command:

```
kubectl apply -f config/kube
```

If you want to make sure that everything works, you can try to print the env variable inside your app.

For development and test you can use a gem called *dotenv* to easily define the env variables.

For production you can also use Kubernetes *ConfigMaps* to define the env variables. The advantage of using this method is that you can define the env variables once and then use them for different *Deployments* or *Pods*. For example a Rails application usually requires the followig variables:

*config/kube/env.yml*

Then add the following attribute inside a *container* definition (e.g. after the `image` attribute):

*config/kube/deployment.yml*

Remember that it is not safe to store secrets, like `SECRET_KEY_BASE`, in your Git repository! In the next section we will see how to use Rails credentials to safely store your secrets.

## Secrets

You can store your secrets either in your Rails configuration or using Kubernetes secrets. I suggest that you use the Rails *credentials* to store your secrets: we will use Kubernetes secrets only to store the master key. Basically we store all our credentials in the Git repository, along with our application, but this is safe because we encrypt them with a master key. Then we use the master key, not stored in the Git repository, to access those secrets.

Enable this option inside *config/environments/production.rb*:

```ruby
config.require_master_key = true
```

Then run this command to edit your credentials:

```
EDITOR="vi" rails credentials:edit
```

While editing the file add the following line and then save and close:

```ruby
example_secret: foobar
```

Then inside your app you can try to print the secret:

*app/views/pages/home.html.erb*

Note that starting from Rails 6 you can also define different credentials for different environments.

If you run your website locally you can see the secret displayed properly. The last thing that we need to do is to give the master key to Kubernetes in a secure way:

```
kubectl create secret generic rails-secrets --from-literal=rails_master_key='example'
```

Your master key is usually stored in *config/master.key*.

Finally we need to pass the Kubernetes secret as an environment variable to our containers. Add this variable to your `env` inside *config/kube/deployment.yml*:

```yaml
- name: RAILS_MASTER_KEY
  valueFrom:
    secretKeyRef:
      name: rails-secrets
      key: rails_master_key
```

In order to test if everything works, you can rebuild the Docker image and deploy the new configuration: you should see the example secret (not a real secret) displayed on your homepage.

## Logging

There are two different strategies for logging:

* send the logs directly from your Rails app to a centralized logging service;
* log to *stdout* and let Docker and Kubernetes collect the logs on the node.

The first option is simple, but you don't collect the Kubernetes logs and might be less efficient. In any case you can use a gem like *logstash-logger* for this.

If you want to use the second option, you can enable logging to stdout for your Rails app by settings the env variable `RAILS_LOG_TO_STDOUT` to `enabled`.

Then you can see the latest logs using this command:

```
kubectl logs -l app=rails-app
```

Basically, when you run the command, the Kubernetes master node gets the latest logs from the nodes (only for pods with label `rails-app`) and displays them. This is useful for getting started, however logs are not persistent and you need to make them searchable. For this reason you need to send them to a centralized logging service: we can use Logz.io for example, which offers a managed ELK stack. In order to send the logs from Kubernetes to ELK we use Fluentd, which is a log collector written in Ruby and a CNCF graduated project.

This is how logging works:

1. your Rails application and other Kubernetes components write the logs to stdout;
2. Kubernetes collects and store the logs on the nodes;
3. you use a Kubernetes DaemonSet to run a Fluentd pod on each node;
4. Fluentd reads the logs from the node and sends them to the centralized logging service;
5. you can use the logging service to read, visualize and search all the logs.

You can install Fluentd on your cluster with these simple commands:

```
kubectl create secret generic logzio-secrets --from-literal=logzio_token='MY_LOGZIO_TOKEN' --from-literal=logzio_url='MY_LOGZIO_URL' -n kube-system

kubectl apply -f https://raw.githubusercontent.com/collimarco/logzio-k8s/use_k8s_secrets/logzio-daemonset-rbc.yaml
```

If you need custom configurations, you can download the file and edit it before running `kubectl apply`. Note that if you use services different from Logz.io the strategy is very similar and you can find many configuration examples on the Github repository *fluent/fluentd-kubernetes-daemonset*.

You can verify if everything works by visiting your website and then checking the logs.

## Background jobs

Let's see how to run Sidekiq on Kubernetes, in order to have some background workers.

First of all you need to add Sidekiq to your Rails application:

*Gemfile*

Then run `bundle install` and create an example worker:

*app/jobs/hard\_worker.rb*

Finally add the following line to your `PagesController#home` method (or anywhere else) to create a background job every time a request is made:

```ruby
HardWorker.perform_async
```

Now the interesting part: we need to add a new deployment to Kubernetes for running Sidekiq. The deployment is very similar to what we have already done for the web application: however, instead of running Puma as the main process of the container, we want to run Sidekiq. Here's the configuration:

*config/kube/sidekiq.yml*

Basically we define a new deployment with two pods: each pod runs our standard image that contains the Rails application. The most interesting part is that we set a `command` which overrides the default command defined in the Docker image. You can also pass some arguments to Sidekiq using an `args` key.

Also note that we define a `REDIS_URL` variable, so that Sidekiq and Rails can connect to Redis to get and process the jobs. You should also add the same env variable to your web deployment, so that your Rails application can connect to Redis and schedule the jobs. For Redis itself you can use Kubernetes *StatefulSets*, you can install it on a custom server or use a managed solution: although it is easy to manage a single instance of Redis, scaling a Redis cluster is not straightforward and if you need scalability and reliability probably you should consider a managed solution.

As always, you can apply the new configuration to Kubernetes with `kubectl apply -f config/kube`.

Finally you can try to visit your website and make sure that everything works: when you load your homepage, the example job is scheduled and you should see *It works!* in your logs.

## Cron jobs

There are different strategies to create a cron job when you deploy to Kubernetes:

* use Kubernetes built-in cron jobs to run a container periodically;
* use some Ruby background processes to schedule and perform the jobs.

A problem with the first approach is that you have to define a Kubernetes config file for each cron job. If you want to use this solution you can use *Kubernetes CronJobs* in combination with *rake tasks* or *rails runner*.

If you use the second method, you can schedule the cron jobs easily using Ruby. Basically you need a Ruby process that is always running in background and takes care to create the jobs when the current time matches a cron pattern. For example you can install the *rufus-scheduler* gem and then run a dedicated container: however in this case you have a single point of failure and if the pod is rescheduled a job may be lost. In order to have a more distributed and relaible environment, we can use a gem like *sidekiq-cron*: it runs a scheduler thread on each sidekiq server process and it uses Redis in order to make sure that the same job is not scheduled multiple times. For example, if you have N sidekiq replicas, then there are N processes that check the schedule every minute and if the current time matches a cron line, then they try to get a Redis lock: if a thread manages to get the lock, it means that it is responsible for scheduling the Sidekiq jobs for that time, otherwise it simply does nothing. Finally the Sidekiq jobs are executed normally, as the other background jobs, and thus can easily scale horizontally on the existing pods and get a reliable processing with retries.

Let's add this gem to the Rails application:

*Gemfile*

Then run `bundle install` and create an initializer:

*config/initializers/sidekiq.rb*

Finally define a schedule:

*config/schedule.yml*

Then when you start Sidekiq you will see that the worker is executed once every minute, regardless of the number of pods running.

## Console

You can connect to a pod by running this command:

```
kubectl exec -it my-pod-name bash
```

Basically we start the bash process inside the container and we attach our interactive input to it using the `-it` options.

If you need a list of the pod names you can use `kubectl get pods`.

Altough you can connect to any pod, I find it useful to create a single pod named `terminal` for maintenance tasks. Create the following file and then run `kubectl apply -f kube/config`:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: terminal
spec:
  containers:
  - name: terminal
    image: username/kubernetes-rails-example:latest
    command: ['sleep']
    args: ['infinity']
    env:
    - name: EXAMPLE
      value: This env variable is defined by Kubernetes.
```

All containers must have a main running process, otherwise they exit and Kubernetes considers that as a crash. Running the default command for the image, the Rails server, would be a waste of resources, since the pod is not connected to the load balancer: instead we use `sleep infinity`, which is basically a no-op that consumes less resources and keeps the container running.

Once you are connected to the bash console of a pod you can easily run any command. If you want to run only a single command, you can also start it directly. For example, if you want to start the *Rails console* inside a pod named `terminal`, you can run this command:

```
kubectl exec -it terminal rails console
```

If you need to pass additional arguments to the process you can use `--` to separate the Kubernetes arguments from the command arguments. For example:

```
kubectl exec -it terminal -- rails console -e production
```

## Rake tasks

There are different ways to run a rake task on Kubernetes:

* you can create a Kubernetes Job to run the rake task in a dedicated container;
* you can connect to an existing pod and run the rake task.

For simplicity I prefer the second alternative.

Run the following command to list all the pods available:

```
kubectl get pods
```

Then you can run a task with this command:

```
kubectl exec my-pod-name rake task-name
```

Note that `kubectl exec` returns the status code of the command executed (i.e. `0` if the rake task is executed successfully).

## Database migrations

Migrating the database without downtime when you deploy a new version of your application is not a simple task. The origin of most problems is that both deploying the new code to all pods and running the database migration are long tasks that take some time to be completed. Basically they are not instant and atomic operations, and during that time you have at least one of the following situations:

* old code is running with the new database schema;
* new code is running with the old database schema.

Let's analyse in more detail some strategies:

* **Downtime**: If you could afford some downtime during migrations, than you would simply scale down your replicas to zero, run the migrations with a Kubernetes Job and then scale up your application again. **Pros**: simple, with no special requirements in your application code; no runtime errors during deployment due to different schemas. **Cons**: some minutes of downtime.
* **Deploy new code, then migrate**: You deploy the new code, updating the images on all pods, and then, when everything is finished, you run the migration. At first this seems to works perfectly if you can make your new code support the old schema (which is not always easy). However when the new database schema is applied, you still have to deal with ActiveRecord caching the old database schema in the Ruby process (thus making an additional restart required). If you choose this trategy, you can deploy your new code and then simply connect to one of the pods and run `rake db:migrate`. **Pros:** zero downtime; deployment is very simple. **Cons**: it is very difficult to make code backward compatible; you probably need an additional restart after the migration.
* **Migrate, then deploy new code**: This is the most common approach and it is used by Capistrano, Heroku and other CI/CD tools. The problem is that rolling out an update to many pods takes time and during that period you have the old code running with the new database schema. In order to avoid transient errors, you need to make the new schema backward compatible, so that it can run with both the old code and the new code: however it is not alwasy easy to write *zero downtime migrations* and there are many pitfalls. Also, in order to avoid additional problems, you should use a different Docker tag for each image (and not just the tag `latest`), otherwise an automatic reschedule may fetch the new image before the migration is complete. If you choose this strategy, you need to use a *Kubernetes Job* to deploy a single pod with the new image and run the migration and then, if the migration succeeds, update the image on all pods. **Pros**: common and reliable strategy. **Cons**: if you don't write backward compatible migrations, some errors may occur while the new code is being rolled out; you need to use different Docker tags for each image version if you want to prevent accidental situations where the new code runs before the migration.

If we choose the latest solution, we must define a job like the following:

*config/kube/migrate.yml*

Then you can run the migration using this command:

```
kubectl apply -f config/kube/migrate.yml
```

Then you can see the status of the migration:

```
kubectl describe job migrate
```

The above command also displays the name of the pod where the migration took place. You can then see the logs:

```
kubectl logs pod-name
```

When the job is completed you can delete it, so that you free the resources and you can run it again in the future:

```
kubectl delete job migrate
```

## Continuous delivery

In the previous sections we have configured the Kubernetes cluster and deployed the code manually. However it would be useful to have a simple command to deploy new versions of your app whenever you want.

Create the following file and make it executable (using `chmod +x deploy.sh`):

*deploy.sh*

Then you can easiliy release a new version with this command:

```
./deploy.sh
```

The above command executes the following steps:

1. use sh as the interpreter and set options to print each command and exit on failure;
2. build and publish the docker image;
3. run the migrations and then wait the completion of the job and delete it;
4. finally release the new code / image.

Also remember that if you change the Kubernetes configuration you need to run this command:

```
kubectl apply -f kube/config
```

## Monitoring

You need to monitor the Kubernetes cluster for various reasons, for example:

* understand the resource usage and scale the cluster accordingly;
* check if there are anomalies in the usage of resources, like pods that are using too many resources;
* check if all the pods are running properly or if there are some failures.

Usually the Kubernetes provider already offers a *dashboard* with many useful stats, like CPU usage, load average, memory usage, disk usage and bandwidth usage across all the nodes. Usually they collect the stats using a *DaemonSet*, in a way similar to what we have previously described for logging. If you prefer, you can also install custom monitoring agents on all nodes: you can use open source products like Prometheus or services like Datadog.

Other ways to monitor your application performance are:

* installing the Kubernetes *metrics-server* that stores the stats in memory and allows you to use commands like `kubectl top nodes/pods`;
* using stats from the load balancer;
* collecting synthetic metrics generated by ad-hoc requests sent from an external service to your application, for example in order to measure the response time from an external point of view;
* collecting stats directly from your Rails application using a gem for *application performance monitoring*, like Datadog APM or New Relic APM.

## Security updates

There is a misconception that you can forget about security updates when you use containers. That is not true. Even if Docker and containers add an additional layer of isolation, in particular from the host and from other containers, and they are also ephemeral, which is good for security, they still need to be updated in order to avoid application exploits. Note that an attack at the application layer it is also possible when there is a security bug at a lower layer, for example in the OS or inside libraries included in the base image.

If you run Rails on Kubernetes remeber to apply updates to the following layers:

* **Kubernetes and nodes**: most Kubernetes providers will apply the updates for you to the underlying nodes and to Kubernetes, so that you can forget about this layer. However you may need to enable an option for automatic updates: for example, if you use DigitalOcean, remember to go the Kubernetes settings from the dashboard and enable the automatic updates option.
* **Docker and containers**: you need to keep your containers updated. In particular make sure that you are using an updated version of the base image. If you use Ruby as the base image, use a tag like `2.5` instead of `2.5.1`, so that you don't forget to increase the patch version when there is a new patch available. However that is not enough: when a new OS patch is available, the Ruby maintainers release a new version of the image, with the same tag (for example the image with tag `2.5` is not always the same). This means that you should ckeck Docker Hub frequently to see if the base image has received some updates (or subscribe to the official security mailing lists for Ruby, Ubuntu, etc.): if there are new updates, build your image again and deploy.
* **Rails application and dependencies**: remember to update the versions of Ruby, Rails, Gems and Yarn packages used by your application, and any other dependencies.
* **Other**: you also need to update the database and other services outside Kubernetes. Usually it is useful to use managed databases so that your provider applies the security patches automatically and you can forget about this layer.

Basically, if you use managed services (for Kubernetes and database) and you deploy your application frequently, you don't need to do anything special: just keep your Rails app updated. However, if you don't release your application frequently, remember to rebuild the image and don't run your application for months on an outdated base image.

## Conclusion

We have covered all the aspects required for deploying a Ruby on Rails application in production using Kubernetes.

Scaling the application or updating the configuration across hundreds of nodes is now a simple operation that can be managed by a single DevOp. Thanks to the widespread support for Kubernetes you also get a better pricing and portability, compared to PaaS solutions like Heroku.

Remember that in order to achieve availability and world-scale scalability you need to avoid bottlenecks and single points of failure, in particular:

* **the load balancer**, when it is a simple server, may become a bottleneck; you can use better hardware if available, but finally you will have to use *Round-Robin DNS* to increase capacity, by distributing the clients over different load balancers and deployments; if you use a global network like CloudFlare, they can even perform health checks on your load balancers, protect them from DDoS attacks and cache most requests;
* **the database** hosted on a single server may become a bottleneck; you can use better hardware and *hot standby* servers, but finally you may have to move to a DBMS that supports *sharding*, meaning that the data is distributed automatically across different database instances, each one managing a range of keys; the database clients (e.g. inside your Rails app) first query a server in the database cluster to understand the current cluster configuration and then query the correct database instances directly, thus avoiding any kind of bottleneck; moreover each *shard* is usually replicated in order to preserve data in case of hardware failures, and thus it is called a *sharded replica*; strategies similar to what we have described are provided for example by *MongoDB* and *Redis Cluster* and there are many managed solutions available on the market.


# Awesome Products

### Writing

* [Notion](https://notion.so) - All-in-one workspace for writing, project management, and more.
* [Typora](https://www.typora.io/) - Awesome WYSIWYG Markdown editor.
* [GitBook](https://www.gitbook.com/) - Document Everything! For you, your users and your team (Offers bidirectional GitHub sync).

### Task Management

* [TickTick](https://ticktick.com) - The to do worklist to organize work & life.

### Blogging

* [Ghost](https://ghost.org) - Fiercely independent, professional publishing.

### Project Management

* [Trello](https://trello.com/) - Trello lets you work more collaboratively and get more done.
* [Linear](https://linear.app/) - The issue tracking tool you'll enjoy using
* [JIRA](https://www.atlassian.com/software/jira) - Issue & project tracking software.
* [Basecamp](https://basecamp.com/) - Project management and team communication software.
* [Unito](https://unito.io/) - Integrate and sync project management tools (Trello, JIRA, Asana, and more).

### Design

* [Figma](https://www.figma.com/) - Where teams design together.
* [Whimsical](https://whimsical.com/) - The Visual Workspace, communicate visually at the speed of thought.

### Sketching

* [Excalidraw](https://excalidraw.com/) - Hand-drawn look & feel

### Whiteboarding

* [Miro](https://miro.com/) - The online collaborative whiteboard platform to bring teams together, anytime, anywhere.

### Automation

* [Zapier](https://zapier.com/) - Connect Your Apps and Automate Workflows.
* [IFTTT](https://ifttt.com) - Every thing works better together.

### Authentication

* [Auth0](https://auth0.com/) - Never Compromise on Identity.

### Collecting Feedback

* [Canny](https://canny.io) - Track feedback to build better products.

### Internal Tools

* [Retool](https://retool.com/) - Build internal tools, remarkably fast

### Scheduling

* [Calendly](https://calendly.com) - Calendly helps you schedule meetings without the back-and-forth emails.
* [FreeBusy](http://freebusy.io/) The complete solution to automate meeting scheduling.
* [There](https://there.team/) - The base for collaborating clearly & smoothly in remote teams.

### Calls / Confrencing

* [Zoom](https://zoom.us) - Cloud video conferencing and simple online meetings.
* [Google Meet](https://apps.google.com/meet/?hl=en) - Premium video meetings for everyone.
* [Discord](https://discordapp.com) - All-in-one voice and text chat for gamers that's free, secure, and works on both your desktop and phone.

### Forms

* [Typeform](https://www.typeform.com) - The future of forms is here.
* [Google Forms](https://www.google.com/forms/about/) - Create effortless forms.

### Search

* [Algolia](https://www.algolia.com) - Fast, reliable and modern search and discovery.

### PaaS (Platform as a Service)

* [Heroku](https://heroku.com).
* [Render](https://render.com).
* [Google App Engine](https://cloud.google.com/appengine/).

### Communication APIs

* [Twilio](https://twilio.com) - Communication APIs for SMS, voice, video, and authentication.

### Email Clients

* [Spark](https://sparkmailapp.com/) - Love your email again.

### Password Managers

* [1Password](https://1password.com) - Go ahead. Forget your passwords.

### Email Services

* [Mailgun](https://www.mailgun.com/) - The Email Service for Developers.
* [SendGrid](https://sendgrid.com) - Partner with the email service trusted by developers and marketers for time-savings, scalability, and delivery expertise..

### Payment Gateways

* [Stripe](https://stripe.com) - The new standard in online payments.

### Continuous Integration

* [CircleCI](https://circleci.com/) - Automate your development process quickly, safely, and at scale..
* [GitHub Actions](https://github.blog/2019-08-08-github-actions-now-supports-ci-cd/) - Fast CI/CD for any OS, any language, and any cloud.
* [GitLab CI](https://about.gitlab.com/product/continuous-integration/) - GitLab CI/CD pipelines build, test, deploy, and monitor your code as part of a single, integrated workflow.

### JAMstack

*See:* [*https://jamstack.org*](https://jamstack.org)

#### Hosting

* [Netlify](https://netlify.com) - All-in-one platform for automating modern web projects.

#### Management / Building

* [Stackbit](https://www.stackbit.com/) - Build Modern JAMstack Websites in Minutes. Combine any Theme, Site Generator and CMS without complicated integrations.

#### CMS

* [NetlifyCMS](https://www.netlifycms.org/).
* [TinaCMS](https://tinacms.org/).
* [Contentful](https://www.contentful.com/).


# Engineering Management


# Overview

Learn what to expect and strive for as a developer transitioning into Engineering Management.

Based on [a post on my blog](https://omar.engineer/engineering-management-an-overview/).

I was an engineer for most of my career and one day I became a manager and I was quite lost and afraid in the beginning, I realized I hadn't shared my experiences from back then. I've been doing engineering management on and off for the past couple of years and I decided to share a short summary of what I learned.

A lot of this is stuff I learned from books, articles, websites, and some of this is stuff I learned on the job. Full credit to all the resources I learned from is at the bottom in the "Great resources" section, and I heavily recommend exploring them if this article captured your interest. I like to use the term "Maker" (engineer, designer, or anyone who makes stuff) and "Manager", both of which I learned from the book Leading Snowflakes. I wrote this from the perspective of someone who has been a Maker for most of his career and transitioned to Management a couple of years ago.

## What to expect

### You need to write, a lot

Meeting notes, peer feedback, feedback to you, documenting a task before you hand it to an engineer, and so on. Think of yourself as the standard for documentation in your organization or team, because you kind of represent that now. You'll get so much in your plate that it's impossible to remember it all, writing is your best friend.

### You'll talk to people and be in meetings a lot more

Management is all about people interactions, you'll be dealing with people a lot more than before and that means meetings. Too many meetings for your team is a productivity killer and that's also something you'll need to figure out how to cut down if it becomes an issue.

### Learn to measure your success by your team's success, not what you make yourself.

When I started being a leader I was used to measuring myself as an engineer or as a "maker" (measuring by what I make). When I first started being a manager I used to feel like I haven't accomplished anything in my day and this was difficult.

### You'll be exposed to a lot more conflicts

You'll see a lot of conflicts between people, sometimes over very small issues. A lot of your time will go towards handling them. This may not be the case for every manager but it was the hardest part of the job for me.

### Hard Choices

You're gonna have to let some people go, you're gonna have to step in to break up conflicts, and sometimes you'll have people who are trying their best but still hurting teammates. This isn't an easy path and it's full of people issues and conflicts. Even if you have a great hiring process, it won't be perfect and you'll hire the wrong person sometimes. There's no right or wrong answer in almost any situation, you need to have peers and mentors you can trust and ask for guidance. Don't make hard decisions alone, and definitely don't make them when you're not calm.

### You'll need to do things that many people don't want or aren't willing to do

Improving code standards, bug report structures, doing some things manually until they're automated, improving communication between your team and other departments, and so on. Your goal is to keep the team focused and removing obstacles from their way, and those obstacles are often easy/boring tasks that no one wants to do.

## What to strive to do

### 1:1 meetings

Regularly sit down with your team mates, connect with them, understand their goals and what they want to achieve and learn, understand what's troubling them. Put this on a schedule, it's extremely important to do this regularly. Write down the main points that go on in your 1:1 meetings, it's important for everything you need to do on this job.

### Mentorship

You need to help your teammates get better, sometimes in what they want to get better at, and sometimes in things you believe they need to learn for the sake of their team. This can be accomplished by pairing them with someone more experienced in an area they want to get better at, getting them access to websites like PluralSight, etc. Sometimes by mentoring them yourself. You have to give them goals, ask them to experiment with new things, help them write articles, give talks, and so on.

### Scheduling

You'll probably be in a lot of meetings and people need an easy way to sit down and talk with you. Have an easy way for them to book some of your time, Google Calendar, [Freebusy.io](https://freebusy.io/), [Calendly](https://calendly.com/), etc. are all great tools for this.

You also need to learn to switch between your Maker and Manager modes, it's good to regularly have this scheduled either some days of the week or some hours of certain days. You need time to hop on in and review some work (pull requests, designs, etc.) and contribute to them too.

### Keeping your "Maker" skills polished

You won't have as much time for "Making" as you used to before, and you might get rusty. Take the time to keep yourself up to date with what your team is doing, and of new developments in your field. You need to be able to understand at a general level what your team does so you can help them prioritize.

### Keep a list of tasks

Don't forget anything. Write things down. Anything anyone asks of you, jot it down on some app and slap a due date on it (I use [Todoist](https://todoist.com)). This list is also great for delegating. Got a complex task coming up that you'd like a teammate to have experience in? Delegate it. Got a lot of meetings today and your teammate asked for access to something or to try out a new tool, write down the task, don't forget them.

### Hiring

The first thing you need to look at when hiring is whether these candidates will fit in with your team or not. Sit down with your teammates and peers and decide what's a must-have and what's a nice-to-have for your workplace. Do you care about about writing well in your organization? Are you ok with people learning it on the job? Or is it a must-have to even join?

Avoid brain teaser interview questions, avoid theoretical scenarios and whiteboard questions that people will never have to do on the job, and make sure your hiring process involves the most real scenarios as possible. Get the candidate to work with your team on an actual task, and compensate them for it. [Microsoft's new process](https://blog.usejournal.com/rethinking-how-we-interview-in-microsofts-developer-division-8f404cfd075a) is my favorite way to hire, I highly recommend reading it. Not all organizations have the resources to do all this stuff and that's understandable but try your hardest to accomplish these two things when hiring:

1. Get the candidate to do some real work.
2. Have your teammates and the candidate interact when doing real work.

#### Hire juniors

Too often organizations only look for people who already know most of what they need and don't hire enough juniors. Juniors are great to have in any organization because they want to learn and there's a lot of them. A lot of the time after juniors grow and gain more skills within your organization, they'll start to get better offers from other places so some think this isn't a good investment, but I believe this is something organizations must try their hardest to do, especially in developing countries, because those juniors need to grow and they need help and training, and because you'll have more manpower to deal with the many small bugs that are often overlooked.

### Let your team grow / delegate

As a manager, you need to delegate. Jumping into Maker mode to get something done is something you were probably used to before but you now need to let your team be responsible for most of the "Making".

Learning by doing is the best way to learn. If there's an area you're an expert in as a Maker, don't immediately jump in to Maker mode and try to do it, let your teammates experience it and grow. Write them a short guide of what needs to be done and things to look out for and trust them to get it done.

### Encourage knowledge sharing

Ensure your teammates have an accessible place to find and share lessons learned, no matter how small they are (Something like [StackOverflow for teams](https://stackoverflow.com/teams), etc.). Ensure that nobody on your team is afraid or embarrassed to ask questions, regularly encourage your team to ask anything they can think of, and make sure your team is composed of people who are ok with that.

### Always be open for feedback and change

I like to have a GitHub repo in my organization for everything related to how I do management, how 1:1 meetings are run, what I strive to do as a manager, what teammates can expect from me, what I expect from them, and so on. I put it in a GitHub repo because engineers are generally used to working with GitHub and are used to changing things with pull requests, so I wanted to be like that. Make a pull request to change how your manager does something 😛

### Get your team to be able to function without you

This isn't to say that you're supposed to be useless or make yourself obsolete but that you should not be a blocker for your team. Imagine yourself going on a vacation for a month, what would go wrong? What tasks are you a blocker for? Will the person leading in your stead know what to do? Have you left them the necessary information? Does the team know what to work on? Do they know what to do or who to talk to if some disaster happens? Try going on short vacations every once in a while and see what goes wrong. At the beginning you'll most likely find that a lot of the things you handled as a "Maker" aren't being done anymore and you need to both make the team aware of them and delegate them.

## Great resources

### Books

[Leading Snowflakes](https://leadingsnowflakes.com/) - Full of practical advice on day-to-day things you need to do as a manger.

[The Manager's Path](https://www.amazon.com/dp/B06XP3GJ7F/ref=dp-kindle-redirect?_encoding=UTF8\&btkr=1) - A guide for tech managers going through increasing levels of management complexity.

[High Output Management](https://www.amazon.com/dp/B015VACHOK/ref=dp-kindle-redirect?_encoding=UTF8\&btkr=1) - Great book on running and scaling a company.

### Websites

[Manager Readme](https://managerreadme.com/) - A community of sharing your "manager guides" with other managers.

[The Watercooler](https://thewatercooler.io/) - A community for leaders looking to become better.

### Articles

[Rethinking how we interview in Microsoft's Developer Division](https://blog.usejournal.com/rethinking-how-we-interview-in-microsofts-developer-division-8f404cfd075a) - Great article on hiring processes.

[Undervalued Software Engineering Skills: Writing Well](https://blog.pragmaticengineer.com/on-writing-well/).

### Newsletters

[Software Lead Weekly](http://softwareleadweekly.com/) - A weekly email for busy people who care about people, culture and leadership.


# Software Architecture

## Links

* [Modular Application Architecture - Intro](https://www.goetas.com/blog/modular-application-architecture-intro/).


# Microservices

## Links

### Videos

* [10 Tips for failing badly at Microservices by David Schmitz](https://www.youtube.com/watch?v=X0tjziAQfNQ)

### Articles

* [Should that be a Microservice? Keep These Six Factors in Mind](https://content.pivotal.io/blog/should-that-be-a-microservice-keep-these-six-factors-in-mind).
* [Goodbye Microservices](https://segment.com/blog/goodbye-microservices/)
* [These are the most effective microservice testing strategies, according to the experts](https://medium.freecodecamp.org/these-are-the-most-effective-microservice-testing-strategies-according-to-the-experts-6fb584f2edde)


# Event sourcing

## Links

### Articles

* [Event Sourcing made Simple](https://kickstarter.engineering/event-sourcing-made-simple-4a2625113224)
* [Follow these practical principles to get well-designed microservices boundaries](https://medium.freecodecamp.org/follow-these-practical-principles-and-get-well-designed-microservices-boundaries-ef2deffd69e3)
* [Getting a Competitive Edge with a Microservices Based Architecture](https://auth0.com/blog/getting-a-competitive-edge-with-a-microservices-based-architecture/)
* [3 Secrets to Building Microservices | Atlassian](https://www.atlassian.com/continuous-delivery/microservices/building-microservices)
* [CQRS and Event Sourcing Intro For Developers - Software House Altkom Software & Consulting | ASC LAB](https://altkomsoftware.pl/en/blog/cqrs-event-sourcing/)
* [Why Event Sourcing basically requires CQRS and Read Models](https://blog.arkency.com/why-event-sourcing-basically-requires-cqrs-and-read-models/)
* [Developing Transactional Microservices Using Aggregates, Event Sourcing and CQRS - Part 2](https://www.infoq.com/articles/microservices-aggregates-events-cqrs-part-2-richardson)
* [Event sourced domain objects in less than 150 LOC](https://blog.arkency.com/2016/12/event-sourced-domain-objects-in-less-than-150-loc/)
* [Event Sourcing with Aggregates in Rust](https://medium.com/capital-one-tech/event-sourcing-with-aggregates-in-rust-4022af41cf67)

### Videos

* [Scaling Event Sourcing for Netflix Downloads](https://www.youtube.com/watch?v=rsSld8NycCU\&t=2670s)
* [Building a Real-Time Microservices Payments Platform - An Engineer's Dream!](https://www.brighttalk.com/webcast/679/311839/building-a-real-time-microservices-payments-platform-an-engineers-dream)

### Slides

* [Event Sourcing in practice](https://ookami86.github.io/event-sourcing-in-practice/)


# Serverless

## Links

[Organizing Serverless Projects](https://serverless-stack.com/chapters/organizing-serverless-projects.html).


# Centralized Authentication

## Links

* [How to Implement a Secure Central Authentication Service in Six Steps](https://engineering.shopify.com/blogs/engineering/implement-secure-central-authentication-service-six-steps)


# Talks


# Software Architecture


# An Insider's Look at the Technology That Powers Shopify

{% embed url="<https://www.youtube.com/watch?v=Th7XN__ltyc>" %}

[Talk link](https://www.youtube.com/watch?v=Th7XN__ltyc).

## Quick Summary

Watched this talk because I thought it was more software architecture related but it turned out to be a very general and high level overview on 4 areas of Shopify's technology. Lots of cool Silk Road. economical, and historical references though.

## Open Internet

**Payments**

* Joined W3C in 2016 as a founding member of the [payments working group](https://www.w3.org/Payments/WG/), to standardize payment actions on the internet for 3 years+.
  * Principles:
    * "Do you have a wallet?"
    * "Can you make a payment?"
    * "Let's make it fast."

**3D Models**

* Shopify is working with the [Khronos Group](https://khronos.org) (an open standards body) to advance 3D model formats for the internet.

## Security & Privacy

* A merchant's data belongs to the merchant.
* "We have the tools to support you, but you have to use them".
* The biggest threat to a merchant's data is compromised credentials, mainly from phishing emails.
* New identity vault launched: <https://accounts.shopfiy.com>
  * One account and one setting for everything you do on Shopify.
  * Supports new [Web Authentication API](https://www.w3.org/TR/webauthn/) (WebAuthn, [nice guide here](https://webauthn.guide/)).
    * Allows Shopify servers to interact with secure identity meechanisms on devices (TouchID, Windows Hello, etc.)
* Additional support on data ownership and privacy through new webhooks and APIs&#x20;
  * Data request and deletion API.
  * Greater app permission transparency.
* Ranking and monitoring apps on Shopify platform to make sure that their partners are building applications that respect data on behalf of the merchant.
* Processing more than 10 billion events everyday, which totals to 10 petabytes of data.
* Network of 3000+ security engineers who have been paid out more than 1 million USD to find and report vulnerabilities (on test shops).
* Increasing investment on [metafields](https://help.shopify.com/en/manual/products/metafields) on Shopify platform.

## Extensibility

* Providing unique products and experience is a key product to make commerce exciting and enjoyable.
* Guided by the philosophy of The Silk Road.
* 5 years ago Shopify API had lag between announcing APIs and features.
* As of this talk, Shopify is powered by the same REST and GraphQL APIs used by clients. No lag between feature announcements and API announcements anymore.
* New PoS is being built with apps as a core feature built into the experience.
* APIs are features.
* Comitting to building as many APIs they can to keep Shopify the most creative platform for commerce experiences.

## Global Infrastructure

* Scaled from 2 points of presence to 180 in over 80 countries in a single year. Decreasing latency for buyers by 30%  - 50%.
* Launching 4 new regions in next 18 months from this talk's date.
* Requests reach around 5 million per minute.
* Some merchants are selling more than 8000 orders a minute.
* Speed matters more than scale for most merchants.
* Two big performance wins:
  * Upgrade to image delivery service by using WebP (30% smaller and faster).
  * New Liquid renderer that's 7x faster.


# Building Extensible Platforms

{% embed url="<https://www.youtube.com/watch?v=GqGNA8GnOOE>" %}

[Talk link](https://www.youtube.com/watch?v=GqGNA8GnOOE).

## Platforms

> "Platforms provide a layer of abstraction to simplify otherwise complex tasks."

In platforms, all functionalitywe care about as users is generally provided by apps. The difference between platforms and apps is that platforms don't necessarily provide functionality but they are an important foundation for apps.

The purpose of a platform is to simplify application development and manage shared resources and services and access them through APIs.

Examples of platforms:

* Operating systems.
* Web browsers (They're an application from the perspective of an OS, but they also provide platform capabilities).

## Extensibility

> "Designed to allow the addition of new capabilities and functionality"

Extensibility is core to Shopify product philosphy.

* Platform team works with teams involved in the entire customer journey (from business operations to selling in-person) towards making Shopify extensible as a whole.
* [Shopify Flow](https://apps.shopify.com/flow), a tool for merchants to build automated workflows based on their business operations. Works a lot like Zapier (triggers, conditions, and actions). Third party developers can also build extensions to build custom triggers, conditions, and actions. It works and feels like Shopify first party functionality.

> "The experience of third-party developers is just as important as end-users' experience."

**Other examples**

* [Todoist Chrome extension](https://chrome.google.com/webstore/detail/todoist-to-do-list-and-ta/jldhpllghnbhlbpcmnajkpdmadaolakh?hl=en). Uses Chrome to render an icon button and present an overlay to manage Todoist. It feels like a native part of the browser experience thanks to the extension capabilities offered by Google Chrome.
* [Slack apps](https://slack.com/apps). Extend Slack functionality using bots and slash commands. Use apps without distrupting flow of conversations.

## [App Extensions](https://help.shopify.com/en/api/embedded-apps/app-extensions)

> ""App Extensions" are a deep third-party integration that feels indistinguishable from native functionality."

*Example of using Shopify Flow to build new triggers and actions starts at 11:23.*

App extensions are the counterpart to APIs. When using APIs the app initiates the data exchange with your platform. When using app extensions allow your platform to initiate the data exchange aas required.

With app extensions, Shopify is in charge of pushing/pulling data to/from the extension, and the merchants never leave Shopify and have a consistent and seamless experience.

> "It is the combination of great user experience and great developer experience that make app extensions so powerful."

Merchants use Shopify Core (can be accessed through web, apps, smart text-based assistants), developers use the Partners Dashboard.

> "You might ask: "Why provide a graphical interface (to programmers) instead of a programmatic one?"
>
> Our goal is to create a developer experience that doesn't require reading any API documentation, we want to promote exploration and allow developers to learn about the capabilities of our platform interactively."

## App Extensions From Internal Teams' Perspective

> "Treat your internal teams as your customers too and provide them with the same quality tools you provide yor external customers"

To reduce friction they built an internal framework that manages all persistence and communication between their core product and partner's dashboard. This significantly reduced the number of steps involved in building app extensions to only 3 steps:

* Domain model.
* Merchant facing UI.
* Partner facing UI.

They also provide a wealth of UX resources to make it easier for teams to design app extensions.

Good internal documentation (education) was a must to enable this process. Broadcasting information on a regular basis is just as important as being around to answer questions. They maintained a Slack channel to support teams during the initial phases of the project.

## How to Build Extensibility

> "Make it easy to think about 'What to make extensible' rather than 'how to build extensibility'"
>
> "What parts of a user workflow should we enable our ecosystem to plug into?
>
> Which parts of our user experience should we make adaptable to whatever our ecosystem might build?"

Shopify tackles these problems by:

1. Building features that **most** of their users need (the 80/20 rule, 80% is a core commerce need, 20% is divergent needs. Shopify enables their ecosystem to build for their niche divergent needs).
2. Enable their partner ecosystem to build the rest.

> "Company-wide platform commitment has an incredible network effect (when a product or service gains more value as more people use it). Merchants are drawn to Shopify because of the ways Shopify and Shopify apps solve their end to end commerce needs. At the same time partners are drawn to our platform for our growing user base and our growing business needs and growing opportunities for innovation and entrepreneurship."


# Documentation

**Links**

* [How to Write Good Documentation (And Its Essential Elements)](https://www.sohamkamani.com/blog/how-to-write-good-documentation/).


# Machine Learning


# Terminology

* **Feature** - An input variable used to infer things or make predictions.
* **Example** - One "row" (or entry) in a data set containing features and possibly labels.
* **Label (only for supervised learning)** - An identifier for what an example is. If you're training a machine to detect if a picture contains a cat or not, the label would be the correct answer that you want the machine to be able to answer.&#x20;


# Regression

Loss metrics to optimize for:

*Lower values indicate a better performing model.*

* Mean Squared Error (How close a regression line is to the set of points from it).
* Root Mean Squared Error (to get back to the same units used in the examples).


# Overfitting

Overfitting is when a machine learning model is fitting a data set too exactly and will cause it to fail when new data is introduced to it. In this case, the model fails (or simply has a high error rate) to deal with anything outside of the data set it was trained on.

tl;dr It's as if the model memorized the data set and knows nothing else outside of it.

Possible solutions:

Split the data set into one used for training, and one used for validation. If the model generalizes well, it should have similar loss metrics across the training and validation data sets.


# Programming Languages


# Constructs

### Links

* [Flow-Charts of Programming Language Constructs](https://www.progsbase.com/blog/flow-charts-of-programming-language-constructs/)


# Go


# Syntax

### Entry Point

You should always have a `main` function as your entry point.

```go
package main

func main() {
}

```

### Type Declarations

Type declarations always go to the right of variable or function names, e.g.

```go
var i int = 1
```

*This isn't necessary in this case as it can be inferred from the right hand side*

#### Type Inference

Go has the `:=` operator which can infer the type and assign it to a variable at the same time. Using it we can write the same example as above and omit the `var` and `int`.

```go
i := 5
```

### Function Declarations

Function declarations are actually quite similar (the return type is to the right of the function name), here's a function that just returns a string when called.

```go
func returnHello() string {
	return "Hello"
}
```

Here's the same version but this time it takes in a name as well

```go
package main

import "fmt"

func main() {
	fmt.Println(returnHello("Omar"))
}

func returnHello(name string) string {
	return fmt.Sprintf("Hello %s", name)
}

```

### Conditionals

Conditionals are quire straightforward in Go, they're similar to most languages but without the parentheses

```go
package main

import "fmt"

func main() {
	name := "Omar"

	if name == "Omar" {
		fmt.Println("It's me")
	} else {
		fmt.Println("Hey there stranger!")
	}
}
```

### Loops

There's only one type of loop in Go, a `for` loop.

Here's a loop that prints "Hello" five times.

```go
package main

import "fmt"

func main() {
	for i := 0; i < 5; i++ {
		fmt.Println("Hello")
	}
}
```

Here's an infinite loop (useful for servers, game engines, etc.)

**Warning:** You have to kill the program to end this

```go
package main

import "fmt"

func main() {
	for {
		fmt.Println("Hello")
	}
}
```

This form can also be used to make a finite loop when combined with the `break` keyword, this will print 0 to 4.

```go
package main

import "fmt"

func main() {
	i := 0

	for {
		if i >= 5 {
			break
		}

		fmt.Println(i)
		i++
	}
}
```

There's also a form of the loop that you only need to provide with the condition, this will print 0 to 5. (This is used to **stop it before the next run**)

```go
package main

import "fmt"

func main() {
	i := 0
	isLessThanFive := true

	for isLessThanFive {
		if i >= 5 {
			isLessThanFive = false
		}

		fmt.Println(i)
		i++
	}
}
```

### Structs

Structs are a useful way to encapsulate and reuse objects, and they can even have methods. They're similar in some ways to classes in other programming languages.

Here's how to declare a struct (can be done outside of `main`)

```go
package main

import "fmt"

// Struct declaration
type Person struct {
	name string
	age  int
}

func main() {
    // Struct usage
	omar := Person{name: "Omar", age: 27}
	fmt.Println(omar)
}
```

You can add methods to structs by using [method receivers](https://spino.tech/blog/method-receiver-types-in-go/).

```go
package main

import "fmt"

type Person struct {
	name string
	age  int
}

func (p Person) description() string {
	return fmt.Sprintf("%s is a Person who is %d years old", p.name, p.age)
}

func main() {
	omar := Person{name: "Omar", age: 27}
	fmt.Println(omar.description())
}
```

This syntax was the most confusing to me whenever I saw Go code, the `(p Person)` part. This basically says that the type that can receive this method call is of type `Person` and that within that function we want to name that receiver `p`.

### Interfaces

Interfaces add a bit of genericness to Go, they're quite similar to other languages except that you don't need to specifically declare that your struct implements an interface, the compiler infers it if the signature of your methods match your interface method's signature.

```go
package main

import "fmt"

// Interface declaration
type Eater interface {
	eat() string
}

type Cat struct {
	name string
}

func (c Cat) eat() string {
	return fmt.Sprintf("%s likes to eat cat food.", c.name)
}

type Dog struct {
	name string
}

func (d Dog) eat() string {
	return fmt.Sprintf("%s likes to eat dog food.", d.name)
}

func printEatResult(e Eater) {
	fmt.Println(e.eat())
}

func main() {
	cat := Cat{name: "Felix"}
	dog := Dog{name: "Spark"}

	printEatResult(cat)
	printEatResult(dog)
}
```

This slightly lengthy example allowed us treat both `Cat` and `Dog` as `Eater`s in the `printEatResult` function. Notice how we didn't have to do anything to explicitly say that Cats and Dogs implement the Eater interface, the compiler just figured it out because the signature of the `eat` function on each of those structs matched the `Eater` 's version of it.

*It's a convention to name interfaces that have only one method (e.g. `eat`) to end with the name of that method plus `er`, so `eater`.*

### Arrays and Slices

#### Arrays

It's quite simple to declare and use an array (*the size of the array goes to the left of the type*).

```go
package main

func main() {
	var names [2]string
	names[0] = "Omar"
	names[1] = "John"
	fmt.Println(names)
}
```

However with arrays you must always declare the size, if you want to use something with flexible size, that's what slices are for.

#### Slices

Slices are a type built on top of arrays to provide more convenience and flexibility, most array programming in Go is done using slices.

```go
package main

func main() {
	var names []string
	names = append(names, "Omar")
	names = append(names, "John")
}
```

*In the example above, the size is omitted (you don't need to know the size of your data when declaring a slice)*


# Ruby

At this point I think I've used Ruby more than any other language in my life. I love how I can write clean, concise, and readable code in Ruby without having to think much about it. I really love the testing tools and community that Ruby has. Ruby has also greatly influenced two languages I really love:

* [Elixir](https://elixir-lang.org/) - Functional programming language heavily inspired by Ruby and runs on the \[Erlang VM]\(<https://www.wikiwand.com/en/BEAM_(Erlang_virtual_machine>)) which makes it easy to write concurrent/parallel code..
* [Crystal](https://crystal-lang.org/) - A compiled and very performant language that's statically typed (with inference) that's heavily inspired by Ruby syntax. A lot of the Ruby code I've written just works in Crystal.


# Ruby on Rails

I've been using Ruby on Rails for 5 years+ and I really love how productive you can be in it. It's the best of Ruby and web development.


# Setup

I like to use Rails with postgres a lot, this is how to make a new Rails project with Postgres as the default database adapter:

```
rails new myapp --database=postgresql
```

In some cases I use Rails without ActiveRecord (e.g. [mongoid](https://github.com/mongodb/mongoid) for [MongoDB](https://www.mongodb.com/), like so:

```
rails new myapp --skip-active-record
```


# Gotchas

## ActiveRecord

* `find` raises an exception that if not handled, will result in controller endpoints returning 404.
* `find_by!` is a variant that can be used to get the same behavior of `find` for `find_by`.

## database.yml

If you try to have a generic database configuration controlled by environment variables, your database actions will run twice in development.

e.g.

```yaml
default: &default
  adapter: postgresql
  encoding: unicode
  pool: <%= ENV['DATABASE_POOL'] %>
  host: <%= ENV['DATABASE_HOST'] %>
  database: <%= ENV['DATABASE_NAME'] %>
  username: <%= ENV['DATABASE_USER'] %>
  password: <%= ENV['DATABASE_PASSWORD'] %>

development:
  <<: *default

test:
  <<: *default
```

Since Rails by default runs migrations for the `test` environment when you're on `development` , when it tries to run them the `RAILS_ENV` will be `development` so it will actually run all the database actions on the development database again.

Instead ensure that you give specific and unique names for the development and test environments:

```yaml
development:
  <<: *default
  database: my_app_development

test:
  <<: *default
  database: my_app_test
```


# Helpers

## Readability

* [annotate\_models](https://github.com/ctran/annotate_models) - Automatically generates attributes as comments at the top of files to see what attributes are in models, fixture files, specs, factories, and more. Example:

  ```ruby
  # == Schema Info
  #
  # Table name: line_items
  #
  #  id                  :integer(11)    not null, primary key
  #  quantity            :integer(11)    not null
  #  product_id          :integer(11)    not null
  #  unit_price          :float
  #  order_id            :integer(11)
  #

   class LineItem < ActiveRecord::Base
     belongs_to :product
  # ...
  ```

## Migrations

* [strong\_migrations](https://github.com/ankane/strong_migrations) - Detect potentially dangerous migrations and prevent them from running by default, along with instructions on safer ways to do what you want.
* [native\_enum](https://github.com/iangreenleaf/native_enum) and [activerecord-postgres\_enum](https://github.com/bibendi/activerecord-postgres_enum) for backing Rails enums (which are application-only) by database enums, so people looking at the database can actually understand what all those integer values actually mean.

## View Helpers

Rails already provides a ton of view helpers (e.g. `number_to_currency`), they're listed [here](https://guides.rubyonrails.org/action_view_overview.html#overview-of-helpers-provided-by-action-view).


# Libraries

## Authentication

* [Devise](https://github.com/plataformatec/devise) - Flexible and modular authentication solution.
* [OmniAuth](https://github.com/omniauth/omniauth) - Flexible authentication solution for multi-provider auth (e.g. Sign in with GitHub or Google).

## Authorization

* [Pundit](https://github.com/varvet/pundit) - Minimal authorization through OO design and pure Ruby classes.

## ActiveJob

* [job-iteration](https://github.com/Shopify/job-iteration) - Make background jobs interruptible and resumable by design.

## Countries

* [countries](https://github.com/hexorx/countries) - All sorts of useful information about every country packaged as convenient little country objects. It includes data from ISO 3166 (countries and states/subdivisions ), ISO 4217 (currency), and E.164 (phone numbers).

## Payments

* [ActiveMerchant](https://github.com/activemerchant/active_merchant) - Payment abstraction library extracted from Shopify.

## Phones

* [phony\_rails](https://github.com/joost/phony_rails) - This Gem adds useful methods to your Rails app to validate, display and save phone numbers. It uses the super awesome Phony gem ([floere/phony](https://github.com/floere/phony)).

## Event-Driven Architecture

* [rails\_event\_store](https://github.com/RailsEventStore/rails_event_store) - A Ruby implementation of an Event Store based on Active Record.
* [sequent](https://github.com/zilverline/sequent) - CQRS & event sourcing framework for Ruby.
* [Eventide](https://eventide-project.org/) - Microservices, Autonomous Services, Service-Oriented Architecture, and Event Sourcing Toolkit for Ruby with Support for Event Store and Postgres.

## Enums

* [Enumerize](https://github.com/brainspec/enumerize) - Enumerated attributes with I18n and ActiveRecord/Mongoid/MongoMapper/Sequel support.

## Database

* [paranoia](https://github.com/rubysherpas/paranoia) - Overriding ActiveRecord `delete` and `destroy` to act as soft deletion.
* [scenic](https://github.com/scenic-views/scenic) - Versioned database views for Rails.
  * See also ["Effectively Using Materialized Views in Ruby on Rails"](https://pganalyze.com/blog/materialized-views-ruby-rails).

## Auditing

* [audited](https://github.com/collectiveidea/audited) - An ORM extension that logs all changes to your Rails models.
* [paper\_trail](https://github.com/paper-trail-gem/paper_trail) - Track changes to your models, for auditing or versioning. See how a model looked at any stage in its lifecycle, revert it to any version, or restore it after it has been destroyed.

## Localization

* [globalize](https://github.com/globalize/globalize) - Rails I18n de-facto standard library for ActiveRecord model/data translation.

## Forms

* [simple\_form](https://github.com/plataformatec/simple_form) - Forms made easy for Rails! It's tied to a simple DSL, with no opinion on markup.&#x20;

## Views

* [view\_component](https://github.com/github/view_component) - A framework for building reusable, testable & encapsulated view components in Ruby on Rails.

## PDF

* [wicked\_pdf](https://github.com/mileszs/wicked_pdf) - PDF generator (from HTML) plugin for Ruby on Rails.

## Monitoring

* [Bullet](https://github.com/flyerhzm/bullet) - Helps to kill N+1 queries and unused eager loading.

## Testing

* [shoulda-matchers](https://github.com/thoughtbot/shoulda-matchers) - Simple one-liner tests for common Rails functionality.
* [letter\_opener](https://github.com/ryanb/letter_opener) - Preview mail in the browser instead of sending.


# Routing

## Quick Cheatsheet

*Mostly taken from* [*devhints*](https://devhints.io/rails-routes)*.*

### Resources

```ruby
resources :books

# BooksController:
# index  =>    GET /books
# new    =>    GET /books/new
# create =>   POST /books/new
# show   =>    GET /books/:id
# edit   =>    GET /books/:id/edit
# update =>    PUT /books/:id
# delete => DELETE /books/:id
#
# Helpers:
# new_book_path
# book_path(id)
# edit_book_path(id)
```

### Member and Collection

`collection` is for routes on the collection.

`member` is for routes on a specific member.

```ruby
Rails.applications.routes.draw do
  resources :events do
    collection do
        post :validate # localhost:3000/events/validate
    end

    member do
      post :publish # localhost:3000/events/1/publish
    end
end
```

### Options

```ruby
resources :photos,
  path_names: { new: 'brand_new' }    # /photos/1/brand_new
  path: 'postings'                    # /postings
  only: :index
  only: [:index, :show]
  except: :show
  except: [:index, :show]

  shallow: true                       # also generate shallow routes
  shalow_path: 'secret'
  shallow_prefix: 'secret'
```

### Single Resource

```ruby
resource :coder

# CodersController:
# new    =>    GET /coder/new
# create =>   POST /coder/new
# show   =>    GET /coder
# edit   =>    GET /coder/edit
# update =>    PUT /coder
# delete => DELETE /coder
```

### Matching

```ruby
match 'photo/:id' => 'photos#show'  # /photo/what-is-it
match 'photo/:id', id: /[0-9]+/     # /photo/0192
match 'photo/:id' => 'photos#show', constraints: { id: /[0-9]+/ }
match 'photo/:id', via: :get
match 'photo/:id', via: [:get, :post]

match 'photo/*path' => 'photos#unknown'    # /photo/what/ever

# params[:format] == 'jpg'
match 'photos/:id' => 'photos#show', :defaults => { :format => 'jpg' }
```

### Redirect

```ruby
match '/stories' => redirect('/posts')
match '/stories/:name' => redirect('/posts/%{name}')
```

### Named Routes

```ruby
# logout_path
match 'exit' => 'sessions#destroy', as: :logout
```

### Scopes

```ruby
scope 'admin', constraints: { subdomain: 'admin' } do
  resources ...
end
```

## Nested Resources (routes)

Assuming an event has many registrations and we want registration routes to be nested under an event, e.g. `localhost:3000/events/1/registrations`, we can do:

```ruby
Rails.applications.routes.draw do
  resources :events do
    resources :registrations
  end
end
```

## Splitting Up Big Routes Files

*(Mostly taken from* [*Matt Boldt's blog post*](https://mattboldt.com/separate-rails-route-files/)*)*

[GitLab's route files](https://gitlab.com/gitlab-org/gitlab/tree/master/config/routes) are also a great example.

**First you have to make a new `draw` method into Rails's routing mapper via an initializer**

```ruby
# config/initializers/routing_draw.rb

# Adds draw method into Rails routing
# It allows us to keep routing splitted into files
class ActionDispatch::Routing::Mapper
  def draw(routes_name)
    instance_eval(File.read(Rails.root.join("config/routes/#{routes_name}.rb")))
  end
end
```

**Update your `config/routes.rb` with the names of files in `config/routes/*.rb`**

```ruby
# config/routes.rb
MyApp::Application.routes.draw do
  draw :api_v1
  draw :api_v2
  draw :admin
end
```

**New route files**

```ruby
# config/routes/api_v1.rb
namespace :api_v1 do
  # lots of routes
end

# config/routes/api_v2.rb
namespace :api_v2 do
  # lots of routes
end

# config/routes/admin.rb
namespace :admin do
  # lots of routes
end
```


# Status Code Symbols

*Taken from this* [*GitHub Gist*](https://gist.github.com/mlanett/a31c340b132ddefa9cca)

## 1xx Informational

100 `:continue`\
101 `:switching_protocols`\
102 `:processing`

## 2xx Success

200 `:ok`\
201 `:created`\
202 `:accepted`\
203 `:non_authoritative_information`\
204 `:no_content`\
205 `:reset_content`\
206 `:partial_content`\
207 `:multi_status`\
226 `:im_used`

## 3xx Redirection

300 `:multiple_choices`\
301 `:moved_permanently`\
302 `:found`\
303 `:see_other`\
304 `:not_modified`\
305 `:use_proxy`\
307 `:temporary_redirect`

## 4xx Client Error

400 `:bad_request`\
401 `:unauthorized`\
402 `:payment_required`\
403 `:forbidden`\
404 `:not_found`\
405 `:method_not_allowed`\
406 `:not_acceptable`\
407 `:proxy_authentication_required`\
408 `:request_timeout`\
409 `:conflict`\
410 `:gone`\
411 `:length_required`\
412 `:precondition_failed`\
413 `:request_entity_too_large`\
414 `:request_uri_too_long`\
415 `:unsupported_media_type`\
416 `:requested_range_not_satisfiable`\
417 `:expectation_failed`\
422 `:unprocessable_entity`\
423 `:locked`\
424 `:failed_dependency`\
426 `:upgrade_required`

## 5xx Server Error

500 `:internal_server_error`\
501 `:not_implemented`\
502 `:bad_gateway`\
503 `:service_unavailable`\
504 `:gateway_timeout`\
505 `:http_version_not_supported`\
507 `:insufficient_storage`\
510 `:not_extended`


# Debugging


# Byebug

## Byebug

## Links

* [Byebug Guide](https://github.com/deivid-rodriguez/byebug/blob/master/GUIDE.md)
* [Ruby on Rails - Debugging with the byebug gem](https://edgeguides.rubyonrails.org/debugging_rails_applications.html#debugging-with-the-byebug-gem)


# Cheatsheet

*Taken from* <https://fleeblewidget.co.uk/2014/05/byebug-cheatsheet/>

## Starting Byebug

If you’re running Byebug on a Rails application in development mode, you no longer need to start the server with `--debugger` – the debugger is on by default.

To get going, simply type `byebug` (or `debugger`) into your source file at the line you’re interested in and run the program. If you’re running it on a Rails application, remember to switch to your terminal window to look at debugger output.

**Note:** `byebug` invocations are just method calls, so you can make them conditional:

```ruby
byebug if foo == “bar”
```

**Another Note:** As is common with debuggers, hitting ‘Enter’ on an empty line in Byebug repeats the last command.

## Stopping Again

### q\[uit] — a.k.a. “exit” *unconditionally*

Quit. It stops the thing running. Also exits your program.Note:\*\* To quit without an ‘are you sure?’ prompt, use `quit unconditionally` (shortened to `q!`)

### kill

*Really* quit. This uses `kill -9`, for situations where quit just isn’t fierce enough.

## Essential Commands

### c\[ontinue] \<line\_number>

Carry on running until program ends, hits a breakpoint or reaches line *line\_number* (if specified).

### n\[ext] \<number>

Go to next line, stepping over function calls. If *number* specified, go forward that number of lines.

### s\[tep] \<number>

Go to next line, stepping into function calls. If *number* is specified, make that many steps.

### b\[ack]t\[race] — a.k.a. “w\[here]”

Display [stack trace](http://en.wikipedia.org/wiki/Stack_trace).

### h\[elp] \<command\_name>

Get help. With no arguments, returns a list of all the commands Byebug accepts. When passed the name of a command, gives help on using that command.

## Breakpoints and Catchpoints

### b\[reak]

Sets a [breakpoint](http://en.wikipedia.org/wiki/Breakpoint) at the current line. These can be conditional: `break if foo != bar`. Keep reading for more ways to set breakpoints!

### b\[reak] \<filename>:\<line\_number>

Puts a breakpoint at *line-number* in *filename* (or the current file if *filename* is blank). Again, can be conditional: `b myfile.rb:15 unless my_var.nil?`

### b\[reak] \<class>(.|#)\<method>

Puts a breakpoint at the start of the method *method* in class *class*. Accepts an optional condition: `b MyClass#my_method if my_boolean`

### info breakpoints

List all breakpoints, with status.

### cond\[ition] \<number> \<expression>

Add condition *expression* to breakpoint \<number<>>. If no *expression* is given, removes any conditions from that breakpoint.

### del\[ete] \<number>

Deletes breakpoint \<number>. With no arguments, deletes all breakpoints.

### disable breakpoints \<number>

Disable (but don’t delete) breakpoint \<number>. With no arguments, disables all breakpoints.

### cat\[ch] exception> off

Enable or (with *off* argument) disable catchpoint on \<exception>.

### cat\[ch]

Lists all catchpoints.

### cat\[ch] off

Deletes all catchpoints.

### sk\[ip]

Passes a caught exception back to the application, skipping the catchpoint.

## Program Stack

### b\[ack]t\[race] — a.k.a. “w\[here]”

Display [stack trace](http://en.wikipedia.org/wiki/Stack_trace).

### f\[rame] \<frame\_number>

Moves to \<frame\_number> (frame numbers are shown by `bt`). With no argument, shows the current frame.

### up \<number>

Move up \<number> frames (or 1, if no number specified).

### down \<number>

Move down \<number> frames (or 1, if no number specified).

### info args

Arguments of the current frame.

### info locals

Local variables in the current stack frame.

### info instance\_variables

Instance variables in the current stack frame.

### info global\_variables

Current global variables.

### info variables

Local and instance variables of the current frame.

### m\[ethod] \<class|module>

Shows instance methods of the given class or module.

### m\[ethod] i\[nstance] \<object>

Shows methods of \<object>.

### m\[ethod] iv \<object>

Shows instance variables of \<object>.

### v\[ar] cl\[ass]

Shows class variables of self.

### v\[ar] co\[nst] \<object>

Shows constants of \<object>.

### v\[ar] g\[lobal]

Shows global variables (same as `info global_variables`).

### v\[ar] i\[nstance] \<object>

Shows instance variables of \ (same as `method iv <object>`).

### v\[ar] l\[ocal]

Shows local variables (same as `info locals`).

## Execution Control

### c\[ontinue] \<line\_number>

Carry on running until program ends, hits a breakpoint or reaches line \<line\_number> (if specified).

### n\[ext] \<number>

Go to next line, stepping over function calls. If \<number> specified, go forward that number of lines.

### s\[tep] \<number>

Go to next line, stepping into function calls. If \<number> is specified, make that many steps.

### fin\[ish] \<num\_frames>

With no argument, run until the current frame returns. Otherwise, run until \<num\_frames> have returned.

### irb

Start an IRB session. This will have added commands `cont`, `n` and `step`, but these can’t take arguments (unlike the proper byebug commands of the same name).

### restart

Restart the program. This also restarts byebug.

## Threads

### th\[read]

Show current thread.

### th\[read] l\[ist]

List all threads.

### th\[read] stop \<number>

Stop thread number \<number>.

### th\[read] resume \<number>

Resume thread number \<number>.

### th\[read] \<number>

Switch context to thread \<number>.

## Display

### e\[val] — a.k.a. “p” \<expression>

Evaluate \<expression> and display result. By default, you can also just type the expression without any command and get the same thing (disabled by using `set noautoeval`).

### pp

Evaluate expression and pretty-print the result.

### putl

Evaluate an expression with an array result and columnize the output.

### ps

Evaluate an expression with an array result, sort and columnize the output.

### disp\[lay] \<expression>

Automatically display \<expression> every time the program halts. With no argument, lists the current display expressions.

### info display

List all current display expressions.

### undisp\[lay] \<number>

Remove display expression number \<number> (as listed by `info display`). With no argument, cancel all current display expressions.

### disable display \<number>

Stop displaying expression number \<number>. The display expression is kept in the list, though, and can be turned back on again using `enable display`.

### enable display \<number>

Re-enable previously disabled display expression \<number>.

## Controlling Byebug

### hist\[ory] \<num\_commands>

View last \<num\_commands> byebug commands (or all, if no argument given).

### save \<file>

Saves current byebug session options as a script file in \<file>.

### source \<file>

Loads byebug options from a script file at \<file>.

### set \<option>

Change value of byebug option \<option>.

### show \<option>

View current value of byebug option \<option>.

Options are:

* `autoeval`
* `autoirb`
* `autolist`
* `autoreload`
* `autosave`
* `basename`
* `callstyle`
* `forcestep`
* `fullpath`
* `histfile`
* `histsize`
* `linetrace`
* `tracing_plus`
* `listsize`
* `post_mortem`
* `stack_on_error`
* `testing`
* `verbose`
* `width`

## Source Files and Code

### reload

Reload source code.

### info file

Information about the current source file.

### info files

All currently loaded files.

### info line

Shows the current line number and filename.

### l\[ist]

Shows source code after the current point. Keep reading for more list options.

### l\[ist] –

Shows source code before the current point.

### l\[ist] =

Shows source code centred around the current point.

### l\[ist] \<first>-\<list>

Shows all source code from \<first> to \<last> line numbers.

### edit \<file:line\_no>

Edit \<file>. With no arguments, edits the current file.


# Libraries

Libraries that help you build and scale Ruby (and Rails) applications.

## Printing (in console)

* [awesome\_print](https://github.com/awesome-print/awesome_print) - Pretty print your Ruby objects with style -- in full color and with proper indentation.&#x20;

## Resiliency and Failure

* [Semian](https://github.com/Shopify/semian) - Resiliency toolkit for failing fast (e.g. Circuit breaker and bulk heading).

## Feature Flags

* [rollout](https://github.com/fetlife/rollout) - Fast feature flags based on Redis.

## Hash Enhancements / Helpers

* [Hashie](https://github.com/intridea/hashie) - Hashie is a collection of classes and mixins that make hashes more powerful.

## Background Jobs

* [sidekiq](https://github.com/mperham/sidekiq) - Simple, efficient background processing for Ruby
* [sidekiq-cron](https://github.com/ondrejbartas/sidekiq-cron) - Scheduler / Cron for Sidekiq jobs to easily make recurring scheduled jobs.
* [sidekiq\_alive](https://github.com/arturictus/sidekiq_alive) - Liveness probe for Sidekiq in Kubernetes deployments.
* [sidekiq-statistic](https://github.com/davydovanton/sidekiq-statistic) - Improved display of statistics for your sidekiq workers and jobs.

## Location

* [geocoder](https://github.com/alexreisner/geocoder) - Complete Ruby geocoding solution.

## Phones

* [phony](https://github.com/floere/phony) - E164 international phone number normalizing, splitting, formatting. .

## Money

* [money](https://github.com/RubyMoney/money) - A Ruby Library for dealing with money and currency conversion.

## Linting and Formatting

* [Standard](https://github.com/testdouble/standard) - Ruby style guide with linter & automatic code fixer, based on Rubocop.

## Pagination

* [pagy](https://github.com/ddnexus/pagy) - The ultimate pagination ruby gem, **much** faster than most pagination gems.

## Testing

* [test-prof](https://github.com/palkan/test-prof) - Ruby tests profiling toolbox (measurments to improve test suite speed).
* [vcr](https://github.com/vcr/vcr) - Record your test suite's HTTP interactions and replay them during future test runs for fast, deterministic, accurate tests.
* [parallel\_tests](https://github.com/grosser/parallel_tests) - Speedup Test::Unit + RSpec + Cucumber + Spinach by running parallel on multiple CPU cores.
* [database\_cleaner](https://github.com/DatabaseCleaner/database_cleaner) - Strategies for cleaning databases in Ruby. Can be used to ensure a clean state for testing.&#x20;
* [factory\_bot](https://github.com/thoughtbot/factory_bot) - A library for setting up Ruby objects as test data.
* [airborne](https://github.com/brooklynDev/airborne) - RSpec driven API testing framework.
* [knapsack](https://github.com/ArturT/knapsack) - Knapsack splits tests across CI nodes and makes sure that tests will run comparable time on each node.


# Lazy Enumerators

## Links

* [Ruby Lazy Enumerators](https://blog.saeloun.com/2019/10/23/ruby-lazy-enumerators.html)
* [Using lazy enumerators to work with large files in Ruby](https://www.honeybadger.io/blog/using-lazy-enumerators-to-work-with-large-files-in-ruby/)


# Snippets

A couple of snippets for things I have to do a lot but forget how every once in a while

## Using `.map` on a Hash

```ruby
{ a: 'a', b: 'b' }.map { |k, str| [k, "%#{str}%"] }.to_h
```

## Convert an Array of Hashes to a Hash

```ruby
# I have an array of hashes
array = [ {id: 1, name: "test"} ]

# I want to turn it to a hash, where the key is `id`
array.map { |h| [h[:id], h] }.to_h
#=> { 1 => {:id => 1, :name => "test"} }
```


# Version Control


# Git

**Links**

* [How to Write a Git Commit Message](https://chris.beams.io/posts/git-commit/).
* [How to discover a bug using git bisect](https://flaviocopes.com/git-bisect/).

## Hook managers

Git hook managers are very useful if you want to ensure code is formatted/tests are run before you can commit, and you can even prevent comitting if the fail.

* [overcommit](https://github.com/sds/overcommit) - A fully configurable and extendable Git hook manager.


# Conventions


# Conventional Commits

"[Conventional Commits](https://www.conventionalcommits.org/en)" is a specification for adding human and machine readable meaning to commit messages.

Example of a subject for a commit adding an ability to parse arrays to a JavaScript parser

`feat(javascript_parser): add ability to parse arrays`

## Why Use Conventional Commits?

* Automatically generating CHANGELOGs.
* Automatically determining a semantic version bump (based on the types of commits landed).
* Communicating the nature of changes to teammates, the public, and other stakeholders.
* Triggering build and publish processes.
* Making it easier for people to contribute to your projects, by allowing them to explore a more structured commit history.


# gitmoji

[Gitmoji](https://github.com/carloscuesta/gitmoji) make quickly visually identifying what a commit changed pretty fast and fun.

## List

### Bugs

* 🐛 - `:bug:` for fixing a bug.
* 🚑 - `ambulance` for a critical hotfix.

### Features

* ✨ - `:sparkles:` Introducing new features.

### Code Quality

* 🎨 - `:art:` Improving structure / format of code.
* ♻️ - `:recycle` Refactoring code.
* 👌 - `:ok_hand:` Updating code due to code review changes.
* 🏗 - `:building_construction:` Making architectural changes.
* 💥 - `:boom:` Introducing breaking changes.
* 🍻 - `:beers:` Writing code drunkenly.
* 💩 - `:poop:` Writing bad code that needs to be improved.

### Testing

* ✅ - `:white_check_mark:` Updating tests.
* 🤡 - `:clown_face:` Mocking things.

#### CI

* 👷 - `:construction_worker:` Adding CI build system.
* 💚 - `:green_heart:` Fixing CI build.

### Database

* 🗃 - `:card_file_box:` Performing database related changes.
* 🌱 - `:seedling:` Adding or updating seed files.

### Logs

* 🔊 - ​`:loud_sound:` Adding logs.
* 🔇 - ​`:mute:` Removing logs.

### UI/UX

* 📱 - `:iphone:` Working on responsive design.
* 🚸 - `:children_crossing:` Improving user experience / usability.
* ♿️ - `:wheelchair:` Improving accessibility.
* 💄 - `:lipstick:` Updating UI or style files.

### Types

* 🏷️ - `:label:` Adding or updating types (Flow, TypeScript).

### Feature flags

* 🚩 - `:triangular_flag_on_post:` Adding, updating, or removing feature flags.

### Documentation

* 💡 - ​`:bulb:` Documenting source code.
* 📝 - `:pencil:` Writing docs.

### Dependencies

* ➕ - `:heavy_plus_sign:` Adding a dependency.
* ➖ - `:heavy_minus_sign:` Removing a dependency.
* 📌 - `:pushpin:` Pinning dependencies to specific versions.
* ⬆️ - `:arrow_up` Upgrading dependencies.
* ⬇️ - `:arrow_down` Downgrading dependencies.

### Security

* 🔒 - `:lock:` Fixing security issues.

### Platform-specific

* 🍏 - `:green_apple:` Fixing something on iOS.
* 🤖 - `:robot:` Fixing something on Android.
* 🏁 - `:checkered_flag:` Fixing something on Windows.
* 🍎 - `:apple:` Fixing something on macOS.
* 🐧 - `:penguin:` Fixing something on Linux.
* 🐳 - `:whale:` Work about Docker.
* ☸️ - `:wheel_of_darma:` Work about Kubernetes.

### Git

* 🎉 - `:tada:` Initial commit.
* 🙈 - `:see_no_evil:` Adding or updating a .gitignore file.
* 👥 - `:busts_in_silhouette` Adding contributor(s).
* 🔀 - `:twisted_rightwards_arrows:` Merging branches.
* ⏪ - `:rewind:` Reverting changes.

### Files

* 🚚 - `:truck:` Moving or renaming files.
* 📦 - `:package:` Updating compiled files or packages.
* 🔥 - `:fire:` Removing code or files.

### Linters

* 🚨 - `:rotating_light:` Removing linter warnings.

### Releases / Tags

* 🔖 - `:bookmark:` Releasing / Version tags.
* 🚀 - `:rocket:` Deploying stuff.

### Configuration

* 🔧 - `:wrench:` Changing configuration files.

### Localization / Text

* 🌐 - `:globe_with_meridians:` Internationalization and localization.
* 💬 - `:speech_balloon:` Updating text and literals.
* ✏️ - `:pencil2:` Fixing typos.

### Licenses

* 📄 - `:page_facing_up` Adding or updating license.

### Misc.

* 🔍 - `:mag:` Improving SEO.
* ⚗ - `:alembic:` Experimenting new things.
* 📸 - `:camera_flash:` Adding or updating snapshots.
* 🥚 - `:egg:` Adding an easter egg.
* 🍱 - `:bento:`​ Adding or updating assets.
* 📈 - `:chart_with_upwards_trend:` Adding analytics or tracking code.
* 🚧 - `:construction:` Work in progress.
* ⚡️ - `:zap:` Improving performance.
* 👽 - `:alien:` Updating code due to external API changes.


# Education

I share links to courses, books, and in general "learning paths" for multiple topics here.

## Links

* [1,300 Free Online Courses from Top Universities](http://www.openculture.com/freeonlinecourses).


# Programming & Computer Science


# Courses

## Computer Science

### Specific Courses

#### [Computer Science Crash Course](https://www.youtube.com/watch?v=tpIctyqH29Q\&list=PL8dPuuaLjXtNlUrzyH5r6jN9ulIgZBpdo)

{% embed url="<https://www.youtube.com/watch?v=tpIctyqH29Q&list=PL8dPuuaLjXtNlUrzyH5r6jN9ulIgZBpdo>" %}

#### [CS50 By Harvard University](https://www.youtube.com/channel/UCcabW7890RKJzL968QWEykA)

{% embed url="<https://www.youtube.com/channel/UCcabW7890RKJzL968QWEykA>" %}

#### [Introduction to Computer Science by MIT](https://www.youtube.com/watch?v=nykOeWgQcHM\&list=PLUl4u3cNGP63WbdFxL8giv4yhgdMGaZNA)

{% embed url="<https://www.youtube.com/watch?v=nykOeWgQcHM&list=PLUl4u3cNGP63WbdFxL8giv4yhgdMGaZNA>" %}

## Programming

### Specific Courses

* [Udacity - School of Programming](https://www.udacity.com/school-of-programming)

## General Course Websites

* [CloudAcademy](https://cloudacademy.com) - Cloud computing and certifications.
* [Codecademy](https://www.codecademy.com/) - Interactive way to learn programming.
* [Coursera](http://coursera.com/) - General courses website, has degree-granting paths.
* [edX](https://www.edx.org/) - General courses website.
* [O'Reilly](https://oreilly.com) - Good courses and amazing books.
* [PluralSight](https://pluralsight.com) - Video courses for beginners and practitioners.
* [Team Treehouse](https://teamtreehouse.com/) - Courses website with a great community and great articles.
* [Udacity](https://udacity.com) - General courses website, grants nanodegrees.
* [Udemy](https://udemy.com) - General courses website.

## Credits

Resources are compiled from various users in these Twitter threads:

* [E7san](https://twitter.com/E7san/status/1164760773857431553)
* [sultan\_7raag](https://twitter.com/sultan_7raag/status/1164973226092564482)


# Design


# Courses and Books

## Refactoring UI

[Refactoring UI](https://refactoringui.com/) is everything you need to start producing better designs today.

> Learn how to design beautiful user interfaces by yourself using specific tactics explained from a developer's point-of-view.


# Frontend Development

## Links

* [Team Treehouse Frontend Web Development Track](https://teamtreehouse.com/tracks/front-end-web-development)
* [FreeCodeCamp Courses](https://www.freecodecamp.org/learn)
  * Responsive Web Design
  * JavaScript Algorithms and Data Structures
  * Front End Libraries


# HTTP


# Status Codes

See [Mozilla's MDN documentation](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) for a full list, I'm just going to cover the odd cases that have bitten me in the past here:

[HTTP 301](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/301) (Moved Permanently) and [302](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/302) (Found, or a temporary redirect) are supposed to leave the HTTP method unaltered (based on this [RFC from 2014](https://tools.ietf.org/html/rfc7231#section-6.4.2)), but many HTTP clients (including curl) do not respect that for historical reasons and change the method to a GET, as a result of that it's recommended to use [HTTP 308](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/308) (Permanent Redirect) and [307](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/307) (Temporary Redirect) if preserving the HTTP method is important.


# Design


# Icons


# Icon Sets

* [Font Awesome](https://fontawesome.com/).
* [Line Awesome](https://icons8.com/line-awesome).


# Arabic Content | محتوى عربي


# Learning


# مصادر لتعلم البرمجة و علوم الحاسب

## مواقع

{% embed url="<https://academy.hsoub.com/>" %}

{% embed url="<https://www.rwaq.org/>" %}

{% embed url="<https://harmash.com>" %}

{% embed url="<http://www.coursat.org/>" %}

{% embed url="<https://www.doroob.sa/ar/>" %}

{% embed url="<https://edlal.org/>" %}

{% embed url="<https://www.edraak.org/ar/>" %}

## المصادر

{% embed url="<https://twitter.com/Twi_Hajar/status/1165156126868086784>" %}


# Advice


# إدارة المطورين: نبذة

**تحت الترجمة من** [**هذه المقالة**](https://dev.to/obahareth/engineering-management-an-overview-1814)**.**

## **التوقّعات**

### **لازم تكتب كثير**

### **أغلب وقتك بيروح في محادثات و إجتماعات**

### **أتعلم تحسب نجاحك بأداء فريقك، مش بأدائك الشخصي**

### **حتواجه صراعات كثيرة**

### **قرارت صعبة**

### **حتضطر تاخذ قرارات صعبة**<br>

## **اسعى انك تعمل الحاجات هذه**

### **الاجتماعات الانفرادية**

### **الإرشاد**

### **الجدولة**

### **الصقل المستمر لمهاراتك كـ"صانع"**

### **كتابة و تحديث قائمة بجميع المهام**

### **التوظيف**

#### **توظيف ال juniors**

### **اسمح لفريقك ينمو \ تفويض المهام**

### **شجّع مشاركة المعرفة و العلم**

### **كون دائماً منفتح للملاحظات و التغيير**

### **خلي فريقك يقدر يشتغل بدونك**

## **مصادر رهيبة**

### **كتب**

* [**Leading Snowflakes**](https://leadingsnowflakes.com/)  **- مليء بنصائح عملية على المهام اليومية لمدير مطورين.**
* [**The Manager’s Path**](https://www.amazon.com/dp/B06XP3GJ7F/ref=dp-kindle-redirect?_encoding=UTF8\&btkr=1)  **- دليل لمراحل مختلفة من المسؤولية و التعقيد في إدارة المبرمجين و إدارة مدراء المبرمجين.**
* [**High Output Management**](https://www.amazon.com/dp/B015VACHOK/ref=dp-kindle-redirect?_encoding=UTF8\&btkr=1)  **- كتاب رهيب عن تشغيل و "تضخيم؟" الشركات التقنية.**

### **مواقع**

* [**Manager Readme**](https://managerreadme.com/)  **- مجتمع لمشاركة "دليلك للإدارة" مع مدراء آخرين**
* [**The Watercooler**](https://thewatercooler.io/) **- مجتمع لمدراء يسعوا انهم يسيروا أفضل.**

### **مقالات**

* [**Rethinking how we interview in Microsoft’s Developer Division**](https://blog.usejournal.com/rethinking-how-we-interview-in-microsofts-developer-division-8f404cfd075a) **- مقالة عظيمة عن إجراءات مايكروسوف الجديدة للتوظيف.**
* [**Undervalued Software Engineering Skills: Writing Well**](https://blog.pragmaticengineer.com/on-writing-well/) **- مقالة عن أهمية الكتابة الجيدة في مجال التطوير.**

### **نشرات بريدية (إيميل)**

* [**Software Lead Weekly**](http://softwareleadweekly.com/) **- ايميل أسبوعي للمهتمين في ما يخص الناس, الثقافة, و الإدارة؟**


# نصائح  تصحيح معتقدات خاطئة  أمنيات للمبرمجين الجدد

*المحتوى الأصلي من* [*هذه التغريدة*](https://mobile.twitter.com/o_bahareth/status/1115699605197348864)*, شكر كثير* [*لإحسان*](https://e7san.gitbook.io/wiki/) *للمساعدة القيّمة في إعداد المحتوى.*

## *المعتقدات الخاطئة*

### أغلب وقتك تكتب كود

أنا دائما أقول ان المبرمج يقرأ أكثر مما يكتب، بتقرأ كود زملائك و بترجع في ال commit history عشان تفهم الكود (نية الكاتب)، عشان تفهم ايش سبب مشكلة معينة, او عشان تعرف كيف تضيف عليه.

### نبدأ بكتابة الكود مباشرةً

لأ, كتابة الكود بدون خطة من أكثر أسباب المشاكل، لازم تكون فهمت المشكلة اللي تبغى تحلها مضبوط و عملت خوارزم مبدأي لحلها قبل الكتابة. مرات التجربة و اللعب بالكود يساعد في ابتكار الخوارزم، لكن يفضّل مراجعته او إعادة كتابته بعدها. ما في احسن من انك تبدأ بورقة و قلم و تحط لك أمثلة عن المدخلات و تجرّب أساليب مختلفة لحلها, و ارجع جربها على الكود بعد الكتابة.

### المبرمج انطوائي أو يعمل وحده

من أكثر المعتقدات الخاطئة شيوعًا. الصحيح إن المبرمج يحتاج مهارات تواصل كتابية و منطوقة قوية، لإيصال أفكاره لبقية المبرمجين والعمل معهم (pair-programming)، و لكتابة توثيق و ملاحظات واضحة على الكود، و لطرح الأسئلة بشكل سليم على فريق المشروع/المنتج.

### لغة برمجة س أفضل من ص

هذا أسلوب تفكير غير جيد للمبرمج، عادي يكون عندك لغات مفضلة، لكن لا تقفّل عقلك عن باقي اللغات، اللغات هي فقط أدوات في عُدّة ادواتك. لازم تعرف مضبوط نوع المشاكل اللي لغة س أو ص مصممة لحلها.

### تقدر تستخدم أي لغة برمجة

هذا ليس معتقد خاطئ, لكن البعض يستخدم لغة مش مخصصة للنطاق اللي يبرمج فيه و ياخذ وقت كثير و مجهود كبير (عادي اذا يريحك). ممكن ابرمج حاجات data science بلغة Crystal لكن لأن مالها مكتبات قوية في هذا المجال, أسهل لي استخدم Python او R عشان اوفّر وقتي.

### التعقيد

&#x20;البعض يعتقد إن الكود المعقد او الكثير شيء جيد، لكن الأهم في البرمجة هو سهولة قراءة و فهم الكود و بالتالي سهولة قابلية التعديل عليه.

ركز على اختصار وقت و مجهود فهم الكود عشان تفيد زملاؤك اللي بيقروا الكود بعد كم شهر او سنة. البساطة بتزيد إنتاجيتك انت و كل من يعمل معك.

### المبرمج لازم يحفظ كثير

المبرمج لازم يكون عنده فهم قوي جدا للأساسيات, ما يحتاج يحفظ قد كدا. دائما بيكون معه مرجع (كتاب, توثيق, مقالة, محرك بحث, الخ) و أدوات في محرر النصوص تساعده يوصل للشيء اللي يحتاجه, الإعتماد على الحفظ قليل.

## حاجات انصح بتعلمها لدخول سوق العمل للمبرمجين و بعض النصائح الاخرى

### اللغة الانجليزية

ممكن تكتب كود حتى إذا ما تعرف انجليزي، لكن اذا هدفك تعمل كمبرمج او تكون ضمن فريق او تستخدم أو تساهم في مكتبات مفتوحة المصدر، لازم تصقل لغتك الانجليزية باستمرار.

### كيفية استخدام محركات البحث

المبرمج يحتاج يعرف كيف يبحث مضبوط, لازم يعرف المصطلحات التقنية بشكل جيد و يعرف امكانيات محركات البحث اللي يستخدمها.

{% embed url="<https://moz.com/learn/seo/search-operators>" %}

أول ما نزل فلم Django Unchained, ساروا الكثير من مبرمجين Django يتعبوا بسبب طغيان نتائج بحث الفلم على إطار العمل, كنت أبحث بهذه الطريقة عشان احصل على نتائج خاصة بإطار العمل `django -unchained ~programming`

### Source Control

من اهم الحاجات اللي اغلب الجامعات ما تغطّيها. باختصار، بيساعدكم تحافظوا على تاريخ التغييرات على الكود و على العمل جماعياً على الكود، و هو أساس المشاركة في المكتبات مفتوحة المصدر. أشهر تقنية هي Git و اشهر استضافة هي GitHub.

### فهم كيف الكود يتفاعل مع مكونات الأجهزة

كيف الكود يتفاعل مع الذاكرة و المعالج, كيف تشغل الكود حقك على اكثر من نواة (core) في المعالج, كيف تحدد متى تزيد أو تقلل استخدام أي واحد من المكونات هذي. كيف تلاقي و تحل مشاكل اداء في مكون معين.

### فهم في مجالك الدقيق

اذا انت frontend developer لازم تسعى انك تفهم كيف ال web browser يشغل كودك و كامل امكانياته و حدوده. اذا انت mobile developer لازم تفهم كيف تكتب كود يحافظ على البطارية وما يعلّق الجوال. فكّر في مستخدمينك و انت تبرمج.

### تعلّم مهارات حل المشاكل

الكود في النهاية أداة تستخدمها لحلّ مشكلة، إيجاد هذا الحل يحتاج لمهارات تفكير وتحليل واستنتاج قبل مهارات كتابة الكود، إذا ما تقدر تحل المشكلة بدون كتابة كود حتتعب في محاولة كتابة كود لأوامر انت أصلاً ما تعرفها.

#### بعض المواقع الجيدة اللي ممكن تتدرب فيها على حل المشاكل

{% embed url="<https://www.codewars.com>" %}

{% embed url="<https://leetcode.com/>" %}

{% embed url="<https://www.hackerrank.com/>" %}

## حاجات أتمنى المبرمجين الجدد يتعلموها\يعملوها

### المشاركة

&#x20;في اجابة الناس و المقالات و في المساهمة في المكتبات مفتوحة المصدر. احنا نستهلك مقالات و مكتبات و StackOverflow بشكل خرافي لكن ما نساهم فيهم كفاية. شاركوا حلولكم و الحاجات اللي اتعلمتوها بأي طريقة, انفعوا مثل ما انتفعتوا.

### إختبار الكود

اختبار الكود عن طريق كود. شيء ضروري انك تقدر تعرف اتوماتيكياً اذا تغيير على الكود سار يطلّع نتائج غير مرغوبة و انه عندك طريقة سهلة تختبره مع مدخلات مرغوبة و غير مرغوبة.

### المراقبة

لازم يكون عندكم ادوات تراقب ايش بيحصل مع تطبيقاتكم (سواء هي لعبة, ويب, او جوال). لازم تعرفوا ايش بيسير مع المستخدمين بدون ما يحتاجوا يكلموكم. أمثلة لبعض الأدوات (للمراقبة من ناحية تقنية):

{% embed url="<https://sentry.io>" %}

{% embed url="<https://newrelic.com>" %}

### قراءة الكتب

البعض يستهين بأهمية الكتب ويعتقد إنها مجرد حشو زائد ويكتفي بالمقالات والفيديوهات، أو يخاف منها خصوصًا الكتب الانجليزية، الكتب تقدّم لك معلومات تفصيلية بشكل أحيانًا ما تلاقيه في المقالات القصيرة أو حتى الكورسات الّي غالبًا موجهة للمبتدئين.


# Biology


# Species


# Ants

### Links

* [Here's Why Ants Are Practically Immune to Traffic Jams, Even on Crowded Roads](https://www.sciencealert.com/ant-roads-are-practically-immune-to-traffic-jams-even-when-it-gets-crowded).