Ajat Prabha

Unleashing xLoad: Your Ultimate Data Loader for Go Structs!

2024-07-07T01:00:00+00:00

Meet xLoad, your data-loading superhero 🦸‍♂️! It effortlessly brings data from various corners of the universe straight into Go structs. Whether you’re juggling configurations, internationalization labels, or experiments, xload is your trusty sidekick.

🤷‍♂️ Why xload?

Diverse Data Sources: Load data from environment variables, http requests, command-line arguments, files, or even remote sources like apis.
Avoid Boilerplate: Say goodbye 👋 to repetitive code and hello to a cleaner, more maintainable and readable codebase.
Flexibility: Inspired by go-envconfig but with more muscle 💪! Customize struct tags and use custom data loaders.

Getting Started 🛠️

go get github.com/gojekfarm/xtools/xload

How does xload work its magic? 🎩✨

Feed xload a Go struct with annotations.
Let xload populate it with data from any source you want.
- A source simply has to implement Loader interface, or use one of existing ones!
Enjoy the separation of data loading from its usage.

Here’s an example of the inbuilt OSLoader:

type Config struct{ Key string `env:"KEY"` }

func main() {
	var cfg Config
	_ = xload.Load(context.TODO(), &cfg)
}

📝 Dive into Examples & Tutorials

🌍 1. Loading from Environment Variables

Imagine you have an application that needs to fetch configurations from the environment it’s running in. Instead of manually fetching each variable, xload lets you define a struct and populates it for you.

type AppConfig struct {
    LogLevel string `env:"LOG_LEVEL"`
    // ...
    // your application config here
}
func main() {
    ctx := context.Background()
    cfg := DefaultAppConfig()
    err := xload.Load(ctx, &cfg, xload.OSLoader())
    // use cfg
}

🎨 2. Using Custom Types

With xload, you can also use custom types, such as Timeouts can be tricky. But with xload, you can directly specify durations like 5s without ambiguity. No more guessing if a timeout value is in seconds or milliseconds, and say goodbye to naming conventions like TIMEOUT_SEC!

type AppConfig struct {
    Timeout time.Duration `env:"TIMEOUT"` // TIMEOUT=5s is 5 seconds
}

Or implement your own by defining one of the following methods for xload:

interface{ Decode(string) error }
encoding.TextUnmarshaler
json.Unmarshaler
encoding.BinaryUnmarshaler
encoding.GobDecoder

Example:

// URL is a type alias for url.URL.
// The general form represented is: [scheme:][//[userinfo@]host][/]path[?query][#fragment]
type URL url.URL

func (u *URL) String() string { return (*url.URL)(u).String() }

func (u *URL) Decode(v string) error {
	parsed, err := url.Parse(v)
	if err != nil {
		return err
	}

	*u = URL(*parsed)

	return nil
}

🪄 3. Nested Structs

For complex applications, configurations can get intricate. With xload, you can nest structs, making it easier to group, reuse, and maintain configurations.

type HTTPConfig struct {
    Port int `env:"PORT"`
    Host string `env:"HOST"`
}

type AppConfig struct {
    Service1 HTTPConfig `env:",prefix=SERVICE1_"`
    Service2 HTTPConfig `env:",prefix=SERVICE2_"`
}

📦 4. Inbuilt Loaders

xload isn’t just a tool; it’s a toolkit. It comes with a set of inbuilt loaders to make your life easier:

OSLoader: Perfect for fetching data from environment variables.
PrefixLoader: Need to add prefixes to keys before loading? This is your go-to loader.
SerialLoader: When you need to load data from multiple sources, with the last non-empty value taking precedence.

🛠️ 5. Custom Loaders

For those special cases, xload allows you to craft custom loaders. As long as they implement the Loader interface, you’re golden. Similar Loader can be implemented for - In-memory Cached, Vault, S3, etc.

🔍 Spotlight: CRUD-based List API

Let’s dive into a real-world example to see the transformation xload brings:

Before xload:

Parsing request options was a multi-step process. The code was filled with multiple parsers and error handling, the code is verbose and a lot to read through.

func GetList(w http.ResponseWriter, r *http.Request) {
	qry := r.URL.Query()

	search := qry.Get("search")
	page, err := strconv.ParseInt(qry.Get("page"), 10, 32)
	if err != nil {
	// handle error
	}
	size, err := strconv.ParseInt(qry.Get("size"), 10, 32)
	if err != nil {
	// handle error
	}

	field := qry.Get("sort")
	desc := false
	if strings.HasPrefix(field, "-") {
		field = strings.TrimPrefix(field, "-")
		desc = true
	}

	//
}

After xload:

The code is streamlined. Define a struct, let xload populate it, and you’re good to go! The result? Cleaner, more maintainable code that’s a breeze to read.

type RequestOptions struct {
	Search string `query:"search"`
	Page   int    `query:"page"`
	Size   int    `query:"size"`
	Sort   sort   `query:"sort"`
}

type sort struct {
	Field string
	Desc  bool
}

func GetList(w http.ResponseWriter, r *http.Request) {
	var opts RequestOptions

	if err := xload.Load(
		r.Context(),
		&opts,
		xload.FieldTagName("query"),
		xload.WithLoader(queryLoader(r.URL.Query())),
	); err != nil {
	// handle error
	}

	// opts contain typed requestOptions and
	// any errors would also be caught here.
}

The transformation is clear: xload simplifies the data-parsing process and error handling, making the codebase more readable and maintainable.

🌐 Internationalization with Lokalize 🌐

With xload, you can load internationalization labels from files and then tweak them for each request using a remote service. This ensures your application speaks the language of your users, wherever they are.

type I18N struct {
    Title string `i18n:"TITLE"`
    CTA   CTA    `i18n:",prefix=CTA_"`
}

type CTA struct {
    Submit string `i18n:"SUBMIT"`
    Cancel string `i18n:"CANCEL"`
}

🎉 Wrap Up

That’s xload for you! A robust tool designed to simplify your data loading needs in Go. Dive in, explore, and let the magic unfold! 🌟

Check out the docs at pkg.go.dev for more examples.

xLoad is the brainchild of Ravi and I help out in maintaining it.

Introduction to xrun: A Flexible Package for Managing Component Lifecycles in Go

2023-05-24T01:00:00+00:00

Hello, Go community! Today, I am excited to announce a Golang package I have been working on named xrun. This package provides a streamlined way of running multiple components, specifically long-running components like an HTTP server or background worker. In this blog post, I will walk you through the basic functionality of xrun and how it can help you manage your long-running components more efficiently.

Introducing xrun

As the size and complexity of a Go service increase, managing the lifecycles of its various components becomes a daunting task. This post introduces xrun, a flexible and versatile Go package that simplifies this process, making it easier to reason about and manage your application’s components.

The package provides a Manager struct that handles starting and stopping components, coordinating their lifecycles in a way that makes your application more maintainable and less prone to bugs.

The Manager works by starting each component in its own goroutine and waiting for them to either finish or fail. It also ensures that the stop signals are propagated in reverse order of the components’ starting order, ensuring graceful shutdowns.

xrun is available at github.com/gojekfarm/xrun, and you can read the full documentation on pkg.go.dev.

Getting Started

Let’s start by importing the xrun package:

import "github.com/gojekfarm/xrun"

To illustrate how xrun works, let’s create an instance of an HTTP server:

import "net/http"

server := http.Server{Addr: ":9090"}

Next, create a new manager with xrun.NewManager(). Then use m.Add() to add the HTTP server to the manager:

import (
	"github.com/gojekfarm/xrun"
	"github.com/gojekfarm/xrun/component"
)

m := xrun.NewManager()
_ = m.Add(component.HTTPServer(component.HTTPServerOptions{Server: &server}))

Finally, run the manager using m.Run(ctx). If there is an error, the process will exit:

import (
	"os"
	"os/signal"
)

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()

if err := m.Run(ctx); err != nil {
	os.Exit(1)
}

Features of `xrun`

Now, let’s dive into some features of xrun.

Component

The Component interface is the foundation of xrun. It allows you to start a component that will run until the context is closed or an error occurs:

type Component interface {
    Run(context.Context) error
}

ComponentFunc

ComponentFunc is a helper that allows you to implement Component inline. It’s perfect for writing custom components on the fly:

type ComponentFunc func(ctx context.Context) error

Manager

The Manager type helps you to run multiple components and waits for them to complete. You can add components with the Add() method and run them with the Run() method:

type Manager struct {
    // ...
}

func (m *Manager) Add(c Component) error {...}
func (m *Manager) Run(ctx context.Context) (err error) {...}

Note: Since Manager has a Run(context.Context) error method, it is a Component in itself, and hence, it is possible to nest multiple Manager(s).

Creating and Using Components

A component in xrun is anything that implements the Component interface. This interface only requires a single method: Run(context.Context) error.

Here’s an example of a basic component, a ScheduledTask:

type ScheduledTask struct {
	Ticker *time.Ticker
	Task   func()
}

func (s *ScheduledTask) Run(ctx context.Context) error {
	for {
		select {
		case <-s.Ticker.C:
			s.Task()
		case <-ctx.Done():
			return nil
		}
	}
}

In the main function, you create a manager and add your components to it:

m := xrun.NewManager()

tick := time.NewTicker(1 * time.Second)
task := func() { fmt.Println("Task executed") }
s := &ScheduledTask{Ticker: tick, Task: task}

_ = m.Add(s)

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()

if err := m.Run(ctx); err != nil {
    log.Fatal(err)
}

Combining Components with `xrun.All`

Components can also be combined together using the xrun.All function. This function creates a new Component that starts all of the provided components in the order they were given and stops them in the reverse order.

Here is an example of its usage:

tick1 := time.NewTicker(1 * time.Second)
task1 := func() { fmt.Println("Task1 executed") }
s1 := &ScheduledTask{Ticker: tick1, Task: task1}

tick2 := time.NewTicker(2 * time.Second)
task2 := func() { fmt.Println("Task2 executed") }
s2 := &ScheduledTask{Ticker: tick2, Task: task2}

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()

if err := xrun.All(xrun.NoTimeout, s1, s2).Run(ctx); err != nil {
    log.Fatal(err)
}

Extending xrun: Custom Component Examples

While xrun provides some built-in components, you can create your own to suit your specific needs.

HTTP Server

The xrun package includes an HTTP server component out of the box. It is available in the xrun/component subpackage.

gRPC Server (Experimental)

For more advanced use cases, you can create custom components like a gRPC server. An experimental gRPC server component is available in the component/x/grpc subpackage.

Here’s an example of how you might create such a component:

package grpc

import (
	"context"
	"net"

	"github.com/gojekfarm/xrun"
	"google.golang.org/grpc"
)

// Options holds options for Server
type Options struct {
	Server      *grpc.Server
	NewListener func() (net.Listener, error)
}

// Server is a helper which returns a xrun.ComponentFunc to start a grpc.Server
func Server(opts Options) xrun.ComponentFunc {
	srv := opts.Server
	nl := opts.NewListener

	return func(ctx context.Context) error {
		l, err := nl()
		if err != nil {
			return err
		}

		errCh := make(chan error, 1)

		go func(errCh chan error) {
			if err := srv.Serve(l); err != nil && err != grpc.ErrServerStopped {
				errCh <- err
			}
		}(errCh)

		select {
		case <-ctx.Done():
		case err := <-errCh:
			return err
		}
		srv.GracefulStop()

		return nil
	}
}

func NewListener(address string) func() (net.Listener, error) {
	return func() (net.Listener, error) {
		return net.Listen("tcp", address)
	}
}

In this example, we define a new Options struct that includes our gRPC server instance and a NewListener function. This function creates a network listener on the given address.

Next, we define a Server function that takes an Options instance and returns a xrun.ComponentFunc. This ComponentFunc starts the gRPC server and manages its lifecycle. It starts the server in a goroutine and then enters a select block. If the context is done, the gRPC server is stopped gracefully. If an error occurs while serving, it’s returned.

The NewListener function is a helper that generates a function for creating a network listener.

Here’s how to use it:

package main

import (
	"context"
	"net"
	"os"
	"os/signal"

	xgrpc "yourgrpcpackage"
	"github.com/gojekfarm/xrun"
	"google.golang.org/grpc"
)

func main() {
	s := grpc.NewServer()
	m := xrun.NewManager()

	grpcComponent := xgrpc.Server(xgrpc.Options{
		Server:      s,
		NewListener: xgrpc.NewListener(":8500"),
	})

	_ = m.Add(grpcComponent)

	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
	defer stop()

	if err := m.Run(ctx); err != nil {
		os.Exit(1)
	}
}

Conclusion

The xrun package makes it easier to manage component lifecycles in your Go applications. By allowing you to define and control how each component starts and stops, you can make your application more maintainable and robust.

Finally, I’d like to give credits to the authors at the Kubernetes community. The manager source of xrun has been heavily influenced by the sigs.k8s.io/controller-runtime package.

Thanks for reading, and I hope you find xrun as useful as I do!

Contributions, questions, and feedback are most welcome on the xrun GitHub repository. Happy coding, Gophers!

Making a DIY Sound System that looks minimalistic

2020-10-06T12:00:00+00:00

I’m a huge music enthusiast and I listen to music almost half the day while working. Good quality speakers are a must for me. Recently I was changing the furniture in my house and I wanted to add a new sound system in the living room. I had two options:

Either I could go to the market and get one off-the-shelf
Or I could create one myself

I really liked the second idea because I wanted the speaker system to be as minimalistic as possible and it should be hidden in the room so that no one would even notice that it is there. I also wanted to have really punchy bass while having crisp and crystal clear mids and highs.

The arm rest thing in the sofa above is the sound system!😜
Serving dual purpose, and is also hidden and minimalistic.

Demo at the end!

Studying about sound systems

Now the problem was that I have never done it before! So how do I go about it? As anyone would do, I Googled how to make DIY sound system and it flooded me with lots of DIY videos and information around how to decide the characteristics of the sound that the system will produce or how to design the closure for the subwoofer.
But this was overkill for me! After lots and lots of study around this topic, I came to know that there are many decisions that I have to take like:

If I want to use 4 ohm speakers or 8 ohm speakers?
What type of amplifier should I use? Should it be a class A, B, AB, C or D amplifier?

I decided that I will go with 2x 50 watt Pioneer TS-1601IN speakers and a 150 watt subwoofer, all of the components are 4 ohms.
Digging in a little deeper I found an IC, TPA3116D2, which is a dual channel class D amplifier and is also configurable, it was perfect for my use case. The were only two problems:

It can only supply 100 watts to the subwoofer(I was okay with this)
It had to be supplied with exactly 21 volts to drive the 4 ohm speakers

In order to supply it with exactly 21 volts of voltage and around 7.5 amperes of maximum current, I tried to find a few power supplies with that exact description but wasn’t able to find any. So the last resort that I had was to build that power supply myself and I did exactly that.

The DIY Power Supply

I again went back to Google and started looking for how can I make a power supply myself. I had a few requirements that the voltage should be constant at 21 volts and the maximum current that can flow through the circuit should be up to 7.5 ampere, because 200W/21V ~ 9.5 Amps and 7.5 Amps should be enough to produce good quality sound without the amplified being shut-off.
At last I found an IC named LM317 whose output voltage can be configured with the help of a resistance or a potentiometer. But the problem with this IC is that, it can only support upto 1.5 Amps of current.
In basic electronics, we have learnt that if you connect these ICs in parallel, the current would divide across them but the voltage drop across them would remain the same. So I thought that I would connect five of these ICs in parallel and I created a circuit for that. Finally, I added a transformer and bridge rectifier with some capacitors to get a decent 35-36V DC input for the power supply I created.

I can go into more details about how exactly I did this but I think that would make this post a bit long. Let’s keep that for another post, if required, and here’s a video of how the speaker sound and look.

Your browser does not support the video tag.

This is the bass response at <50% volume though! 😛

Speaker covers are from JBL, couldn’t find Pioneer ones!

Until next DIY 👋🏻

Writing Kubernetes Operator with an API Server for Custom GUI

2020-06-24T12:00:00+00:00

If you have read my previous blog posts, I worked on an image proxy called Darkroom which manipulates the images on the fly. I also started learning more about Kubernetes and then I thought I should build something to get a deeper understanding of K8S. So the task that I took up is to create a GUI that is simple to use and it deploys Darkroom on-demand and everything else should happen without manually running kubectl commands.

Background

Before starting off, let me tell you about the current state of Darkroom. It is a stateless application, configurable via environment variables and it has first-class support for containerization and the image size is less than 20MB. These are some of the reasons that I decided to pick this for the hands-on as the complexity to deploy is quite less. However, any application that follows the Twelve Factor App principles should be a good candidate.

Why make an operator?

There can be many ways on how you manage your workloads on K8S viz.

Applying manifests via kubectl or kustomize
Using helm as a package manager to install and upgrade apps
Use custom operator to make the system smart enough work on its own
- Operators can install, upgrade, maintain lifecycle, provide insights of workloads, they can essentially put your workload on auto-pilot

Now moving back to Darkroom, in order to deploy this on Kubernetes, a simple deployment is enough. We can apply a Deployment manifest with the following command.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: darkroom-deployment
  labels:
    app: darkroom
spec:
  replicas: 1
  selector:
    matchLabels:
      app: darkroom
  template:
    metadata:
      labels:
        app: darkroom
    spec:
      containers:
      - name: darkroom
        image: gojektech/darkroom
        ports:
        - containerPort: 3000
        env:
        - name: SOURCE_KIND
          value: WebFolder
        - name: SOURCE_BASEURL
          value: https://ajatprabha.in/assets
EOF

Use port-forwarding now to access the application

kubectl port-forward $(kubectl get pods -l app=darkroom -o=jsonpath='{.items[0].metadata.name}') 3000:3000

You should be able to see this image http://localhost:3000/images/ IMG_20180604_125819.jpg?w=512 now, and change its width, height, etc by changing the query parameters.

Say Hello to Operators (CRD + Controller)

With the help of CustomResourceDefinition aka CRD, we can create our own custom APIs and extend the funcitonality of Kubernetes. We can then create a controller which will work on this CRD! An overview of this is shown in this diagram:

Since operator runs in background we can create these CRDs from anywhere via the kube-api-server, we will create our own api-server to talk to the kube-api-server later in this post and add a nice GUI to expose it.

Note: All the code for this example can be found at github.com/ajatprabha/operator-example

Prepare Environment

First off, we need to have a working kubernetes development cluster running locally, minikube is a really good option for that. Install it(brew install minikube) and start a local cluster with minikube start --driver=kvm2(I prefer to use KVM, but you may use –driver=none).

😄  minikube v1.11.0 on Ubuntu 20.04
✨  Using the kvm2 driver based on user configuration
💾  Downloading driver docker-machine-driver-kvm2:
    > docker-machine-driver-kvm2.sha256: 65 B / 65 B [-------] 100.00% ? p/s 0s
    > docker-machine-driver-kvm2: 13.88 MiB / 13.88 MiB  100.00% 739.31 KiB p/s
💿  Downloading VM boot image ...
    > minikube-v1.11.0.iso.sha256: 65 B / 65 B [-------------] 100.00% ? p/s 0s
    > minikube-v1.11.0.iso: 174.99 MiB / 174.99 MiB  100.00% 743.99 KiB p/s 4m1
👍  Starting control plane node minikube in cluster minikube
💾  Downloading Kubernetes v1.18.3 preload ...
    > preloaded-images-k8s-v3-v1.18.3-docker-overlay2-amd64.tar.lz4: 526.01 MiB
🔥  Creating kvm2 VM (CPUs=2, Memory=3900MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.18.3 on Docker 19.03.8 ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

Next, we need to install kubebuilder which will help us in scaffolding operator for K8S. To test your installation, run kubebuilder version which should exit without errors.

Setup

Now we can inititialize a blank repository to start working on.

mkdir operator-example
cd operator-example
git init && git add --all && git commit -m "init"

Use kubebuider to initialize a new operator project

go mod init github.com/ajatprabha/operator-example
kubebuilder init --domain example.com
git add --all && git commit -m "kubebuilder init"

You should change the module name from github.com/ajatprabha/operator-example to something else. The domain should also be unique to your organisation.

API Definition

Once the initial scaffolding is done, we can start defining our APIs. Kubernetes has a set of internal APIs for different kinds viz. Pod, Service, ConfigMap, etc. We will create a new CRD called Darkroom which will be responsible for this automation that we want to achieve.

kubebuilder create api --group deployments --version v1alpha1 --kind Darkroom
    Create Resource [y/n] y
    Create Controller [y/n] y
    Writing scaffold for you to edit...
    api/v1alpha1/darkroom_types.go
    controllers/darkroom_controller.go
git add --all && git commit -m "scaffold resource and controller"

Modifying Resource

Once the resources is generated, fire up your favourite IDE and take a look at the api/v1alpha1/darkroom_types.go file. Here you can see a typed definitions for the resource Darkroom.

// DarkroomSpec defines the desired state of Darkroom
type DarkroomSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file

	// Foo is an example field of Darkroom. Edit Darkroom_types.go to remove/update
	Foo string `json:"foo,omitempty"`
}

// DarkroomStatus defines the observed state of Darkroom
type DarkroomStatus struct {
	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
	// Important: Run "make" to regenerate code after modifying this file
}

// +kubebuilder:object:root=true

// Darkroom is the Schema for the darkrooms API
type Darkroom struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   DarkroomSpec   `json:"spec,omitempty"`
	Status DarkroomStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true

// DarkroomList contains a list of Darkroom
type DarkroomList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []Darkroom `json:"items"`
}

Notice the comments such as +kubebuilder:object:root=true, KubeBuilder makes use of a tool called controller-gen for generating utility code and Kubernetes YAML. This code and config generation is controlled by the presence of special “marker comments” in Go code.

Now let’s modify these Go structs to hold information that we care about.

// +kubebuilder:validation:Enum=WebFolder;S3
type Type string

const (
	WebFolder Type = "WebFolder"
	S3        Type = "S3"
)

type WebFolderMeta struct {
	BaseURL string `json:"baseUrl,omitempty"`
}

type S3Meta struct {
	AccessKey  string `json:"accessKey,omitempty"`
	SecretKey  string `json:"secretKey,omitempty"`
	Region     string `json:"region,omitempty"`
	PathPrefix string `json:"pathPrefix,omitempty"`
}

type Source struct {
	// Specifies storage backend to use with darkroom.
	// Valid values are:
	// - "WebFolder" (default): simple storage backend to serve images from a hosted image source;
	// - "S3": storage backend to serve images from an S3 bucket;
	Type Type `json:"type"`
	// +optional
	WebFolderMeta `json:",inline"`
	// +optional
	S3Meta `json:",inline"`
}

// DarkroomSpec defines the desired state of Darkroom
type DarkroomSpec struct {
	// +optional
	Version string `json:"version"`
	Source Source `json:"source"`
	// +kubebuilder:validation:MinItems=1
	SubDomains []string `json:"subDomains"`
}

// DarkroomStatus defines the observed state of Darkroom
type DarkroomStatus struct {
	// +optional
	Domains []string `json:"domains,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:JSONPath=".spec.version",name=VERSION,type=string
// +kubebuilder:printcolumn:JSONPath=".spec.source.type",name=TYPE,type=string
// +kubebuilder:resource:shortName=dr

// Darkroom is the Schema for the darkrooms API
type Darkroom struct {
	/* ... */
}

Run make manifests to generate YAMLs for this CRD, kubebuilder will use the marker comments to add utilities, validation, etc. printcolumn will specify what fields to show when we do kubectl get darkrooms, yes now we can use kubectl to interact with this new Kind.

Now update the sample manifest to define a Darkroom object, just as you would define a Pod or Service

apiVersion: deployments.example.com/v1alpha1
kind: Darkroom
metadata:
  name: darkroom-sample
spec:
  source:
    baseUrl: https://ajatprabha.in/assets
    type: WebFolder
  subDomains:
    - ajatprabha
  version: 0.1.0

Note: I’ve used my blog’s address, you can modify it according to your usecase.

Install the CRD manifest in the cluster and then apply this new Darkroom manifest with kubectl apply -f config/samples/deployments_v1alpha1_darkroom.yaml

git add --all && git commit -m "update CRD definition and manifests"

make install
/usr/local/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/darkrooms.deployments.example.com created

kubectl apply -f config/samples/deployments_v1alpha1_darkroom.yaml
darkroom.deployments.example.com/darkroom-sample created

kubectl get darkrooms                                             
NAME              VERSION   TYPE
darkroom-sample   0.1.0     WebFolder

Okay, great! We can now persist our new Darkroom objects in Kubernetes’ object store.

Implementing Controller

Now, we will look at how we can implement the controller to act upon these Darkroom objects and work towards matching the current state with the desired state in a reconciliation loop.
We’ve already seen in the starting of this blog post that how we can use the K8S internal APIs to accomplish a simple Darkroom deployment using kubectl.
Any deployment inside K8S requires an image for the Pod, since our application reads configuration from environment variables, we can inject these values in the Pod from a ConfigMap, finally we expose this Pod via a Service within the cluster and update the Ingress to allow external traffic to this Service.

Take a look at the controller code in the controllers/darkroom_controller.go file.

func (r *DarkroomReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
	_ = context.Background()
	_ = r.Log.WithValues("darkroom", req.NamespacedName)

	// your logic here

	return ctrl.Result{}, nil
}

func (r *DarkroomReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&deploymentsv1alpha1.Darkroom{}).
		Complete(r)
}

We will write our reconciliation logic here, but before we do that, let’s run the controller once and see what happens.

make run
/usr/local/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/usr/local/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go run ./main.go
2020-06-24T22:58:27.059+0530	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2020-06-24T22:58:27.059+0530	INFO	setup	starting manager
2020-06-24T22:58:27.060+0530	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2020-06-24T22:58:27.060+0530	INFO	controller-runtime.controller	Starting EventSource	{"controller": "darkroom", "source": "kind source: /, Kind="}
2020-06-24T22:58:27.161+0530	INFO	controller-runtime.controller	Starting Controller	{"controller": "darkroom"}
2020-06-24T22:58:27.161+0530	INFO	controller-runtime.controller	Starting workers	{"controller": "darkroom", "worker count": 1}
2020-06-24T22:58:27.161+0530	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "darkroom", "request": "default/darkroom-sample"}

As you can see, it picked up our created Darkroom object named darkroom-sample, nothing actually happened since their is no logic in the controller yet.
We will first tell the controller manager that what objects this controller interacts with and what is the type of interaction.

func (r *DarkroomReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&v1alpha1.Darkroom{}).
		Owns(&corev1.ConfigMap{}).
		Owns(&corev1.Service{}).
		Owns(&appsv1.Deployment{}).
		Owns(&v1beta12.Ingress{}).
		Complete(r)
}

Here we have specified that this controller is for v1alpha1.Darkroom, and it owns Deployment, ConfigMap, Service and Ingress.

Now for the actual reconciliation logic, we have to keep in mind that the loop should be idempotent. This is very important. The logic invloves 4 simple steps for now:

Create spec for desired ConfigMap
Create spec for desired Deployment which takes environment variables from the above ConfigMap
Create spec for desired Service for above Deployment
Create spec for desired Ingress for routing external traffic to above Service

func (r *DarkroomReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    ctx := context.Background()
    r.Log.WithValues("darkroom", req.NamespacedName)

    var darkroom deploymentsv1alpha1.Darkroom
    if err := r.Get(ctx, req.NamespacedName, &darkroom); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    cfg, err := r.desiredConfigMap(darkroom)
    if err != nil {
        return ctrl.Result{}, err
    }

    deployment, err := r.desiredDeployment(darkroom, cfg)
    // purposely hide repeating error return code
    if err != nil { /* ... */ }
    svc, err := r.desiredService(darkroom)
    if err != nil { /* ... */ }
    ingr, err := r.desiredIngress(darkroom, svc)
    if err != nil { /* ... */ }

    applyOpts := []client.PatchOption{client.ForceOwnership, client.FieldOwner("darkroom-controller")}

    err = r.Patch(ctx, &cfg, client.Apply, applyOpts...)
    if err != nil { /* ... */ }
    err = r.Patch(ctx, &deployment, client.Apply, applyOpts...)
    if err != nil { /* ... */ }
    err = r.Patch(ctx, &svc, client.Apply, applyOpts...)
    if err != nil { /* ... */ }
    err = r.Patch(ctx, &ingr, client.Apply, applyOpts...)
    if err != nil { /* ... */ }

    darkroom.Status.Domains = domainsForService(ingr)
    err = r.Status().Update(ctx, &darkroom)
    if err != nil { /* ... */ }

    return ctrl.Result{}, nil
}

The above code might feel repitive since we want to achieve idempotency. However, it can be refactored better but that is out of scope for this demo.
The actual implementation of desiredConfigMap, desiredDeployment, desiredService and desiredIngress can be found in helpers.go
We can now run the controller and it will work as expected.

Your browser does not support the video tag.

At this point, we can build and push the controller to an image registry with make docker-build && make docker-push and then deploy the controller to the K8S cluster with make deploy

API Server and GUI

Now we can finally create a custom api-server with a GUI to perform CRUD on Darkroom CRD, we do not need to use kubectl after this. Some of the trivial code is not explained and commented out, it can be found in the package api_server

// main.go

func init() {
  // Initialize the scheme so that kubernetes dynamic client knows
  // how to work with new CRD and native kubernetes types
	_ = clientgoscheme.AddToScheme(scheme)
	_ = deploymentsv1alpha1.AddToScheme(scheme)
}

func main() {
	/* ... */

	mgr, err := apiserver.NewManager(ctrl.GetConfigOrDie(), apiserver.Options{
		Scheme:         scheme,
		Port:           5000,
		AllowedDomains: []string{},
	})
	if err != nil { /* ... */ }

	runLog.Info("starting api-server manager")
	if err := mgr.Start(signals.SetupSignalHandler()); err != nil { /* ... */ }
}

Note: Registering the types with scheme in init function is the key here. Without this, the K8S dynamic client cannot work.

Now we will create an API Server Manager that will create the K8S client and keep a reference to it. It will also create a cache that will be used to create a cached K8S client, initialize the cache properly and in the end handle the termination signals.

// manager.go

// Options to customize Manager behaviour and pass information
type Options struct {
	Scheme         *runtime.Scheme
	Namespace      string
	Port           int
	AllowedDomains []string
}

type Manager interface {
	Start(stop <-chan struct{}) error
}

type manager struct { /* ... */ }

func NewManager(config *rest.Config, options Options) (Manager, error) {
	mapper, err := apiutil.NewDynamicRESTMapper(config)
	if err != nil { /* ... */ }
  
	// we use cache to optimize fetch from the kube-api-server
	cc, err := cache.New(config, cache.Options{
		Scheme:    options.Scheme,
		Mapper:    mapper,
		Resync:    &defaultRetryPeriod,
		Namespace: options.Namespace,
	})
	if err != nil { /* ... */ }

	c, err := client.New(config, client.Options{Scheme: options.Scheme, Mapper: mapper})
	if err != nil { /* ... */ }

	stop := make(chan struct{})
	return &manager{
		config: config,
		cache:  cc,
		client: &client.DelegatingClient{
			Reader: &client.DelegatingReader{
				CacheReader:  cc,
				ClientReader: c,
			},
			Writer:       c,
			StatusClient: c,
		},
		internalStop:    stop,
		internalStopper: stop,
		port:            options.Port,
		allowedDomains:  options.AllowedDomains,
	}, nil
}

func (m *manager) Start(stop <-chan struct{}) error {
	defer close(m.internalStopper)
	// initialize this here so that we reset the signal channel state on every start
	m.errSignal = &errSignaler{errSignal: make(chan struct{})}
	m.waitForCache()

	srv, err := newApiServer(m.port, m.allowedDomains, m.client)
	if err != nil { /* ... */ }

	go func() {
		if err := srv.Start(m.internalStop); err != nil {
			m.errSignal.SignalError(err)
		}
	}()
	select {
	case <-stop:
		return nil
	case <-m.errSignal.GotError():
		// Error starting the cache
		return m.errSignal.Error()
	}
}

// start cache in separate goroutine
func (m *manager) waitForCache() {
	if m.started {
		return
	}

	go func() {
		if err := m.cache.Start(m.internalStop); err != nil {
			m.errSignal.SignalError(err)
		}
	}()

	// Wait for the caches to sync.
	m.cache.WaitForCacheSync(m.internalStop)
	m.started = true
}

The server uses go-restful to provide RESTful endpoints to a JavaScript based GUI.

// server.go
type apiServer struct {
	server *http.Server
}

func (as *apiServer) Address() string {
	return as.server.Addr
}

func init() {
	restful.MarshalIndent = func(v interface{}, prefix, indent string) ([]byte, error) {
		var buf bytes.Buffer
		encoder := restful.NewEncoder(&buf)
		encoder.SetIndent(prefix, indent)
		if err := encoder.Encode(v); err != nil { /* ... */ }
		return buf.Bytes(), nil
	}
}

func newApiServer(port int, allowedDomains []string, client client.Client) (*apiServer, error) {
	container := restful.NewContainer()
	srv := &http.Server{
		Addr:    fmt.Sprintf(":%d", port),
		Handler: container.ServeMux,
	}

	cors := restful.CrossOriginResourceSharing{
		ExposeHeaders:  []string{restful.HEADER_AccessControlAllowOrigin},
		AllowedDomains: allowedDomains,
		Container:      container,
	}

	ws := new(restful.WebService)
	ws.
		Path("/").
		Consumes(restful.MIME_JSON).
		Produces(restful.MIME_JSON)

	addEndpoints(ws, client)
	container.Add(ws)
	container.Filter(cors.Filter)
	return &apiServer{
		server: srv,
	}, nil
}

func addEndpoints(ws *restful.WebService, client client.Client) {
	resources := []endpoints.Endpoint{
		endpoints.NewDarkroomEndpoint(client),
	}
	for _, ep := range resources {
		ep.SetupWithWS(ws)
	}
}

func (as *apiServer) Start(stop <-chan struct{}) error {
	errChan := make(chan error)
	go func() {
		err := as.server.ListenAndServe()
		if err != nil { /* ... */ }
	}()
	log.Info("Starting api-server", "interface", "0.0.0.0", "port", as.Address())
	select {
	case <-stop:
		log.Info("Shutting down api-server")
		return as.server.Shutdown(context.Background())
	case err := <-errChan:
		return err
	}
}

Finally, in the DarkroomEndpoint we have the reference to the K8S client and we can use it perform operations on the Darkroom CRD. For example, list, create, edit and delete Darkrooms!

// endpoints/darkroom.go

type DarkroomEndpoint struct {
	client client.Client
}

func NewDarkroomEndpoint(client client.Client) *DarkroomEndpoint {
	return &DarkroomEndpoint{client: client}
}

func (de *DarkroomEndpoint) SetupWithWS(ws *restful.WebService) { /* route paths to handler funcions */ }

func (de *DarkroomEndpoint) list(request *restful.Request, response *restful.Response) {
	dl := new(v1alpha1.DarkroomList)
	err := de.client.List(request.Request.Context(), dl, &client.ListOptions{})
	if err != nil { /* ... */ } else {
		l := From.List(dl)
		if err := response.WriteAsJson(l); err != nil { /* ... */ }
	}
}

func (de *DarkroomEndpoint) create(request *restful.Request, response *restful.Response) {
	d := new(Darkroom)
	err := request.ReadEntity(d)
	if err != nil { /* return error */ }

	if err := d.Validate(); err != nil { /* ... */ }
	obj := &v1alpha1.Darkroom{ /* fill all common values */ }

	// Use obj.Spec.Source.Type to populate other values like 

	err = de.client.Create(request.Request.Context(), obj, &client.CreateOptions{})
	if err != nil { /* return error */ } else {
		d := From.Object(obj)
		if err := response.WriteAsJson(d); err != nil { /* return error */ }
	}
}

Fire up the server now with go run ./api-server/cmd/main.go or with make target(if defined) make api

make api
/use/local/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./api-server/cmd/main.go
2020-06-25T00:37:44.391+0530	INFO	darkroom-cp.run	starting api-server manager
2020-06-25T00:37:44.392+0530	INFO	api-server	Starting api-server	{"interface": "0.0.0.0", "port": ":5000"}

And then test it with curl to list the current Darkroom CRD instances

curl localhost:5000/darkrooms
{
 "items": [
  {
   "name": "darkroom-sample",
   "version": "0.1.0",
   "source": {
    "type": "WebFolder",
    "baseUrl": "https://ajatprabha.in/assets"
   },
   "status": {
    "domains": [
     "ajatprabha.darkroom.example.com"
    ]
   }
  }
 ]
}

The new API server can also be used to create new Darkroom instances.

Your browser does not support the video tag.

Now you can connect this to a GUI, I will include the GUI code in the repo but won’t go through it as it’s out of scope of this blog post.

And this concept of leveraging K8S using operators and a custom GUI can be used to create many solutions. If you can do it manually, you can automate it with operators!

I hope this was an helpful post, feel free to drop your comments and queries.

Week 4 - Load testing with Vegeta and trip to Puducherry

2018-06-17T12:00:00+00:00

This week, I acknowledged the importance of load testing before deploying a service into production which has to serve 50K images per minute.

To know what service I’m testing head over to week 2’s post here if you haven’t already.

Day 1

We had to load test the image manipulation service that we built until now. My mentor asked us to get our hands dirty with a Golang based HTTP load testing tool called Vegeta, yes the grumpy character from Dragon Ball obsessed with surpassing Goku. It is quite a good load testing tool that I’ve come across. I also tried a python tool called Locust. While it provides a nice UI and other options, I liked Vegeta more because it’s more reliable. We now had to validate our service’s performance and make sure that it can survive the load. The day went mostly around getting to know Vegeta and doing a few load tests locally.

Day 2-3

Initial load tests were not very good, we could hardly hit 60 rpm! After looking at the code we realized that there was a memory leak, which was constantly increasing our heap size. We optimized the memory allocation/de-allocation for each request and the heap size was then under control. This optimization increased the rpm a little bit but not significantly.

We got a lot of server timeout errors because the service was not able to respond within 30 seconds and the response time used to increase linearly. For some strange reason, the disk read-writes were taking time. We got to the root of this in the coming days.

On Wednesday evening, I went out to an office meeting at Hotel Royal Orchid, a very nice place. I had blue cheese there for the first time, and as expected by the smell of it, it did not taste very good to me.😂

Day 4-5

We were supposed to take the service live by the weekend but due to performance issues, it got delayed. It sounds easy to build such a thing but believe me it’s not!

So all of us looked deeper into the code flow, we set up logs and found that downloading the image from S3 bucket was taking time. We tried to overcome this by setting up a download job worker and sending the results to a channel. But it didn’t help much. Apparently, the result of Golang’s ioutil.ReadAll(response.Body) was taking up too much time which was blocking the service, read more here. We got around this by saving the image temporarily to the disk.

The service was now able to handle 2K rpm of traffic and with 4 application VMs we predicted 8K rpm as the threshold, but it turned out later that it was incorrect!

Weekend

For the weekend, I went to Puducherry along with my seniors from college. We were 13 people in two cars. The trip started with a car that had no central locking remote and the electronics were behaving weird because of that. After wasting an hour in getting the correct key, the journey began. Our first stop was McDonald’s for the breakfast. I was driving one of the cars and the other car missed the service road, so we took there order too and later met alongside the highway and had our meal.

We refueled, put on some music and headed straight towards Puducherry. The road was better to Coorg because of the greenery as compared with the road to Puducherry. Also, we got to see more and more people wearing Lungi in Tamil Nadu.

There is this one incident that took place, we were crossing a town and a person out of nowhere came in the middle of the road with his hands wide open, said something, all of us were shocked and then he just went away.🤣 After crossing the town, we stopped near a lake to take a group photo!

It took us roughly 6 hours to reach Puducherry and as soon as we reached, we went to the beach. To be honest, it was not what I expected! I’ve not been to Goa yet but every other beach is just okay, not anything like what is shown in movies/videos.

Late in the evening, we went to Promenade Beach and the White Town, it was a good place. We had our dinner and me and Krishna went on to hunt a place to crash during the night.

Next morning, we went to Serenity beach, it was good but not as good as what it is shown in images. I spent an hour or two there. Next stop was again the Puducherry city. After visiting the city in daylight, we thought to visit Paradise beach but seeing the waiting line we decided to not go there and head back to Bangalore.

Around late evening we reached the suburbs of Bangalore. This was a good road trip but I liked the Coorg trip more! Pro tip: Always travel with fewer people!😜

Week 3 - Lots of code and learning

2018-06-10T12:00:00+00:00

Week 3 was filled with more of coding and learning new things down the line. I learned how dependency injection in Golang is done, writing mocks and stubs for the test cases, etc. to name a few. How an application is deployed to tens of servers with the push of a button.

To know what I’m building head over to week 2’s post here if you haven’t already.

Day 1

Mudit was deploying an incremental update to one of the services that power the Go-Food stack at GoJek. I asked him if I can join and he was happy to show me how it’s done. The scale at which GoJek works and how a single engineer is responsible for half a million user’s requests is mind-boggling. When you have that kind of requests coming in every second you can’t afford your systems to go down even for a blink of an eye.

There are several layers through which a request goes and one of them is the load balancer layer, HAProxy is used in this case. So every request falls on to HAProxy and then it is diverted to one the application servers for the request to be processed. GoJek has quite a number of them and when an update is deployed, it is deployed to one application server at a time. This way the other servers are still up to serve the requests coming in.

Although to deploy an update you need to inform other concerned teams about it and create a document mentioning what changes it is going to make, what other services will it affect, etc? Once the teams are informed over internal IM, there are a few monitoring services to be fired up in order to track any undesired effect during the rollout.
The actual rollout is very easy because it is all automated. You just click a button and boom, the deployment starts one-by-one on every machine in the stack. Since the code is added, refactored and changed at blazing speeds here, we need such Continuous Integration and Continuous Deployment tools to keep us going without a bottleneck. More on the process in Day 4.😉

The deployment was successful and the new feature was up and the Apdex score was satisfactory. It took hardly 15-20 mins in the whole process without any decrease in the overall throughput of the application requests coming in. We worked on our own project for the rest of the day.

Day 2-3

I and Rajat started working on the handler that’ll serve the images after processing them, we wrote some tests and then wrote the code for the handler. We stubbed the function that’ll actually process the image but it wasn’t feeling right to go top-down, so we decided to go bottom-up and started working on the ImageProcessor service first and we chose to implement resize capability first. We’re using imagick library which provides Go bindings to the underlying ImageMagick’s MagickWand C API.

We faced a lot of problems because the image object was being mutated in different parts of the existing code. And we had to also save the processed image back to the S3 bucket but with a new UUID which has to be generated based on an SHA-1 hash of the ordered query string that was used to process the image so that we never process one image again and again.

We implemented the resize and crop capability and it was working as expected. We then linked these new image processing capabilities to the handler that’ll actually serve the requests on the next day and tested it locally. But we had made a mess as we wrote all the code inside the handler itself. The next task was to fix the code smell.

Day 4

It was now time to introduce a new decision-making service that’ll take in the requested image’s name and the query parameters and based on the data it’ll decide what to do and call the image processor to actually process the image. I had to use some of the existing services and since I was practicing TDD, I had quite a hard time in stubbing the existing services.
I was writing tests for a function and had to call the same function twice inside the decision-making service but wanted it to behave differently based on the input I was giving to the function. That’s when I came across mocks in Golang and how they provide additional ways to ease the task of testing. Dependency injection was very important in this part as I had to use some of the existing services inside the new service while practicing TDD.😅

Learning session

Puneet held a learning session in which I learned how actually the deploying process works, what happens inside all this one click voodoo that deploys your app automatically. There’s a Chef, a knife, a cookbook, and many recipes. Yes, they’re actually named like this.
A chef knows everything, it is a server which is used to communicate during the deployment process and tells the chef-clients what to do. Every VM has a chef-client which is instructed to pick up a cookbook and start the deployment. A cookbook is nothing but contains the information on how to build an application, just like an actual cookbook has recipes for dishes. Now the individual recipes are analogous to applications.

Let’s say your application needs database. You can avoid all the pain to set up the database server and just include the postgresql or mysql recipe and voila, chef-client does it for you. Same happens with your own application, you just need to specify a recipe on how to build the app.

Day 5

We added some analytics utilities to the code in order to track the performance and any other potential errors. Sainath also added the ability to turn an image all black and white. Deepesh made some adjustments and merged our branch into master and deployed the new code to integration environment where it’ll be tested thoroughly in the coming days.

The next step is to load test our image processing service, after all, it has to handle 50K rpm!

Week 2 - Exploring Golang and the trip to Coorg

2018-06-03T12:00:00+00:00

It’s been great till now and our mentor came back from Jakarta. It’s time to start building the service that will handle 50K+ image requests per minute. I also went on a road trip to Coorg at the weekend, the scenes were jaw-dropping, one of the best trips I went on.

Day 1

I reached the office at around 1000 hours and joined Puneet and Deepesh a few moments later. Puneet briefed us about the project that we are assigned to and how it should be able to serve at 50K+ rpm(requests/minute) at normal load and up to 80K rpm during peak times.

Rajat, Sainath and I started to code the functionality. At first, I thought to code the handler that’ll cater to the requests but later I realized that I should first implement the ImageProcessor service because it was a primitive one that could be tested in isolation to the rest of the program.

Let me explain a bit, suppose a request comes for an image at URL host[dot]com/image.jpg then the handler should return the requested image which is 600x400 pixels. But now I want to show the same image on multiple platforms, be it a 5” iPhone, a 6” Android, or an 8” tablet. The image should be crisp on large screens but the same image is useless for 5” iPhone since much of the pixels are not utilized. So I can ask for a smaller image by hitting the URL host[dot]com/image.jpg?w=320&h=240 and the returned image will not be more than 320x240 pixels.

Now the handler has to figure out what to do with the image based on the query parameters in the URL. Since we have to implement resize, crop and grayscale functionality, for now, I decided to work on individual image processing parts first. It’ll be easier to test also. But it was not that easier first because we used a Go wrapper on ImageMagick library for image manipulation and the version was missing some features that we wanted to use. The solution was to create our own logic.

Original

Resized image

Cropped image

There are two major use cases: Resize and Crop
Now the Go wrapper stretches the image while resizing, so we wrote some logic to not distort the image while processing. Suppose I ask for a width of 320 pixels and a height of 240 pixels. While resizing, the image processor should pick appropriate width and height and resize it, in the case above see how the image is contained within 320x240 pixels, the image here is actually 320x213 pixels.

The second case is cropping the image. When I make a request with URL host[dot]com/image.jpg?w=320&h=240&fit=crop&crop=top,left, the service should return me a cropped image as shown in the image above.

So the requirement is clear, next step is to write a test to define the behavior. This is not the very first test I wrote for this functionality but I’m taking this as an example here.

After this and a few more tests, the production code looked something like the code below.

But we did a mistake! We wrote all the logic in the handler for deciding what type of processing is to be done on the image. I was reluctant to write everything in the handler at first but I thought maybe this is the way one codes in Go i.e. being very verbose, but as it turned out, I was not completely right. We realized this much later but now we have to refactor the code.

It’s not as easy as it seems to write code along with TDD while you’ve to understand the structure of the code that is written by someone else. It takes many iterations to figure out correctly why a certain piece of code was written. By the weekend, we implemented resize and crop very vaguely in the handler. The next week’s task is to refactor the code and move everything in its correct place.

That’s all for the work part, wanna know how the weekend went?

Weekend

It was 0400 hours in the morning, I woke to pick the car we booked for our trip to Coorg. Krishna took the car, picked up Riya and Anugrah, we were next in line, and then we were off to Coorg probably around 0600 hours. The weather was awesome, there was music playing in the car and we were enjoying the drive. Around 0930 hours we pulled over the car to have breakfast and it was great to find such south Indian food on the highway in the middle of nowhere.

We continued the trip and it would take us 4 more hours to reach our destination. The road was great, people were awesome. You can’t get bored when you’ve Anugrah with you. Stories from college, previous road trip experiences, etc everything was talked about. Our playlist included songs from the year as old as 1970s to 2018, from Mere mehboob qayamat hogi to the latest Tareefan, from Aao milo chalein to Mi Gente. It was already the best trip ever.

We decided to go river rafting but unfortunately, the sport was closed for some reason.😞 We went to see Abbey falls it was not as I expected but the drive to it was great. Abhinav has been to Coorg before, we called him for suggestions and he suggested us to go to Talakaveri. And let me tell you the road was awesome, so full of nature. When we almost reached there, it started raining and continued for around half an hour.

We drove back to Madikeri from Talakaveri, it was around 2200 hours and we looked for a place for spending the night. We came across a homestay and it was very good, the neighborhood was awesome. I felt like I was in Kashmir or something.😂 I was very tired and went to sleep quite fast.

The next morning when I woke up and got outside, the view was jaw-dropping. There was mist everywhere, the temperature was around 25°C or low, birds were chirping. Everything was so peaceful.

Rajat was our travel planner, he picked a few places that we had to visit before 1400 hours, but sadly, we could only make it to two of them.

Raja’s Seat

Filled with seasonal flowers and artificial fountains, it was really some kind of a King’s seat. The mountain ranges, the valley below and the greenery were so damn refreshing. The weather was as usual very pleasant. We had our breakfast and zoomed to our next destination.

Nisargadhama

This was some kind of island which was surrounded by a river! I know, seems a little odd to imagine an island in Coorg. We had to walk over a hanging bridge to get to the island. There were deer on the island. We found a way to get to the river bank and removed our shoes to get into the water. It was cold! We spent almost an hour there and decided to head back to Bangalore. The whole time it was raining when we were coming back. But it was a nice drive back in the rain.

Your browser does not support the video tag.

This was the best road trip I went on. Let’s see where I go next! And yes, the next blog will consist more about the code that I wrote and the things I did at GoJek.

Week 1 - Experience as an intern at GoJek

2018-05-27T18:00:00+00:00

It began, the week I’ve been waiting for since months! I was finally interning with GoJek in Bangalore.

Before I go on and talk about my experience with GoJek and the internship, I wrote about how I got it in the first place. Read it here.

20^th May it was, I had packed my bags and was ready to leave for the airport. A quick check of the things I need to carry and the next moment I was driving.

I reached the airport at around 0800 hours and my flight landed in Bangalore at around 1400 hours. I arrived at accommodation provided by GoJek and it was far better than my expectations. It was a fully furnished 3 BHK apartment. And the view outside the balcony was jaw-dropping. Just look at the image below!

I did nothing that day except unpack my bag and I was so tired that I went to straight to take a nap. When I woke up I got a treat by my seniors.😁

Day 1

The next day at 1100 hours I was in the GoJek office waiting with fellow interns for the HR person to brief us. The first thing that she said was that the guys are bringing your laptops shortly and we were asked to fill-up the NDA. And there it was, a 15” MacBook Pro with maxed out specs and the awesome touch bar. 😍

I went ahead and got the initial setup done. I had used my friends MacBook Air before so I knew about brew and installed it. But then I came to know that there is something called brew cask also. And then I was assigned to a team along with Rajat Mangla and Venkata Sainath Thota. Deepesh Naini is our mentor on the current project, he briefed us about GoJek and answered our doubts.

One thing different from most other offices is the culture here at GoJek. One can be comfortable in their own clothing style. And anyone can go grab snacks, coffee, soft drinks, etc. anytime they want. And there is a recreation room too with pool table, PS4, table-tennis table, etc.

We then met Aayush, he showed us our desk and we continued the process of setting up our machines. I finished the setup and asked Aayush what to do next? He gave the three of us access to one of the repositories which are a part of the Go-Food platform. I had to go through it and understand the project structure, it was built on top of Ruby On Rails. The syntax was new to me but since I had worked previously on Django so I was aware of MVC architecture and my blog is built using Jekyll so I knew a little about Ruby’s project structure and bundler, etc.

But obviously, the project wasn’t completely clear to me. The next step was to question everything! What is this? Why is this so? etc. Aayush asked us every possible question. I could answer only a few, in others, I was confused or simply didn’t have the answer.

I learned quite a lot during the next three days. RSpec was completely new to me, I was aware of TDD but RSpec introduced me to BDD. After the initial three days of learning Ruby, we were given a project which was to be done in Go and will involve creating an API for image manipulation. I again looked at the repository for the new project and tried to build it. I went through the code and then paired with Rajat and Sainath to write our first piece of code as an intern. We pushed the code to the remote server and created a merge request. And yes, having an external monitor while coding is a bliss. 🙃

The learning experience has been great so far and there is so much more to explore. I’m meeting new people, many more yet to meet.😅 I’ve made new friends, all of them are great. I’m liking Bangalore so far, especially the weather. It’s been raining almost every day since I’ve arrived. I went to my cousin’s home for the weekend and this is pretty much it about the first week.

How I got the internship at GoJek

2018-05-27T16:00:00+00:00

It was October of 2017 I guess and campus interviews for internships were going on and I could easily spot my seniors wearing formals inside the campus. And believe me, I couldn’t even recognize one of the seniors whom I already knew 😛. One morning I asked my batch-mate that I wanna see the interview process! He was a volunteer in Student Placement Cell. He said, no problem, come with me tonight.

I went and it was GoJek taking the coding round. I saw a familiar face, Aayush Sharda he was, one of my seniors from Defeat The Beat, our dance club. Abhinav Rai, another of my seniors was also there. Abhinav asked me to take the question paper and do it in 1 hour 😂. It was meant to be done in 3 hours though. I read the problem statement, it didn’t seem that difficult to me, I skipped it anyway 😅.

After the coding round was over I asked Aayush that what do they look for while examining the candidates. He replied, we look at various aspects like how do they name variables, how do they react to input while pair programming, what is their coding style, are they focusing on getting shit done or spending their time in optimizing it, etc. to name a few. It was surprising but only 1 person was able to finish all the milestones of the problem statement. Why? Because most of the people were stuck at optimizing the first milestone! 😅

After listening to all this, I decided to show the repository of one of my projects I did that summer to him. He asked me to send the link. Later that night, I received a call from him and he asked me to come over to the guest house where he was staying. He asked me questions which were a little out of context to me at first but he was just making sure that I actually wrote the code. 😅

For the next half an hour he was just telling me about what GoJek is and what does it do. And then came the question, “Tu GoJek k saath internship karega?” like if I would say no! Hell yeah, who would deny an internship in a unicorn. 😅

He then asked me to brush up all the things that I’ve used in that project because next afternoon was my interview scheduled. I barely slept that night and I went through almost everything I had used to code that project be it Git, MVC, OOP concepts or Django’s ORM. I read everything! Next day I attended all my classes and it was time for the interview. I was nervous that I shouldn’t blow up this opportunity.

I called Aayush and he was not gonna be a part of the interview. I was in the guest house now. I was so consumed by many things that I don’t even remember now who opened the door for me and asked me to have a seat. 😂

A few minutes later I was joined by Puneet Goyal and Shishir Joshi. The two were very casual, probably to make me comfortable. The interview started and both of them told me about themselves one after the other like what do they do, briefly about their journey in the tech world so far. And next was me! The questions were mostly about how I was motivated to do that project? Why did I choose Django? How did I cope with problems I had while working on the project? etc.

There were no hard-core coding questions or questions about data structures and algorithms, etc. maybe because I was a sophomore back then and the course was still ongoing. The interview came to an end and we shook hands and Puneet told me that he’ll let me know if we can work on a project together and hack something cool. I left the guest house and while my way back to the hostel room I was thinking, “I messed it up” because there was no code-related questions. 😅

I messaged Aayush that I think I blew up. He replied that everything was great and he’ll convey the details later and I was not supposed to tell anyone yet. I was still not believing all the things that happened and moved on.

2 days later Ashish Sahu, a senior congratulated me! I was like, for what exactly? He said, “Abhinav ne nahi btaya kya?”. I messaged Abhinav about it. He laughed and said, “Room par aake t-shirt le jana” 😂. Now I believed that yes I made it through. I thanked Aayush for the opportunity and asked if I can tell my parents now, “bata de or t-shirt pehen ke ghoom”, he said.

So this is how I got the internship. This was primarily because I was at the right place at the right time. Hadn’t I went to see the interview process, I probably wouldn’t get this opportunity. Thanks to Kuldeep and Nikhil!

The next 1-2 days went coping with memes that my jigri dost were targeting at me.

Do read my first week’s experience here

TDD (Test Driven Development), why bother?

2017-12-20T17:24:00+00:00

Recently I came to know about TDD (Test Driven Development), a development methodology in Agile Software Craftsmanship. At first, I was not even sure that how can we write tests to detect bugs in the code, because we write very specific tests that don’t cover every possible use-case. I didn’t understand the purpose of such tests at all. But then digging more into the topic I discovered what is the utility of these tests which I’m going to share in this article.

What is TDD?

You already know the full-form of TDD. Now, TDD is a process that is used to create software with very small repetitive cycles which include: analysis of the requirement, converting that requirement into a very specific test case which shall fail, then we write the production code to make the test pass, we refactor the code and then again repeat the cycle with new requirement. After each cycle, all the tests must pass. We’re not that great at coming up with a code that has correct behaviour and correct structure simultaneously. To overcome this, we follow the RGR cycle in TDD, while always keeping the three laws of TDD in mind.

The Three Laws of TDD

These laws lock a developer into a cycle that is short but is very crucial to maintain. They are:

You may not write production code until you have written a failing unit test.

You may not write more of a unit test that is sufficient to fail, and not compiling is failing.

You may not write more production code that is sufficient to pass the currently failing test.

The purpose of these laws is just to provide line-by-line granularity to the code. Almost every second you keep these laws into consideration.

RGR Cycle

Red-Green-Refactor cycle is repeated after every complete unit test or after a couple of the three laws cycles. They are:

Write a failing unit test

Write production code that makes the unit test pass, by any means necessary

Clean up the mess, i.e. refactor the code

Source: Self-created.

Purpose of RGR cycle is to write clean code subject to constraints. As you write unit tests, you specify the behaviour of the software. And then you write production code which is constrained by the unit tests, so you can structure your production code while maintaining the behaviour of the software. Refactoring is done after each cycle, it is not to be left after the end of the project. It is this cycle that makes it easy to make changes in the code at any stage a lot easier while still maintaining the behaviour required.

Specific/Generic Cycle

This cycle is observed after every 10-15 minutes. It tells you that

As the tests get more specific, the production code gets more generic.

We’ll understand soon what this means with the help of an example. But in a nutshell, when you add even finer granularity to the unit tests, you should write more generalised production code to make the test pass.

Now let’s take the example of building a stack class using TDD.

I’ll use python in this example. Under a directory, I created two files viz. stack.py and test_stack.py :

stack.py contains the production code and the class Stack
test_stack.py contains unit tests and the class TestStack

So let’s say the very first test is that the size of a new stack should be zero. Note that tests in python should be prefixed with test_

Create an interface method like this.

Now, run the automated tests provided by unittest module in python by command python -m unittest. Let me make the test fail by returning a value -1

I can make this test pass simply by returning the desired value 0, recall point 2 of RGR cycle.

Stack size after push is one

Let me write another test which checks if the size of the stack is 1 or not after one push.

Let’s make the test pass by simply incrementing the value of _size in .push() method and return that variable in .size() method.

Take a moment and notice that I replaced a constant value with a more generic variable. Also, I need to refactor the code now as I’ve duplicated code in both the tests. I can do this by moving the stack initialisation part in setUp method of the class TestStack, this method is called before running every test in the class.

Stack size is zero after a push and a pop

If I push an element and then pop it, the size should be zero after that.

Make this test pass by simply decrementing the value of _size.

Stack raises underflow error

What if the stack is empty and I try to pop a non-existing element? Yes, it should raise an error.

I write the following production code to make this pass by checking for the size equal to zero.

Stack raises overflow error

Great going till now, but what if the stack has a specific capacity and I can’t exceed its maximum capacity? I write another test just to check that and it should raise OverFlowError if pushed on a full stack.

Again, I’m introducing a variable _capacity in the constructor and I’ll check for it when I push on the stack.

Pop last element pushed into stack

Okay, till now every test is passing. But this stack is nowhere close to the actual definition of a stack. So, let me test if zero is popped when I had pushed it before.

This test will pass by simply returning 0. Note the point 3 of TDD laws here.

But what if a more general number, say 1 is pushed? It should return it when popped. I’ll write a test just for that now.

I’ll create a private variable _element and update it in .push() method and return it when popped.

True LIFO operation

Once again notice that I replaced the constant 0 with a more generic variable _element. But now, let’s make this Stack class really perform the Last-In-First-Out operations.

I will now modify the production code in the following way to make the test pass.

Did you notice what happened? I modified _element variable into a more general data type, an array. Now recall Specific/Generic cycle, as the unit tests become more specific, the production code gets more generic. I hope you understand now what was meant before. Also, the three laws were followed at every step of the cycles, recall that I returned a constant 0 to make the test test_size_is_zero_ after_push_and_pop pass. I didn’t generalise it by returning a variable right away. I just somehow made the test pass, that’s it. Finally, I can again refactor the code and since python provides some shortcuts with arrays, I can get rid of _size variable and all the unit tests still pass.

In other languages like C++, you’ll not get rid of _size and instead, can use it as an array index. In this way, even the initial code that I wrote is not a waste of time. I’m simply modifying the existing at every cycle and making sure the tests pass. None of the code that we write to pass the early tests is wasted code. It’s just incomplete and not generalised enough. After every cycle, the code evolves and becomes more general.

References:

The Clean Code Blog
Book: Clean Code by Robert C. Martin

Ajat Prabha

Unleashing xLoad: Your Ultimate Data Loader for Go Structs!

🤷‍♂️ Why xload?

Getting Started 🛠️

How does xload work its magic? 🎩✨

📝 Dive into Examples & Tutorials

🌍 1. Loading from Environment Variables

🎨 2. Using Custom Types

🪄 3. Nested Structs

📦 4. Inbuilt Loaders

🛠️ 5. Custom Loaders

🔍 Spotlight: CRUD-based List API

Before xload:

After xload:

🌐 Internationalization with Lokalize 🌐

🎉 Wrap Up

Introduction to xrun: A Flexible Package for Managing Component Lifecycles in Go

Introducing xrun

Getting Started

Features of xrun

Component

ComponentFunc

Manager

Creating and Using Components

Combining Components with xrun.All

Extending xrun: Custom Component Examples

HTTP Server

gRPC Server (Experimental)

Conclusion

Making a DIY Sound System that looks minimalistic

Studying about sound systems

The DIY Power Supply

Writing Kubernetes Operator with an API Server for Custom GUI

Background

Why make an operator?

Say Hello to Operators (CRD + Controller)

Prepare Environment

Setup

API Definition

Modifying Resource

Implementing Controller

API Server and GUI

Week 4 - Load testing with Vegeta and trip to Puducherry

Day 1

Day 2-3

Day 4-5

Weekend

Week 3 - Lots of code and learning

Day 1

Day 2-3

Day 4

Learning session

Day 5

Week 2 - Exploring Golang and the trip to Coorg

Day 1

Weekend

Raja’s Seat

Nisargadhama

Week 1 - Experience as an intern at GoJek

Day 1

How I got the internship at GoJek

TDD (Test Driven Development), why bother?

What is TDD?

The Three Laws of TDD

RGR Cycle

Specific/Generic Cycle

Now let’s take the example of building a stack class using TDD.

Stack size after push is one

Stack size is zero after a push and a pop

Stack raises underflow error

Stack raises overflow error

Pop last element pushed into stack

True LIFO operation

References:

Features of `xrun`

Combining Components with `xrun.All`