Shubham's Blog

Migrating blogs to Hugo

Wed, 12 Mar 2025 11:34:03 GMT

This is just a notification that my new blogs will be posted on the new site. The site URL will still remain the same - blog.shubham.codes

Using Jmespath in Emacs

Mon, 05 Feb 2024 18:30:00 GMT

Introduction

When you have a small JSON file, it is quite easy to look for what you want. But querying a large JSON data is very troublesome.

This is where people use tools like Jmespath to filter and transform data into their liking. This post is about a small wrapper over jp CLI utility that you can use while working on JSON files in Emacs.

Installing JP

Linux

On Linux, you can install the utility by first downloding the binary and then installing it.

> sudo wget https://github.com/jmespath/jp/releases/latest/download/jp-linux-amd64 \
>   -O /usr/local/bin/jp  && sudo chmod +x /usr/local/bin/jp

Mac

On Mac, you can install Jmespath CLI using brew.

> brew install jmespath/jmespath/jp

Adding Jmespath recipe in Emacs

This is my first recipe that I published on MELPA.Here are the steps to install it in Doom using straight. You can use any package manager to install it using MELPA repository.

Doom Emacs

On Doom, you can just mention the below recipe in your package.el.

> (package! jmespath)

Also, add the below line in your config.el.

> (use-package jmespath)

Using Jmespath

There is an interactive function, jmespath-query-and-show that you can use to query the currently opened buffer or a file. To use it with current buffer, you can simply call this function and enter your query. The output will be shown on a new buffer with name JMESPath Result.

To use this with a different file, you can set the Universal Argument using C-u or SPC-u (using evil). Then you can enter the file to execute the query on.

To learn more about JMESPath, visit Jamespath page.

Running PMML models in Erlang using NIF and C++

Sun, 15 Oct 2023 05:10:32 GMT

Introduction

Erlang is a great language for building concurrent systems that are fault-tolerant and scalable. But it lacks some of the libraries that are available in other languages. One such example is using PMML files for machine learning models. At the time of writing, Erlang doesn't have a library for parsing PMML files. This is a problem for people who want to use Erlang for building machine learning systems. Here I'll show how to use C++ to build a NIF that can be used in Erlang to parse PMML files. More specifically, I'll use the cPMML library to build a NIF that can be used in Erlang.

Erlang NIF

Erlang NIF provides you a way to define your functions in C/C++ and call them in Erlang code natively. The C/C++ program is compiled to generate a library file that can be used in Erlang. This library is dynamically linked to the Erlang VM and is the fastest way of calling the C/C++ code from Erlang. The disadvantage of the approach is that if the C/C++ code crashes, it will crash the Erlang VM. And you will need to maintain the C/C++ code along with the Erlang code.

Prerequisites

cPMML library requires a version of C++ that supports C++11 standard. Also, make sure you have the required header files on the Erlang side.

You can find them on Mac OS or Linux using the find function. If the header files are located in multiple locations, use the Celler one.

find / -name erl_nif.h | grep erl_nif.h

Hello Nif program

C/C++ code

Header file

You need to include the below header file to use the Erlang functionalities. On a lower level, it defines the data structures and environment that Erlang provides to the C/C++ code.

#include <erl_nif.h>

Function Definition

These functions are called from Erlang code. They must follow a specific structure and return a specific type of value. Here, we will define a function that will return a "Hello, World!" string.

ERL_NIF_TERM is the return type of the function. It is an interface for various return types like binary, tuple, list, atom, etc. Meaning, that you can return any of these types from the function.

ErlNifEnv is the pointer to the Erlang environment. It provides you access to various Erlang functionalities like memory management, exception handling and Erlang term creation. For us, it will help in creating Erlang Terms like string. enif_make_string is the function that will create a string from the C/C++ code.

The argc and argv provide the number of arguments and the arguments passed to the function from Erlang code.

Below is the function definition for hello world.

static ERL_NIF_TERM hello_world(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {
    return enif_make_string(env, "Hello, World!", ERL_NIF_LATIN1);
}

Export functions

You need to specify the functions that you want to export to the Erlang code. The structure is a list of ErlNifFunc objects. Each object has the name of the function that Erlang sees, the number of arguments and the function pointer.

static ErlNifFunc nif_funcs[] = {
    {"hello_world", 0, hello_world}
};

To initialize the NIF library, you need to call the ERL_NIF_INIT function. This takes in the name of your Erlang module and the exported functions.

Let's call our Erlang module hello_nif.

ERL_NIF_INIT(hello_nif, nif_funcs, NULL, NULL, NULL, NULL)

Final code

extern "C" {
  #include <erl_nif.h>
  static ERL_NIF_TERM hello_world(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {
      return enif_make_string(env, "Hello, World!", ERL_NIF_LATIN1);
  }
  static ErlNifFunc nif_funcs[] = {
      {"hello_world", 0, hello_world}
  };
}

ERL_NIF_INIT(hello_nif, nif_funcs, NULL, NULL, NULL, NULL);

Erlang code

Erlang module

We will name our Erlang module hello_nif.erl.

-module(hello_nif).

Define your NIF functions

This specifies the functions that are exported from the C/C++ code.

-export([hello_world/0]).
-nifs([hello_world/0]).

Load the library on module load

If you compiled your C/C++ code to a library named hello_nif.so, you can load it using the load_nif function.

-on_load(init/0).

init() ->
    ok = erlang:load_nif("./hello_nif", 0).

Fallback function

The C/C++ code can crash and you need to handle that in the Erlang code. These functions will run when the C/C++ code crashes. You should name each of these functions with the same name in the C/C++ code.

hello_world() ->
    exit(nif_library_not_loaded).

Final code

-module(hello_nif).
-nifs([hello_world/0]).
-on_load(init/0).

init() ->
    ok = erlang:load_nif("./hello_nif", 0).

hello_world() ->
    exit(nif_library_not_loaded).

Compiling and Running

To compile the C/C++ code you can use gcc.

Mac OS

gcc -o hello_nif.so hello.c -I /usr/local/lib/erlang/erts-13.2.2.2/include/ -bundle -bundle_loader /usr/local/lib/erlang/erts-13.2.2.2/bin/beam.smp

Linux

gcc -o hello_nif.so hello.c -I /usr/lib/erlang/erts-13.2.2.2/include -shared -fpic

On Erlng, you can call the hello_world function.

> c(hello_nif).
> hello_nif:hello_world().
"Hello, World!"

cPMML NIF

We saw how to create a simple NIF that returns a string. Now, let's create a NIF that can take input and run prediction using a PMML model file. For this, we will use the same model file we created in the previous post.

We have a PMML model for linear regression, y = 2x + 1 and we want to predict the value of y for a given value of x. Keeping it short, we will expose a function called, predict that will take the input and return the output.

C/C++ code

Assuming you can compile cPMML files as described in the previous post.

Header files

#include <iostream>
#include <vector>
#include <string>
#include <map>
#include "cPMML.h"

using namespace std;

PMML Parser class

We want to load the model once and use it for multiple predictions. For this, let's create a class that will load the model and call predictions on it. We will maintain only one instance of this class and use it for all the predictions.

class PmmlModelParser {
private:
  cpmml::Model model;

public:
    PmmlModelParser(const string& modelname) {
        model = cpmml::Model(modelname);
    }

    string predict(const unordered_map<string, string>& x_input) {
        return model.predict(x_input);
    }
};

// Global variable
PmmlModelParser *pmmlModelParser = nullptr;

NIF implementation

We will create two functions, init and predict. The init function will take the PMML file as input and load it. The predict function will take the input and return the output.

The below code describes the structure of the NIF functions.

extern "C" {
    #include <erl_nif.h>
    
    static ERL_NIF_TERM init(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {}

    static ERL_NIF_TERM predict(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {}

    static ErlNifFunc nif_funcs[] = {
        {"init", 1, init},
        {"evaluate", 1, predict}
    };
}

ERL_NIF_INIT(lr_model, nif_funcs, NULL, NULL, NULL, NULL)

init function

The init function will take the PMML file as input and load it. We will use the enif_inspect_binary function to get the PMML file as a binary. Then we will convert it to a string and pass it to the PmmlModelParser class.

static ERL_NIF_TERM init(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {
    ErlNifBinary input_bin;
    if (!enif_inspect_binary(env, argv[0], &input_bin)) {
        return enif_make_badarg(env);
    }
    string input(reinterpret_cast<char*>(input_bin.data), input_bin.size);
    pmmlModelParser = new PmmlModelParser(input);
    return enif_make_atom(env, "ok");
}

predict function

The predict function will take the input and return the output. The input will be a string and the output will be a string on the Erlang side.

But on the C/C++ side, we will convert the input to a map and pass it to the PmmlModelParser class because cPMML library expects a map as input. Don't get intimidated by the code below, it's just converting the Erlang map to a C++ map by iterating over all the keys and value pairs.

For prediction, we will need a map with the key as X and value as the input.

I could have just used a binary input as X and converted it to a string on the C/C++ side. But, to make this more general, I am converting the Erlang map to a C++ map. Now this can be used for any PMML file and not just a specific one.

static ERL_NIF_TERM predict(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {
  unordered_map<std::string, std::string> cpp_map;

  if (!enif_is_map(env, argv[0]))
  {
    return enif_make_badarg(env);
  }

  ErlNifMapIterator iter;
  if (enif_map_iterator_create(env, argv[0], &iter, ERL_NIF_MAP_ITERATOR_FIRST))
  {
    do
    {
      ERL_NIF_TERM key, value;
      if (enif_map_iterator_get_pair(env, &iter, &key, &value))
      {
        ErlNifBinary key_bin, value_bin;
        if (enif_map_iterator_get_pair(env, &iter, &key, &value))
        {
          char key_str[64], value_str[64];
          if (enif_get_string(env, key, key_str, sizeof(key_str), ERL_NIF_LATIN1) &&
              enif_get_string(env, value, value_str, sizeof(value_str), ERL_NIF_LATIN1))
          {
            cpp_map[key_str] = value_str;
          }
        }
      }
    } while (enif_map_iterator_next(env, &iter));
    enif_map_iterator_destroy(env, &iter);
  }

  string ret = pmmlModelParser->predict(cpp_map);
  return enif_make_string(env, ret.c_str(), ERL_NIF_LATIN1);
}

Erlang code

We will name our Erlang module lr_model.erl.

-module(lr_model).
-export([init/1, evaluate/1]).
-nifs([init/1, evaluate/1]).
-on_load(init/0).

init() ->
    ok = erlang:load_nif("./lr_model", 0).

init(PmmlFile) ->
    exit(problem_loading_nif).

evaluate(Input) ->
    exit(problem_loading_nif).

Compiling and Running

Compiling the C/C++ code will now take an extra parameter to include cPMML library. As the model is predicting the values for y = 2x + 1, we should get ~3 for x = 1 and ~1 for x = 0.

g++ -std=c++11 library.cpp \
-o lr_model.so \
-lcPMML \
-I /usr/local/lib/erlang/erts-13.2.2.2/include/ \
-bundle \
-bundle_loader /usr/local/lib/erlang/erts-13.2.2.2/bin/beam.smp

> c(lr_model).
> lr_model:init(<<"./lr_model.pmml">>).
> lr_model:evaluate(#{"X"=>"0"}).
"0.967265"
> lr_model:evaluate(#{"X"=>"1"}).
"3.007469"

Conclusion

In this blog, we saw how to use C/C++ libraries in Erlang code. We used a PMML file of a linear regression model to predict the values of y = 2x + 1 for a given value of x in Erlang where the code was implemented in C++. We saw how to use the cPMML library to parse the PMML file and use it for predictions.

Running AI/ML predictions in CPP using cPMML library

Sun, 01 Oct 2023 23:39:47 GMT

Introduction

PMML is a markup language to save your AI/ML model files so that you can use them for predictions later on (maybe during production). cPMML is a library created by the AmadeusITGroup to parse and run predictions in C++. In this blog, we will train a linear regression model in python and generate a pmml file and then we will run our predictions in C++.

Creating a model file

Dependencies

We will need pandas, numpy, scikit-learn and sklearn2pmml.

pip install pandas numpy scikit-learn sklearn2pmml

Imports

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn2pmml import sklearn2pmml
from sklearn2pmml.pipeline import PMMLPipeline
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

The model

Dataset

For keeping things simple, let's train a linear regression model to match the equation, y = 2x + 1. We can generate a random dataset for this equation.

X = np.random.rand(100, 1)
Y = 2 * X + 1 + 0.1 * np.random.randn(100, 1)

Test/Train data

Next, we'll divide the data into test and train datasets.

df = pd.DataFrame({'X': X.flatten(), 'Y': Y.flatten()})
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)
X_train = train_df[['X']]
y_train = train_df['Y']
X_test = test_df[['X']]
y_test = test_df['Y']

Training the model

For training the model, we can get the model from scikit learn library and use the dataset we generated above. We can also check the mse to get an idea of the model's accuracy.

pipeline = PMMLPipeline([
    ("regressor", LinearRegression())
])

pipeline.fit(X_train, y_train)

y_pred = pipeline.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
print(f"Mean Squared Error: {mse}")

Saving the pmml file

If you are satisfied by the performance of your model, you can export the model as a pmml file. We will save the model with the name, lr_model.pmml

sklearn2pmml(pipeline, "lr_model.pmml", with_repr = True)

Using the model file

The main step of focus in this blog is using the model in C++ program. For this, you will need to isntall the cPMML library.

Installing cPMML

To install the libray in your system, you just need to run the below command. This will run cmake, so you should have cmake installed in your system.

git clone https://github.com/AmadeusITGroup/cPMML.git && cd cPMML && ./install.sh

For Mac M1

I ran into some problems while installing this on Mac M1. Here are the steps to install this effortlessly.

Ensure you have the latest version of cmake installed in your system.
You can edit the install.sh script to remove -j 4 flag from the cmake -j 4 .. command. This will turn off the multi processing.
The last line of the install.sh script is sudo ldconfig. Change this to sudo update_dyld_shared_cache. This installs the .dylib or .so library files to proper destination.

Running the predictions

Include the library

The first thing is to import the library.

#include "cPMML.h"
#include <iostream>

Load the model

Then you can load the model.

int main() {
  cpmml::Model model("lr_model.pmml");
  return 0;
}

Start predictions

The cPMML library takes input as an unordered_map of strings. For us, there is only one input which is X.

int main() {
  cpmml::Model model("lr_model.pmml");

  // This shoule yield a value close to 1
  std::unordered_map<std::string, std::string> input1 = {
    {"X", "0"}
  };


  // This should yield a value close to 21
  std::unordered_map<std::string, std::string> input2 = {
    {"X", "10"}
  };

  std::cout<<"X = 0 Y = "<<model.predict(input1)<<'\n';
  std::cout<<"X = 10 Y = "<<model.predict(input2)<<'\n';
  
  return 0;
}

Compilation

You can compile the code by including the cPMML library.

> g++ -std=c++11 predict.cpp -o predict.o -lcPMML
> ./predict.o
X = 0 Y = 0.967265
X = 10 Y = 21.369305

Conclusion

In this blog, we saw how to store your model as a PMML file and load it in C++ using cPMML library. You can view the code for the above here.

Embedded Jetty server with handlers for legacy Java applications

Fri, 01 Sep 2023 18:30:00 GMT

import PlantUML from '../../components/PlantUML.astro'

Introduction

Jetty is a very powerful yet lightweight Java library since ages that helps you create servers and clients for HTTP (literally all the versions), Web Sockets, OSGI, JMX, JAAS and much more.

This post will deal with embedding a Jetty server in a Java application and adding handlers for different requests. I'm assuming you have a basic understanding of Java and Maven and have familiarity with different HTTP methods like GET, POST, PUT, DELETE, etc.

Jetty Server Architecture

An incoming request is handled using 4 components - Threadpool, Connectors, Handlers and Server. Server is the core of the system that handles the management of servers and the entire lifecycle of the server. Connectors help in accepting various requests over different protocols like HTTP, HTTPS, etc. Handlers are the components that process the incoming request. Threadpools are like tiny workers that make the system multi-threaded and help in handling multiple requests at the same time.

An incoming request is first accepted at the Connector. Then it is sent to the Server which connects it to the Handler where the main response is generated. And the response is generated using a particular thread as per Threadpool.

<!plantuml/>

Using Jetty in a Java application

New Java App

First, let's create a new Java app using Maven.

> mvn archetype:generate \
-DgroupId = com.schwiftycold.poc \
-DartifactId = poc_jetty_server \
-DarchetypeArtifactId = maven-archetype-quickstart \
-DinteractiveMode = false

Add the following properties to your POM file. This will set the Java version and language encoding for the app.

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>20</maven.compiler.source>
    <maven.compiler.target>20</maven.compiler.target>
  </properties>

Add Jetty dependency

Add the following dependency in your pom.xml file.

<dependencies>
  <dependency>
    <groupId>org.eclipse.jetty</groupId>
    <artifactId>jetty-server</artifactId>
    <version>11.0.3</version>
  </dependency>
  ...
</dependencies>

Main server

The main component of the system is the server class. This class will create a server context and register connectors and handlers. Let's call this class MainServer.

Threadpool

There are different kinds of Threadpool offered by Jetty like QueuedThreadPool, ExecutorThreadPool, ScheduledThreadPool, etc.

QueuedThreadPool maintains a fixed number of threads and a queue to manage incoming requests. When a request comes and the thread is busy then it is added to the queue and will be picked when a thread is available. And, due to its nature, this is widely used for handling http requests.

ScheduledThreadPool extends QueuedThreadPool to handle scheduled tasks. It provides us with a scheduler to execute tasks at a certain interval. It is generally used for scheduling background tasks.

ExecutorThreadPool enables you to use custom Executor as the Threadpool for Jetty.

Here, we will be using QueuedThreadPool.

ThreadPool threadPool = new QueuedThreadPool();

Server

The server is the essential component that manages the connectors and handlers. It takes the Threadpool to create a new server instance.

Server server = new Server(threadPool);

Connector

A connector allows us to accept a variety of different protocols like HTTP, HTTPS, Unix domain socket, etc.

HTTP

A simple HTTP connector can be initialized using the ServerConnector class. And you can change the port to listen to using, setPort function. By default, the port is 8080.

ServerConnector connector = new ServerConnector(server);
connector.setPort(9120);
server.setConnectors(new Connector[]{connector});

HTTPS

For using HTTPS you'll need to register the SSL/TLS certificate. We won't be using this here but a quick look into this will give us a glance of the infinite possibilities.

First, you will need the sslContextFactory which defines your SSL/TLS configurations.

SslContextFactory sslContextFactory = new SslContextFactory();
sslContextFactory.setKeyStorePath("/path/to/keystore.jks");
sslContextFactory.setKeyStorePassword("keystore-password");
sslContextFactory.setKeyManagerPassword("key-password");

ServerConnector httpsConnector = new ServerConnector(
    server,
    new SslConnectionFactory(sslContextFactory, "http/1.1")
);
httpsConnector.setPort(8443);
server.setConnectors(new Connector[] {httpsConnector});

Handlers

Handlers are the ones that will be creating or response. For this, let's create a new singleton class. Let's set this to null for now.

server.setHandler(null);

Final code

public class MainServer {
  public static void startServer()  {
    try {
      ThreadPool threadPool = new QueuedThreadPool();

      Server server = new Server(threadPool);

      ServerConnector connector = new ServerConnector(server);
      connector.setPort(9120);
      
      server.setConnectors(new Connector[]{connector});
      server.setHandler(null);

      server.start();
      server.join();
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
}

Handler class

Handler class is where we will write our logic to process the request. So this requires a special attention.

Let's create a new class that extends AbstractHandler

Extend AbstractHandler

The AbstractHandler required us to implement a handle function. This function is triggered on every new request.

The handle method uses four parameters. The target parameter denotes the endpoint that was triggered. If we trigger the URL, http://localhost:9120/hi then the target will take the value as /hi.

baseRequest and request denote the same thing but in a different context. baseRequest is Jetty specific whereas request donates the servlet-specific APIs. Here, we won't be using the servlet APIs.

The response is the processed result generated by the server which will be sent to the client.

public class MainHandler extends AbstractHandler{

  @Override
  public void handle(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response)
      throws IOException, ServletException {
        System.out.println("target: " + target);
        System.out.println("baseRequest: " + baseRequest);
        System.out.println("request: " + request);
        System.out.println("response: " + response);     
  }
}

The output of the above code would be something as follows on triggering with curl.

> curl -X POST "http://localhost:9120/hi" -d "p=value"

target: /hi
baseRequest: Request(POST http://localhost:9120/hi)@57ffc3ca
request: Request(POST http://localhost:9120/hi)@57ffc3ca
response: HTTP/1.1 200

Here, you can see the target as /hi, the request denotes the method call and reference to the request object and the response is just 200 which denotes a success. We can manipulate this information to create responses based on different inputs.

Manage different methods

We can get the request method using the response object by calling the function, getMethod.

public void handle(...) {
    if ("POST".equalsIgnoreCase(baseRequest.getMethod())) {
      System.out.println("POST request received");
    }
    else if ("GET".equalsIgnoreCase(baseRequest.getMethod())) {
      System.out.println("GET request received");
    }
}

Now, calling curl as above will print POST request received on the server side.

Extract the query parameters

The query parameter and its values can be extracted from the base request object by using the getParameterNames method. Here, I'm parsing the values and storing them in a Map for accessing them later.

if ("POST".equalsIgnoreCase(baseRequest.getMethod())) {
  System.out.println("POST request received");
  Map<String, String> queryParams = new HashMap<>();

  for (Enumeration<String> e = request.getParameterNames(); e.hasMoreElements();) {
    String name = e.nextElement();
    String[] values = request.getParameterValues(name);
    for (String value : values) {
      queryParams.put(name, value);
    }
  }

  System.out.println("Query params: " + queryParams);
  ...

Now, you can see the below response on the server side.

POST request received
Query params: {p=value}

Generating the result

The last part is generating the result for specific triggers. This is done by using getWriter function to write to the response. You can also set the status of the response using setStatus method. You can create more if statements for each endpoint.

try {
  if (target.startsWith("/hi")) {
    response.setStatus(HttpServletResponse.SC_OK);
    response.getWriter().println("Hello World");
    baseRequest.setHandled(true);
  }
} catch (Exception e) {
  e.printStackTrace();
}

Final handle method

  public void handle(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response)
      throws IOException, ServletException {
    if ("POST".equalsIgnoreCase(baseRequest.getMethod())) {
      System.out.println("POST request received");
      Map<String, String> queryParams = new HashMap<>();

      for (Enumeration<String> e = request.getParameterNames(); e.hasMoreElements();) {
        String name = e.nextElement();
        String[] values = request.getParameterValues(name);
        for (String value : values) {
          queryParams.put(name, value);
        }
      }

      System.out.println("Query params: " + queryParams);

      System.out.println("Target: " +target.startsWith("/hi"));

      try {
        if (target.startsWith("/hi")) {
          System.out.println("Target: " +target);
          response.setStatus(HttpServletResponse.SC_OK);
          response.getWriter().println("Hello World");
          baseRequest.setHandled(true);
        }
      }
      catch (Exception e) {
        e.printStackTrace();
      }

    } else if ("GET".equalsIgnoreCase(baseRequest.getMethod())) {
      System.out.println("GET request received");
    }
  }

Connect Handler and MainServer

Before testing your new server, you'll also need to add the handler which can be done using the server's setHandler method.

server.setHandler(new MainHandler());

Now calling the hi endpoint will return Hello World as a response.

> curl -X POST "http://localhost:9120/hi" -d "p=value"

Hello World

Conclusion

In this post, we saw how to configure Jetty server and set handlers to generate responses for a particular endpoint trigger. This looks like a long post but it revolves around just creating a 'Hello World' server using Jetty. You can find the code for the above here.

Communication b/w Java (Maven) and Erlang (rebar3) using Jinterface

Sat, 26 Aug 2023 09:38:06 GMT

Introduction

Jinterface is a way to make Java programs behave like Erlang nodes. This ensures seamless communication between Erlang and Java programs. Jinterface exposes some APIs that you can use to create Erlang data structures and send them to Erlang nodes.

How to use Jinterface

Installation

For using Jinterface you need a jar file provided by Erlang called OtpErlang.jar. You can find the location of the jar by opening erlang shell and typing the below command. This will provide you with the location of the jar file.

> code:priv_dir(jinterface).
"/usr/local/lib/erlang/lib/jinterface-1.13.2/priv"

$ ls "/usr/local/lib/erlang/lib/jinterface-1.13.2/priv"
OtpErlang.jar

It may happen that the file isn't there. This is because Jinterface is not installed by default. In this case, you will need to install Erlang from the source with Java 8+ installed on the path.

If you have installed Erlang using homebrew on macOS then it might not be there.

So first make sure, you have Java runtime installed. If not, then please install this first.

$ java -version
openjdk version "20.0.1" 2023-04-18
OpenJDK Runtime Environment (build 20.0.1+9-29)
OpenJDK 64-Bit Server VM (build 20.0.1+9-29, mixed mode, sharing)

Remove your old installation of Erlang if you have any. Then clone the Erlang repository and build it from the source.

$ git clone https://github.com/erlang/otp.git
$ cd otp
$ git checkout maint-25
$ ./configure && make && make install

Also, verify if the jar has been installed correctly as we did earlier.

Creating a Java node

New Maven Project

Let's first create a new maven project.

$ mvn archetype:generate -DgroupId=com.example \
    -DartifactId=myproject \
    -DarchetypeArtifactId=maven-archetype-quickstart \
    -DinteractiveMode=false

And you might also want to update the Java version and encoding by adding/updating the below properties in pom.xml.

<properties>
    <maven.compiler.source>20</maven.compiler.source>
    <maven.compiler.target>20</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
</properties>

Adding Jinterface dependency

The easiest way is to install your jar in m2 and add it to your dependency.

$ mvn install:install-file \
   -Dfile=/usr/local/lib/erlang/lib/jinterface-1.13.2/priv/OtpErlang.jar \
   -DgroupId=com.ericsson.otp \
   -DartifactId=erlang \
   -Dversion=1.13.2 \
   -Dpackaging=jar \
   -DgeneratePom=true

And then add the dependency in pom.xml.

<dependency>
    <groupId>com.ericsson.otp</groupId>
    <artifactId>erlang</artifactId>
    <version>1.13.2</version>
</dependency>

Creating a Java node

This is the fun part. Each Java app will contain a node and a mailbox. We can send the message to the node on a particular mailbox and the node will receive the message and process it exactly like Erlang nodes.

But first, import the required packages.

import com.ericsson.otp.erlang.*;

Let's create a node called, 'java_node' and a mailbox called, 'java_mailbox'.

OtpNode node = new OtpNode("java_node");
OtpMbox mbox = node.createMbox("java_mailbox");

Next, let's say we will be sending the node a tuple containing the Pid of the Erlang node and a message atom called hello. On receiving {Pid, hello} as the message, we will send 'world' as a message to the calling node.

The mbox.receive() function will pause the execution of the program until it receives a message. We are typecasting the received message to OtpErlangTuple because we know that the message will be a tuple and let's extract the Pid and the message atom from the tuple.

// Get the message
OtpErlangTuple erlTuple = (OtpErlangTuple) mbox.receive();

// Parse the message
OtpErlangPid fromPid = (OtpErlangPid) erlTuple.elementAt(0);
OtpErlangAtom atom = (OtpErlangAtom) erlTuple.elementAt(1);

Now we will check if the message is hello and if that is the case, we will send world as a message to the calling node. Messages can be sent using the mbox.send() function.

if (atom.atomValue().equals("hello")) {
    // Create the reply message
    OtpErlangAtom replyAtom = new OtpErlangAtom("world");
    OtpErlangObject[] replyElements = {new OtpErlangAtom("ok"), replyAtom};
    OtpErlangTuple replyTuple = new OtpErlangTuple(replyElements);
    
    // This will send the message to the calling node with Pid as fromPid
    mbox.send(fromPid, replyTuple);
}

Alive function

You might want to create a health checker function to get a reply on pinging the node. This is done by creating a isAlive function and it comes in handy while debugging and in code as a sanity check.

public boolean isAlive() {
        return true;
    }

Final Java code

Your final Java code will look something like this.

package com.example;

import com.ericsson.otp.erlang.*;

public class HelloWorld {
    public boolean isAlive() {
            return true;
        }

    public static void main(String[] args) throws Exception {
         OtpNode node = new OtpNode("java_node");
         OtpMbox mbox = node.createMbox("java_mailbox");
         System.out.println("Node Created. Now, you can communicate with this node.");
         OtpErlangTuple erlTuple = (OtpErlangTuple) mbox.receive();
         OtpErlangPid fromPid = (OtpErlangPid) erlTuple.elementAt(0);
         OtpErlangAtom atom = (OtpErlangAtom) erlTuple.elementAt(1);
         if (atom.atomValue().equals("hello")) {
             OtpErlangAtom replyAtom = new OtpErlangAtom("world");
             OtpErlangObject[] replyElements = {new OtpErlangAtom("ok"), replyAtom};
             OtpErlangTuple replyTuple = new OtpErlangTuple(replyElements);
             mbox.send(fromPid, replyTuple);
         }
    }
}

You can verify if your Java app is being registered as a node using the below commands in the Erlang shell. As we have implemented the isAlive function, we can also ping the node. A return value of pong means success and pang means failure.

$ erl -sname client
> net_adm:names().
{ok,[{"java_node",59873}]}

> net_adm:ping(java_node@GGN002262).       
pong

Erlang Node

Erlang program

On the Erlang side we can just send the message to the java_node and wait for a message.

The GGN002262 part of my node is my hostname. Use the hostname command to get yours.

-module(client).
-export([start/0]).

start() ->
  {java_mailbox, 'java_node@GGN002262'} ! {self(), hello},
  receive

  {ok, Res} ->
     io:format("Java says: ~p~n", [Res])
  end.

Run a distributed service

You will need to start the Erlang runtime as a distributed service. This can be done using -sname flag.

$ erl -sname client
1> c(client).
2> client:start().
Java says: world

Starting EPMD

epmd program comes prepacked with the Erlang distribution. This is the Erlang port manager. This is required for registering nodes.

If epmd is not running then you will get an error something like this, Nameserver not responding on GGN002262 when publishing java_node.

To start the epmd server, you just need to run epmd from your shell.

# Start epmd server in the background
$ epmd&

# Run the Java app
> java -jar my-app.jar

# Verify if the app is getting registered.
$ epmd -names
epmd: up and running on port 4369 with data:
name java_node at port 59873

If your Java app is registered, you will get the above output.

Conclusion

Here, we discussed how to use Jinterface to communicate with Erlang using Java. We saw that Java programs can behave as Erlang nodes and can communicate with each other through message passing. We also learned about epmd, the Erlang port manager.

One thing to consider while using Jinterface is it simulates the Java node as an Erlang node, which makes the communication a little bit slow hence this approach should not be taken for a system that requires high-frequency message transfer.

It is also important to note that by default, this process of message transfer is async which means we will need to wait for Java to process the result and send a message back. There are ways to make this synchronous by creating a mechanism to send a confirmation message from the client side on receiving the processed message.

Overall, this is a great add-on to Erlang that pumps the power of Java into Erlang's world.

Docker Desktop vs Colima on Mac M1 for working with VSCode containers

Fri, 18 Aug 2023 16:01:22 GMT

Introduction

I am assuming that you have a basic understanding of docker and devcontainers and have used them before. As this post revolves around Mac M1, I'd suggest not using this analysis for comparing other systems in general.

VSCode devcontainers are the new way of starting a project development environment. They are a great way to get started with a project without having to install all the dependencies on your local machine. This reduces the setup time and allows you to get started with the project right away.

Devcontainers are powered by Docker and VSCode. VSCode provides the UI and Docker provides the containerization. This means that you need to have Docker installed on your machine to use devcontainers.

While docker is the main engine, Docker Desktop and Colima are the two main options for creating a docker environment on Mac.

Colima is an open-source alternative to Docker Desktop. While both software are free to use, Docker Desktop requires companies to pay for a license if they have more than 250 employees. And also, Docker Desktop can be enhanced with lots of extensions which Colima can't.

I am using docker for mainly 2 things, creating production-level containers and using devcontainers for development. So a lot of Docker Desktop extensions are not useful for me but the performance is.

I analyzed the read/write performance of Docker Desktop and Colima for working with devcontainers and here are the results. I also compared the build time for my blog on both the software.

System Information

I am using a Macbook Air M1 with 16 GB RAM. I have allocated 8 GB RAM and 4 CPUs to Docker Desktop and Colima each. I also kept the disk storage to 60 GB for both the software which would be enough for our testing.

Docker Desktop

I'm using Docker Desktop v24.2 which comes with a 60% improvement in read/write performance (as they say). It's using Apple's VZ for virtualization and the net says it is far more optimized than Qemu. I have no extensions installed on Docker Desktop. And no other container was running at the time of analysis.

Colima

The latest version of Colima is v0.5.5 at the time of writing this article. It was released on May 2023. For the comparison, I'm using the HEAD of the Colima repository to capture any latest improvements.

I tested Colima with both Qemu and VZ virtualization.

Using the HEAD can be unstable compared to the release branches.

Analysis Procedure

I created a simple Java app for writing files of size 1 MB to 1 GB and calculated the time it takes to write each of the files. File sizes are 1 MB, 10 MB, 64 MB, 128 MB, 256 MB, 512 MB, and 1024 MB.

I created an arbitrary 1 GB file and accessed the random positions 1,000 to 1,000,000 times and calculated the average.

I also calculated the time it takes to print lines on the console using sout. I printed 10,000, 100,000, and 1,000,000 lines of 1 to 1000 characters each and calculated the average time.

You can find the Java app here.

My blog is made on AstroJs which generates a static site on build. The build process involves compressing the images, minifying the CSS and JS, MDX to MD conversion, HTML conversion and so on. This uses a lot of read/write operations hence I think it would be a good test of the performance. So I calculated the build time and the first render time of the blog on both systems to get a real feel of the performance.

Changing between Docker Desktop and Colima

I made sure to shut down Colima before starting Docker Desktop and vice versa. Docker desktop can be switched on/off from the UI.

Commands Used

Colima stop and delete settings

colima stop
colima delete

Start Colima

To start Qemu mode

colima start --cpu 4 --memory 8 --arch aarch64 --vm-type qemu

To start VZ mode

colima start --cpu 4 --memory 8 --arch aarch64 --vm-type=vz --vz-rosetta

Results

Write Performance

Docker Desktop

File size	Duration (Worst)	Duration (Best)
1 MB	311 ms	227 ms
10 MB	1998 ms (2s)	1943 ms (2 s)
64 MB	14433 ms (14 s)	12016 ms (12 s)
128 MB	24745 ms (24 s)	23455 ms (23 s)
256 MB	58937 ms (1 min)	52252 ms (52 s)
512 MB	114213 ms (1.9 min)	110060 ms (1.8 min)
1024 MB	262955 ms (4.38)	194817 ms (3.2 min)

Colima (Qemu)

File Size	Duration (Worst)	Duration (Best)
1 MB	573 ms	286 ms
10 MB	3077 ms (3s)	2345 ms
64 MB	19087 ms (19 s)	14116 ms
128 MB	38191 ms (38 s)	26096 ms
256 MB	81071 ms (1.35 min)	67062 ms
512 MB	151242 ms (2.52 min)	159663 ms
1024 MB	293370 ms (4.89 min)	301629 ms

Colima (VZ + Rosetta 2)

File size	Duration (worst)	Duration (best)
1 MB	291 ms	236 ms
10 MB	2113 ms (2s)	2199 ms
64 MB	12453 ms (12 s)	12262 ms
128 MB	25315 ms (25 s)	24603 ms
256 MB	49837 ms (49 min)	50475 ms
512 MB	101883 ms (1.69 min)	100692 ms
1024 MB	198126 ms (3.30 min)	200163 ms

Read Performance

Shockingly, the read performance of Docker Desktop is very bad compared to Colima with a peak of 13 reads/ms. Colima with Qemu has a peak of 729 reads/ms and Colima with VZ has a peak of 705 reads/ms which is almost comparable.

Surprisingly, the read performance of Qemu is better than VZ.

Docker Desktop

number of random reads	Duration (Total)	Average Speed
1,000,000	80593 - 81941 ms	12 reads/ms
500,000	38238 - 39058 ms	12 - 13 reads/ms
200,000	15299 - 15368 ms	12 - 13 reads/ms
100,000	7755 - 7895 ms	12 reads/ms
10,000	726 - 876 ms	11 - 12 reads/ms
1,000	81 - 107 ms	9 - 12 reads/ms

I ran this 5 times and even restarted the docker engine multiple times but the results were the same. If anyone knows why this is happening, please let me know.

Colima (Qemu)

number of random reads	Duration (Total)	Average Speed
1,000,000	1415 - 1419 ms	704 - 707 reads/ms
500,000	686 - 695 ms	696 - 719 reads/ms
200,000	276 - 278 ms	719 - 728 reads/ms
100,000	137 - 141 ms	709 - 724 reads/ms
10,000	15 - 17 ms	666 - 729 reads/ms
1,000	2 - 3 ms	333 - 500 reads/ms

Colima (VZ + Rosetta 2)

number of random reads	Duration (Total)	Average Speed
1,000,000	1493 - 1507 ms	663 - 669 reads/ms
500,000	709 - 721 ms	693 - 705 reads/ms
200,000	284 - 286 ms	699 - 704 reads/ms
100,000	143 ms	699 reads/ms
10,000	15 - 16 ms	625 - 666 reads/ms
1,000	1 -2 ms	500 - 1000 reads/ms

Print Performance

Docker Desktop has the best print performance of all the three with a peak of 4074 chars/ms. Colima with VZ comes second with a peak of 2984 chars/ms.

Docker Desktop

number of lines to print	number of chars printed	total time	Avg (per line)	Avg (per character)
10,000	4,997,048	1346 ms	7 lines/ms	3712 chars/ms
100,000	49.999,881	12272 ms	8 lines/ms	4074 chars/ms
1,000,000	500,324,217	122849 ms	8 lines/ms	4072 chars/ms

Colima (Qemu)

number of lines to print	number of chars printed	total time	Avg (per line)	Avg (per character)
10,000	5,013,996	2019 ms	4 lines/ms	2483 chars/ms
100,000	50,028,339	19253 ms	5 lines/ms	2598 chars/ms
1,000,000	500,593,068	180024 ms	5 lines/ms	2780 chars/ms

Colima (VZ + Rosetta 2)

number of lines to print	number of chars printed	total time	Avg (per line)	Avg (per character)
10,000	5,030,577	1752 ms	5 lines/ms	2871 chars/ms
100,000	50,010,997	16755 ms	5 lines/ms	2984 chars/ms
1,000,000	499,828,875	169597 ms	5 lines/ms	2947 chars/ms

Blog Performance

System	Build time	Initial rendering time
Host	673 - 711 s	42.69 ms
Colima (Qemu)	817 - 836 s	6.7 s
Colima (VZ + Rosseta)	671 - 924 s	4.8 s
Docker Desktop	824 - 828 s	5.6 s

Compared to the host, the build time is almost the same for all the systems but Colima with VZ is the best among them with the fastest build comparable to the host system. None of them came close to the host system in terms of rendering time (as expected).

Summary

Docker Desktop can be a better option for apps that require a lot of printing but fewer Read/Write operations. When it comes to extensive Read/Write operations, Colima with VZ + Rosseta 2 would be my choice because it gives you better write and print performance than Qemu. For my blog, I will be using Colima with VZ + Rosseta 2 because it gives me the best build time and read/write performance.

Syncing your Elfeed saves between multiple systems

Tue, 15 Aug 2023 19:16:14 GMT

Introduction

I recently started using Emacs with Doom Emacs as my primary text editor. I've been using VSCode for years, but I've always been curious about Emacs and I wanted to give it a try. I'm still exploring different modules, and I'm really enjoying the beginner's keybindings<SPC> h r r and <SPC> q R. I was in an endless loop of searching for a good RSS reader for my needs. I tried some Android-based, web-based, and desktop-based readers, but I couldn't find one that satisfied my need.

I wanted a reader that I can switch to easily in my free time. I wanted a reader that I can sync b/w my devices (which doesn't include mobile devices as I don't use them for reading). I wanted a reader that can tag my feeds according to my customizations. And most important of all, I wanted a simple reader that doesn't have a lot of features that I don't need.

I found Elfeed to be the best RSS reader for my needs. It's simple, it's fast, it's customizable, and it's easy to switch to whenever you're free.

The only thing that I didn't like about Elfeed is that it doesn't have a built-in way to sync your feeds between multiple systems. But there are some workarounds that you can use to sync your feeds.

Data storage in Elfeed

Let's first understand how Elfeed stores your data.

Data storage location

By default, Elfeed stores your data inside ~/.elfeed directory. You can change this location by setting the elfeed-db-directory variable.

I'm using Doom Emacs, so I set this variable inside my config.el file. You can do the same with your installation depending on what package manager you use.

(after! elfeed
  (setq elfeed-db-directory "~/.elfeed-data"))
)

Data storage format

ls -la ~/.elfeed-data
drwxrwxrwx  106 shubham.kumar  1729907015    3392 Aug 15 22:28 data
-rwxrwxrwx@   1 shubham.kumar  1729907015  924723 Aug 15 22:52 index

ls -a ~/.elfeed-data/data
00    21    37  ...  f0
06    24    3c  ...  f2
09    27    3e  ...  ff

ls ~/.elfeed-data/data/00
00e8db47f3a5b93b0fbb9b4c31748f607ae7bae5

The index file is a binary file that contains the metadata of your feeds. The metadata includes the title, link, tags, etc. This means that the index file contains the data that you see when you open Elfeed in Emacs.

The data directory contains the actual data of your feeds which are your list of blogs. The data directory contains the hash of the feed as the name of the file.

And just like git file storage, the data directory contains subdirectories named with the first two characters of the hash and the rest of the hash as a subdirectory inside that.

For more detailed information about the data storage format, you can check the docs by the creator, Chris Wellons.

What to sync?

Looking at the storage format of Elfeed, we can say that we can sync our feeds by syncing the index file and the data directory. But the data file will be generated based on your feeds, so we don't need to sync it if you are already syncing your list of blogs.

Plus comparing the size of org file (1.6K) and data directory (3.3K), we can see that the data directory is much larger than the org file. So I think it's better to sync the index file and the org file that contains your list of blogs than to sync the entire elfeed-data directory.

ls -lh ~/.elfeed-data/
total 1808
drwxrwxrwx  106 shubham.kumar  1729907015   3.3K Aug 15 22:28 data
-rwxrwxrwx@   1 shubham.kumar  1729907015   903K Aug 15 22:52 index

ls -lh ~/Documents/org/elfeed.org
-rwxrwxrwx  1 shubham.kumar  1729907015   1.6K Aug 13 08:00 /Users/shubham.kumar/Documents/org/elfeed.org

For me, I'm using org-mode to store my list of blogs that are being synced using version control. So whenever I start using Emacs on my other system, I always update the org files. This ensures my list of blogs is always up to date.

So the only thing I need now is to sync the metadata of my feeds. And for that, I'm using Syncthing.

Syncing with Syncthing

My current setup

I have 2 systems that I use regularly.

My primary system is my Linux Mint system which I use for personal stuff.
My secondary system is my MacBook Air M1 which I use for work.

On both systems, I use Doom Emacs and Elfeed.

Initial syncing (Syncthing on both systems)

<!plantuml/>

This ensures that whenever one of my systems is offline, the other system can sync with my phone. And when both systems are online, they can sync with each other or my phone. This will ensure that the index file is always synced between the systems.

Problem with this approach

The problem with this approach is that I now have to dedicate some amount of storage to my index file. So I now have an unutilized index file on my phone which is taking up space. And I have to make sure that my phone is always connected to the internet and is always running Syncthing. This will also impact my battery life.

I have to check how much effect this has on my battery life (will update the blog after calculating this).

Best approach (Using Raspberry Pi)

I haven't implemented this yet, but I think this is the best approach. If I get any problem with my current setup, I'll implement this.

I'll buy a Raspberry Pi and install Syncthing on it. I'll add the ~/.elfeed-data directory to the sync list while ignoring the data directory. I'll add both systems to sync with my Raspberry Pi and with each other. I'll ensure that my Raspberry Pi is always connected to the internet and is always running Syncthing. I'll also have to port forward my Raspberry Pi so that I can access it from outside my network. For this, I may need to talk to my ISP about static IP and port forwarding capabilities.

Will write a blog about this if I implement this.

Conclusion

In this post, I talked about how I sync my Elfeed feeds between my systems. I talked about the data storage format of Elfeed and how I sync my feeds using Syncthing. I also talked about the problems with my current setup and how I can improve it by introducing a new system to sync files. If you are using Emacs, I highly recommend you try Elfeed as your RSS reader. And if you are already using Elfeed, I hope this post helps you to sync your feeds between your systems.

Inter-language communication between Erlang (rebar3) and Python using Erlport

Sat, 12 Aug 2023 23:40:40 GMT

When it comes to fault-tolerant systems, there are very few languages that can beat Erlang. While Erlang is a great language for building fault-tolerant systems, it is not the best language for building AI/ML applications. There are instances where you would want to use Python for building AI/ML applications and Erlang for building fault-tolerant systems. In such cases, you would want to use Erlang and Python together. This is where Erlport library comes in.

But before understanding Erlport, you need an understanding of ports in Erlang.

Ports

In simple terms, ports are used to communicate with external programs which are not written in Erlang. So if you have a C program or a Python program and you want to communicate with it from Erlang, you can use ports. It's not a restriction that you can only use ports to communicate with external programs, you can also use ports to communicate with other Erlang nodes. Port provides you with a byte-oriented interface. Which means, you send a list of bytes and receive a list of bytes during the communication. This also means you need to handle the encoding and decoding of the data at both ends by yourself.

Erlport

Erlport is a library that internally uses the port mechanism for communication and provides a wrapper around it. This makes the integration of Erlang and Python very easy. Right now, Erlport supports Python and Ruby. In this article, we will be focusing on Erlang and Python implementation.

Installation

If you are using rebar3 for building your Erlang application, you can add the following dependency to your rebar.config file.

...
{deps, [
    erlport
]}.
...

Or you can use the GitHub link for the dependency.

{erl_opts, [debug_info]}.
{deps, [
  {erlport, {git, "https://github.com/erlport/erlport.git", {tag, "v0.10.1"}}}
]}.

You'll also need to tell Erlang to wait for the dependencies to be available before starting the app. This can be done by adding the entry in *.app.src file.

 {applications,
   [kernel,
    stdlib,
    erlport
   ]},

Python program

Let's create an add program in Python and keep the file in the priv directory.

def add(a, b):
    return a + b

Erlang program

The Erlang program will initialize the Python port using python:start/1 function. And will call the add function of the Python program using python:call/4 function. The python:call/4 function takes the Pid of the Python port, the module name, the function name, and the list of arguments to be passed to the function.

Here, my Python module is named, math and the function is add with 2 and 4 as parameters.

-module(main).
-export([call_python/0]).

call_python() ->
  PythonCodePath = code:priv_dir(aiml_model_wrapper),
  {ok, P} = python:start([{python_path, PythonCodePath}, {python, "python3"}]),
  python:call(P, math, add, [2, 4]).

Running the program

Running the program is as simple as compiling and running the Erlang program.

reabr3 compile
reabr3 shell
1> main:call_python().

Conclusion

In this article, we saw how we can use Erlang and Python together using Erlport. This is a very simple example of how you can use Erlang and Python together. You can use this to build complex systems where you can use Erlang for building fault-tolerant systems and Python for building AI/ML applications. At GreyOrange I created an Erlang wrapper for our AI/ML models using the same approach.

Download files via REST API using chunked transfer encoding

Fri, 26 May 2023 01:04:53 GMT

Chunked transfer encoding is a protocol to send data in chunks over HTTP. This allows us to transfer a large amount of data in chunks of small size.

With chunked encoding, the server splits the response into a series of smaller "chunks" of data. Each chunk includes a size indicator followed by the actual chunk data. The size indicator specifies the length of the chunk data in bytes. The chunks are sent to the client one by one and can be processed by the client as they arrive.

Here, we will create a Python server that will send us a heuristic file in chunks and a Java client will read these files and download them over the REST API.

Project structure

The project is a Maven project at root containing a sub-directory for our Python server.

❯ tree .
.
|-- README.MD
|-- pom.xml
|-- python
|   |-- requirements.txt
|   `-- server.py
`-- src
    `-- main
        `-- java
            `-- com
                `-- gor
                    `-- poc
                        |-- App.java
                        `-- FileDownloader.java

Python server

This is a Flask server that exposes a download route.

For a POC, we will just send a particular file called 'heuristics' whenever a request is made to the server. The server will send the files in chunks of 1024 which can be configured as per your need.

# Define the route
@app.route('/download')
def download_file():
    # This is the file we want to download whenever a client requests
    file_dir = 'python/public/stream/data/'
    file_name = "heuristics.bin"
    file_path = file_dir + file_name
    chunk_size = 1024
    
    # Create chunks and yield them whenever required
    def generate():
        # Chunks size of each chink of data     
        with open(file_path, 'rb') as file:
            while True:
                chunk = file.read(chunk_size)
                if not chunk:
                    break
                yield chunk

    response = Response(

        # stream_with_context is a Flask functionality to send files in chunks
        # You can read more about it here https://flask.palletsprojects.com/en/1.0.x/patterns/streaming/
        stream_with_context(generate()),
        mimetype='application/octet-stream'
    )
    response.headers.set('Content-Disposition', 'attachment', filename=file_name)
    return response

# Let's create a heuristic file for our server to send 
# We'll create a random file of size 1GB
dd if=/dev/urandom of=python/public/stream/data/heuristics.bin bs=1G count=1

# The below commands only works on Mac OS
# Check the size of the file generated in bytes
stat -f "%z" python/public/stream/data/heuristics.bin
# 1073741824

# Let's check the md5 of the file
md5 -r python/public/stream/data/heuristics.bin | awk '{print $1}'
# c5b8959732d3359791bcd06ca5a92dc2

# For linux, you can use the below code
# Check the size of the file generated in bytes
stat -c "%s" python/public/stream/data/heuristics.bin

# Check the md5 of the file
md5sum python/public/stream/data/heuristics.bin | awk '{print $1}'

# Let's run the python server now
# We'll use a virtual environment to run the server (You can also use conda))
source .venv/bin/activate

# Install the dependencies
pip install -r python/requirements.txt

# Run the server (and keep it running)
python python/server.py

We can also use curl to check the header.

The presence of Transfer-Encoding: chunked tells us that the server is using the chunked protocol to send the data.

# curl the header
# -I asks for headers only 
# -XGET says it to use GET method (can be omited as it is the default)
curl -I -XGET "http://localhost:5000/download"

# HTTP/1.1 200 OK
# Server: Werkzeug/2.3.4 Python/3.10.10
# Date: Fri, 26 May 2023 00:52:01 GMT
# Content-Type: application/octet-stream
# Content-Disposition: attachment; filename=heuristics.bin
# X-Chunk-Size: 1024
# Transfer-Encoding: chunked
# Connection: close

You can also use curl to download the file.

# Download the file in the current directory
# First let's remove the file if it exists
rm -rf heuristics.bin

# -O says to save the file with the same name as the server
# -J says to use the filename from the server
# -L says to follow redirects (for us it's optional as we are not using any redirects)
curl -O -J -L http://localhost:5000/download

# Also let's verify the file size and md5 of the downloaded file
stat -f "%z" heuristics.bin
# 1073741824

md5 -r heuristics.bin | awk '{print $1}'
# c5b8959732d3359791bcd06ca5a92dc2

You can match the md5 of the server file and what we downloaded to check the successful file transfer.

Java Client

The code for downloading the file using Java is located at com.gor.poc.FileDownloader.

Below is the explanation of the code.

// This is the REST API which will serve you the file
String fileUrl = "http://localhost:5000/download";
String savePath = ""; 

try {
    // Defining the URL
    URL url = new URL(fileUrl);
    HttpURLConnection connection = (HttpURLConnection) url.openConnection();
    connection.setRequestMethod("GET");

    int responseCode = connection.getResponseCode();
    if (responseCode == HttpURLConnection.HTTP_OK) {
        // Get the file name from the header
        String fileName = connection.getHeaderField("Content-Disposition");
        fileName = fileName.substring(fileName.lastIndexOf("=") + 1);
        String filePath = savePath + "/" + fileName;

        // Create an Input stream to download the file
        InputStream inputStream = connection.getInputStream();

        // This is where the file will be saved
        FileOutputStream outputStream = new FileOutputStream("./"+filePath);

        // Read the data in chunks and save it to the file
        byte[] buffer = new byte[4096];
        int bytesRead;
        while ((bytesRead = inputStream.read(buffer)) != -1) {
            outputStream.write(buffer, 0, bytesRead);
        }
        
        outputStream.close();
        inputStream.close();

        System.out.println("File downloaded successfully!");
    } else {
        System.out.println("File download failed. Server returned response code: " + responseCode);
    }

    connection.disconnect();
} catch (IOException e) {
    e.printStackTrace();
}

# You can run the java program using maven 
# First remove the file if it exists
rm -rf heuristics.bin

# Compile and run the java program
mvn clean compile
mvn exec:java -Dexec.mainClass="com.gor.poc.FileDownloader"
# [INFO] Scanning for projects...
# [INFO] 
# [INFO] --------------------< com.gor.poc:stream_download >---------------------
# [INFO] Building stream_download 1.0-SNAPSHOT
# [INFO] --------------------------------[ jar ]---------------------------------
# [INFO] 
# [INFO] --- exec-maven-plugin:3.1.0:java (default-cli) @ stream_download ---
# Downloading file in chunks...
# 27 bytes read
# 997 bytes read
# 25 bytes read
# 999 bytes read
# 25 bytes read

# Get the size and md5 of the downloaded file
stat -f "%z" heuristics.bin
md5 -r python/public/stream/data/heuristics.bin | awk '{print $1}'

The output shows us that the files are downloaded in small chunks like 27 bytes, 997 bytes even though our client is reading a buffer of size 4096. This is because the server is sending chunks at the rate of 1024 bytes which is being consumed by the Java program much faster. This is one disadvantage of using chunk encoding where the client has no control over the data transfer. Even though the client is faster, the download is dependent on the server configurations.

To solve this let's configure our Python server to send 1 MB chunks instead of 1 KB. Restart your Python server.

    file_path = file_dir + file_name
-    chunk_size = 1024  # Adjust the chunk size as per your requirements
+    chunk_size = 1024 * 1024
    
    def generate():

rm -rf heuristics.bin

# Compile and run the java program
mvn clean compile
mvn exec:java -Dexec.mainClass="com.gor.poc.FileDownloader"
# Downloading file in chunks...
# 24 bytes read
# 4096 bytes read
# 4096 bytes read
# 4096 bytes read

Now the file transfer is much faster but still limited because the client is consuming the data at a slower rate than what the server is sending. We would have increased the file transfer rate by configuring the client to read 1 MB.

Let's configure our client to use the exact bytes our server is sending.

For this, we'll send the chunk size used by our server as header metadata.

    response.headers.set('Content-Disposition', 'attachment', filename=file_name)
+   response.headers.set('X-Chunk-Size', str(chunk_size))  # Add chunk size as a header

And on the Java side, we can read this header and set our buffer size to match the chunk size.

    // This is the REST API which will serve you the file
    String fileUrl = "http://localhost:5000/download"; 
    String savePath = ""; 

    try {
      // Define the URL 
      URL url = new URL(fileUrl);
      HttpURLConnection connection = (HttpURLConnection) url.openConnection();
      connection.setRequestMethod("GET");

      int responseCode = connection.getResponseCode();
      if (responseCode == HttpURLConnection.HTTP_OK) {
        
        // Get the file name from the header
        String fileName = connection.getHeaderField("Content-Disposition");
        fileName = fileName.substring(fileName.lastIndexOf("=") + 1);
        String filePath = savePath + "/" + fileName;

        // Get the chunk size from the response headers
+       String chunkSizeHeader = connection.getHeaderField("X-Chunk-Size");
+       int chunkSize = Integer.parseInt(chunkSizeHeader);

        // Create an inout stream to download the file
        InputStream inputStream = connection.getInputStream();
        FileOutputStream outputStream = new FileOutputStream("./" + filePath);

        // Read the data in chunks and save it to the file
+       byte[] buffer = new byte[chunkSize];
        int bytesRead;

        // Check if the server supports chunked transfer encoding
        String transferEncoding = connection.getHeaderField("Transfer-Encoding");
        boolean isChunked = "chunked".equalsIgnoreCase(transferEncoding);

+        // Just to distinguish b/w chunked protocol and normal file transfer
+        if (isChunked) {
+          // Read and write the response data in chunks
+          System.out.println("Downloading file in chunks...");
+          while ((bytesRead = inputStream.read(buffer)) != -1) {
+            System.out.println(bytesRead + " bytes read");
+            outputStream.write(buffer, 0, bytesRead);
+          }
+        } else {
+          // Read and write the entire response data
+          System.out.println("Downloading file as whole");
+          while ((bytesRead = inputStream.read(buffer)) != -1) {
+            System.out.println(bytesRead + " bytes read");
+            outputStream.write(buffer, 0, bytesRead);
+          }
+        }

        outputStream.close();
        inputStream.close();

        System.out.println("File downloaded successfully!");
      } else {
        System.out.println("File download failed. Server returned response code: " + responseCode);
      }

      connection.disconnect();
    } catch (IOException e) {
      e.printStackTrace();
    }

rm -rf heuristics.bin

# Compile and run the java program
mvn clean compile
mvn exec:java -Dexec.mainClass="com.gor.poc.FileDownloader"

# Downloading file in chunks...
# 24 bytes read
# 768664 bytes read
# 279888 bytes read
# 24 bytes read
# 391942 bytes read
# 393112 bytes read

Looks like we are now using the full capability on the client side.

Analysis

Now the only optimization we will require to send the files quickly is on the server side. Let's log the time it takes to download the file while using different chunk sizes on the server.

Chunk size	Time (in ms)
512 B	9337
1 KB	5150
1 MB	465
64 MB	639
128 MB	744
256 MB	740
1 GB	1169

The time taken to transfer the file first decreases as we increase the chunk size and then start increasing on increasing the chunk size.

The above analysis was done on Mac M1 - 2020 Model with 16 GB RAM (4 GB available).

Package Java JNI libraries in a JAR using Maven

Tue, 02 May 2023 11:51:43 GMT

Introduction

If you are using a native library in Java, you can just load the dll or dylib or so files using System.loadLibrary method. But if you are packaging your Java program as a JAR file, the native library will not be available to the System.loadLibrary method. In this article, we'll see the correct way to package the native library in the JAR file using Maven so that it will be accessible in our Java program without any other user intervention.

Project Structure

Maven follows a specific project structure where all the source code is placed in src/main/java and all the resources are placed in src/main/resources. The resources directory is added to the classpath when the project is compiled and packaged. This means you can access the contents of the resources directory directly in the Java program by specifying the path relative to the resources directory.

Let's say you have a native library libhello.dylib in the resources/lib directory. After packaging this will be available at the path lib/libhello.dylib in the JAR file.

Use Native Utils

The default way to load a native library in Java is to use the System.loadLibrary method. The maven package command will generate the jar file of the project. The native library will be packaged in the lib directory inside the jar file which will not be available to the System.loadLibrary method.

Now running our app will require us to extract the library from the jar file and put it somewhere on the system. This can be done using Native Utils. Internally this utility is extracting the native library from the jar file to a temporary directory and loads the library from there. This will keep our hands out of the dirty work of extracting the library and loading it. It provides a method loadLibraryFromJar which can be used to load the native library from the jar file.

Configurations

As Native Utils is not available as a release package, we'll need to add it directly from the GitHub repository. For this, we can use the JitPack service which allows us to add GitHub repositories as Maven dependencies. You'll need to add JitPack as a repository in your POM.XML file.

  <repositories>
    <repository>
      <id>jitpack.io</id>
      <url>https://jitpack.io</url>
    </repository>
  </repositories>

Also, add the dependency for Native Utils. For this, we need to give groupId as com.github.<user-name> and artifactId as <repository-name>. Since we are using the latest commit from the repository, we'll need to specify the commit hash in the version tag. Below is how the dependency will look like in POM.XML.

  <dependency>
    <groupId>com.github.adamheinrich</groupId>
    <artifactId>native-utils</artifactId>
    <version>e6a39489662846a77504634b6fafa4995ede3b1d</version>
  </dependency>

Now, you can confirm the dependency is added by running the maven dependency:tree command.

> mvn dependency:tree

[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ javaunsafe ---
[INFO] com.gor.poc:javaunsafe:jar:1.0-SNAPSHOT
[INFO] +- log4j:log4j:jar:1.2.17:compile
[INFO] +- junit:junit:jar:3.8.1:test
[INFO] \- com.github.adamheinrich:native-utils:jar:e6a39489662846a77504634b6fafa4995ede3b1d:compile

You can see, my project is using 3 dependencies, log4j, junit, and native-utils with their version as specified.

Load the native library

In your Java file, you can call the loadLibraryFromJar method to load the native library.

import cz.adamh.utils.NativeUtils;

public class NativeMemoryLoader {

  static {
    try {
      NativeUtils.loadLibraryFromJar("/lib/libhello.dylib");
    } catch (IOException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
    }
  }

  public static native void sayHello();

  ...
}

Load library based on your OS

The problem with loading native libraries like C/C++ is that your compiled file is dependent on the platform. This means you'll need to compile the library for each platform and then package it in the JAR file. So you'll have different versions of the library for different platforms. And you'll end up with so, dylib, and dll files in your lib directory. Let's modify our Java code to use the library based on the platform.

  // Get the file extension based on OS
  String osName = System.getProperty("os.name").toLowerCase();
  String libExtension = osName.contains("win") ? ".dll" :
                        osName.contains("mac") ? ".dylib" : ".so";
  String libPath = "/lib/libhello" + libExtension;
  NativeUtils.loadLibraryFromJar(libPath);

Conclusion

In this article, we saw how to package native libraries in a JAR file using Maven and load them in our Java program. We also saw how to load the library based on the platform.

Configure Google Java Formatter with VSCode

Sat, 22 Apr 2023 05:07:34 GMT

Introduction

Using the same code format is helpful while working on a team project. This helps you in code readability and also helps you to avoid merge conflicts and makes pull requests easier to focus on. For this people use different code formatter like Google Java Formatter, Prettier, etc. In this blog, we will see how to configure Google Java Formatter with VSCode.

Download the jar file

The first thing you need to do is download the Google Formatter jar file from here and keep it in your home directory. You can keep it anywhere you want but I prefer to keep it in my home directory. Make sure you download the jar file with -all-deps in the name.

You can also download the jar file using the following command. The -O flag is used to specify the location where you want to download the jar file. The below command will download the jar file and store it in the home directory with the file name google-code-formatter.jar.

> wget https://github.com/google/google-java-format/releases/download/v1.16.0/google-java-format-1.16.0.jar \
  -O ~/google-code-formatter.jar

Configure the VSCode

Now we will configure the VSCode to use the Google Java Formatter. For this, you'll need an extension called External Formatters extension. This extension helps you use an external code formatter with VSCode.

To install the extension, open the VSCode and press Cmd + Shift + X to open the extension tab. Search for External Formatters authored by Stefan van der Haven and install it.

Now open the VSCode settings by pressing Cmd + , and search for External Formatters. Then click on the Edit in settings.json button for modifying the language setting.

<!image/>

Now add the following code in the settings.json file.

[
  ...
  "externalFormatters.languages": {
      "java": {
          "command": "java",
          "arguments": [
              "-jar",
              "/Users/shubham.kumar/google-java-format-1.16.0-all-deps.jar",
              "-"
          ]
      }
  },
  "[java]": {
      "editor.formatOnSave": true,
      "editor.defaultFormatter": "SteefH.external-formatters",
      "editor.tabSize": 2
  }
]

The above configuration defined inside externalFormatters.languages configures the External Formatters extension to use the Google Java Formatter jar file for formatting the Java files. This configuration tells VScode to run java -jar /Users/shubham.kumar/google-java-format-1.16.0-all-deps.jar <your-file.java> command for formatting your java files.

The [java] configuration is language specific setting that tells VSCode to use the External Formatters extension for formatting the Java files and run the formatter whenever a file is saved. Google Java Code formatter uses 2 spaces for indentation by default. So we set the editor.tabSize to 2 for Java files.

Note: You can also use .vscode/settings.json for configuring the VSCode settings for a particular project.

Conclusion

In this blog, we saw how to configure Google Java Formatter with VSCode. Now you can use the same code formatter for all your Java projects and also use it with your team members to avoid merge conflicts and make your pull requests easier to review.

Upgrade a simple HTML, CSS and JS site to a WebSocket application

Thu, 26 Jan 2023 20:53:46 GMT

Intro

Sarvesh, one of my friends, created a gaming project called "Pig Game" during his Javascript learning journey. It is a basic site built using HTML, CSS, and JS in which two people compete by rolling a die.

At any moment, you have two options: roll the dice to increase your score or give up so that the next person may play. The catch is that if the dice stop at 1, your current score becomes 0, and the game is passed to the next player. But if you willingly pass the game to the next person, your current score gets added to the total score which ultimately saves your hard work.

For the time being, this game is only a static web page with all of its capabilities running in the browser. And because of this, it is unplayable if you are not in the same room (or virtually sharing the screen of the host). We want to make this a server-based game so that we may play together even from afar.

Here, let's implement a WebSocket server using express.js & socket.io. Also, we'll configure the front-end to establish a connection with this server.

HTTP vs WebSocket protocols

HTTP

Let's have a bird-eye view of the traditional HTTP protocol. With HTTP, a client sends a request to the server and the server process the request and sends a response in return. The connection is then closed after this request-response cycle. HTTP is stateless and because it runs on TCP, data delivery is ensured.

A stateless protocol does not require the server to retain information or status about each user for the duration of multiple requests. -- wikipedia

You will be required to use cookies or server-side sessions to not forget the user in an HTTP protocol.

<!plantuml/>

I think that's all we need to talk about at least during the starting phase.

Steps 2 and 3: Onboarding the players

The first step is already taken care of by 'socket.io'. Let's configure our server for steps 2 and 3. We want to send a 'ready' signal when 2 players are connected. If more than 2 people join, we can reject their connection on the server side using socket.disconnect() function and send them a 'reject' signal.

socket.emit() sends the message to a particular socket (client), socket.broadcast.emit() sends the message to everyone except the socket (client) itself and io.emit() sends the message to all the clients connected to a socket.

`server.js`

...
let players = {};    // This will store the details about the players like their score
let activePlayer = 0;

io.on('connection', (socket) => {   
  players[socket.id] = {    // Initialize the player on establishing a connection
    "socket": socket,
    "total_score": 0,
    "current_score": 0,
    "isActivePlayer": false,
  } 
  let total_players = Object.keys(players).length
  console.log(`Player with socket id ${socket.id} connected.`)
  console.log(`Total number of players: ${total_players}`)

  // Sending the connection status
  // Reject if we already have 2 players connected
  if (total_players > 2) {
    delete players[socket.id]
    socket.emit("connection_status", {"connection_status": "reject"});
    socket.disconnect();
    console.log("Already 2 players are onboarded", Object.keys(players))
  }
  // Send ready signal when we have 2 players 
  else if (total_players == 2) {
    // Tell player 1 to go first
    activePlayer = 0
    io.emit("connection_status", {"connection_status": "ready", "active_player": Object.keys(players)[activePlayer]})
  // Else send a waiting signal
  else {
    io.emit("connection_status",  {"connection_status": "waiting"})
  }

  socket.on('disconnect', () => {
      delete players[socket.id];
      console.log('Client disconnected ' + socket.id);
  });
});

On the client side, we will consume the messages received to perform certain actions. This can be done by adding a socket message listener, socket.on(). Right now, let's use alerts for notifying the current player about the game status.

`public/script.js`

socket.on("connection_status", (args) => {
    console.log("Received a connection_status signal", args.connection_status)
    if (args.connection_status === 'waiting') {
        alert("Waiting for second player to begin the game")
    }
    else if (args.connection_status === 'ready') {
        // Enable player is a function to activate and deactivate the roll and hold buttons
        EnablePlayer(args.active_player);
        alert(`Let's begin the game. ${args.active_player===socket.id ? "It's your turn" : "Oponent's turn"}`)
    }
    else if (args.connection_status === 'reject') {
        alert("Can't join, already 2 players onboarded")
    }
});

Step 4: Play the game

Looks like this is all we wanted in the steps 2 and 3. For step 4, we will send a signal from the client's side about his decision. This can be achieved by the button event listners. And the server will calculate a random number b/w 1 to 6 and broadcast the response.

`public/script.js`

...
// Same for btnHold with decision "hold"
btnRoll.addEventListener('click', function(){
    if(playing){
        socket.emit("decide", {
            "player_id": socket.id,
            "decision": "roll"
         });
    }
});
...

On the server, we can decide the score and broadcast it back to clients. We will send the current and total scores of the player along with the next player's socket id and dice score.

`server.js`

...
 function NotifyScores(active_player, dice) {
    let player1 = Object.keys(players)[0];
    let player2 = Object.keys(players)[1];

    io.emit("score_update", {
      'player1_current': players[player1].current_score,
      'player1_total': players[player1].total_score,
      'player2_current': players[player2].current_score,
      'player2_total': players[player2].total_score,
      'next_turn': Object.keys(players)[activePlayer],
      'acitive_player_roll': activePlayer,
      'dice': dice
    });
  }

  socket.on("decide", (args) => {
    console.log("decide", args);
    if (args.player_id === Object.keys(players)[activePlayer]) {
      if (args.decision === 'roll'){
        const dice = Math.trunc(Math.random()*6) + 1;
        players[args.player_id].current_score = dice == 1 
          ? 0 
          : players[args.player_id].current_score + dice;
        activePlayer = dice == 1 ? (activePlayer + 1) % 2 : activePlayer
        NotifyScores(activePlayer, dice);
      }
      else if (args.decision === 'hold') {
        players[args.player_id].total_score += players[args.player_id].current_score;
        players[args.player_id].current_score = 0; 
        activePlayer = (activePlayer + 1) % 2;
        NotifyScores(activePlayer, 0);
      }
    }
  });

Step 5: Update the scores

Step 5 is very simple. Just update the data received by the client from the server in step 4.

`public/script.js`

socket.on("score_update", (args) =>{
    console.log("score update", args);
    document.getElementById(`current--0`).textContent = args.player1_current;
    document.getElementById(`current--1`).textContent = args.player2_current;
    document.getElementById(`score--0`).textContent = args.player1_total;
    document.getElementById(`score--1`).textContent = args.player2_total;
    diceEl.classList.remove('hidden');
    if (args.dice != 0) diceEl.src = `assets/images/dice-${args.dice}.png`;
    EnablePlayer(args.next_turn)
});

Step 6: Decide the winner

Step 6, is deciding the winner. Whoever scores greater than 60 will be entitled the winner. We have many options to send this information. This can be sent by the server as a new event, "winner". Or we can include this with the 'score_update' event.

Let's update the "decide" event to send a new message whenever a player wins.

`server.js`

 socket.on("decide", (args) => {
    if (args.player_id === Object.keys(players)[activePlayer]) {
      ...
      else if (args.decision === 'hold'){
        ...
        players[args.player_id].total_score += players[args.player_id].current_score;
        players[args.player_id].current_score = 0; 

        // Deciding the winner
        if (players[args.player_id].total_score >= 60) {
            io.emit("winner", activePlayer);
        }
        else {
            activePlayer = (activePlayer + 1) % 2;
            NotifyScores(activePlayer, 0);
        }
      }
      ...
    }
 });

This can be updated on the client side.

`public/script.js`

socket.on("winner", (winner) => {
    playing = false;
    document.querySelector(`.player--${winner}`).classList.add('player--winner');
    document.querySelector(`.player--${winner}`).classList.remove('player--active');
});

This is it, now you can play the game with your friend. You can always host it online using ngrok. Now what's left is adding more features like adding an in-game chat, viewers, a gaming room and much more. Also, there is a need to clean the code.

Conclusion

In this blog, we witnessed the working of a WebSocket app using Socket.io. We focused on establishing a connection and sending messages from server to client as well as from client to server. There are 3 ways in which the messages can be communicated, io.emit(), socket.emit() and socket.broadcast.emit(). io.emit() is used for broadcasting the message to all the clients. socket.broadcast.emit() broadcasts the message to all the clients except the active client. socket.emit() is for server to client or client to server messaging. For receiving the messages we implement a listener, that triggers on receiving a specific event as defined by us. We also saw some special events like connection and disconnect.

Automate your local environment setup using dev containers

Sun, 08 Jan 2023 19:06:49 GMT

Index

Topic	Description
Problem	Automate the set up process for the projects
Dev Containers	What are dev containers?
Dev container custom configurations	Configuring a dev container as per your needs.
Open your container in VSCode	How to open a dev container in `VSCode`.
Open your container in a browser	Opening dev container in `github.dev` text editor.
Creating a dev container using templates available	`Java` and `Postgres` (as service) dev container set up.
Adding fish terminal to the dev container	Using `fish` shell and setting the default shell for a container.
oh-my-zsh with powerlevel10k	Using `zsh` along with `oh-my-zsh` and `powerlevel10k`.
Make the dev container distributable	You should not add personal changes to source control. Use external script to modify the dev container configurations.
Conclusion	Final words

Problem

I use a dual-booted ROG with Ubuntu 22.10 and Windows 10 for development, along with a MacBook Air 2020. Every time I begin a new project that makes use of a new tech stack, I am required to take some time to set up the environment on all 3 machines. And sometimes, my laziness compels me to use only one machine instead of setting up other ones. The issue is that I don't want to go through the same setup process again and again.

The same problem exists on a bigger scale too. In GreyOrange, I've seen my supervisors spending a lot of time helping freshers set up their new laptops. And even after following the setup guide on confluence, it took me a lot of time to build my first project successfully. Imagine helping 300 freshers set up their laptops for a particular project.

Plus, the building process can be different for different Operating Systems. Wouldn't it be great to use a system that your CICD platform uses?

This problem can be solved using dev containers that only require Docker and git installed on your machine.

Dev containers

Dev containers are just Docker containers that are fully equipped with the necessary tech stacks and tools to begin developing. Simply establish a connection with your container and begin writing code. Additionally, a version control program like git may be used to share this setup. So, anyone with a computer may run these Docker containers and begin developing them. GitHub codespace uses dev container configurations to set up their cloud development environment. You can also customize your dev container using the Dockerfile and docker-compose configurations.

Dev container custom configurations

A simple dev container configuration has a .devcontainer directory consisting of devcontainer.json and a Dockerfile. The devcontainer.json contains all the configurations used by the container while Dockerfile contains the instruction to create a Docker container. You can also use an image from the Docker hub directly by using the image attribute.

`Directory structure`

.devcontainer
├── Dockerfile
└── devcontainer.json

`devcontainer.json`

{
	"name": "Hello DevContainer",   // This is the name of the container
    
    // build defines the image configurations
    // Instead of building you can use `image` attribute to provide the docker image hosted on Docker hub
    "build": {  
        "dockerfile": "Dockerfile", // This is the Dockerfile located at '.devcontainer/Dockerfile'
        "context": "..",            // This is where the project lies
        "args": {                   // Anything supplied to Dockerfile as an argument
            "PYTHON_VERSION": "3.9" 
        }
    },

	// Features to add to the dev container. More info: https://containers.dev/features.
	// "features": {},

	// Use 'forwardPorts' to make a list of ports inside the container available locally.
	// "forwardPorts": [],

	// Will run these after the container is created
    // Generally used to install the dependencies
	// "postCreateCommand": "pip install -r requirements.txt",

    // You can use root as remoteUser but it's not advisable to do so
	"remoteUser": "root"
}

`Dockerfile`

ARG PYTHON_VERSION
FROM python:${PYTHON_VERSION}.0-slim
# And other setup instructions

Open your container in VSCode

To connect to a dev container using VSCode, you need to install Dev Containers extension. You can open your current directory in a dev container by clicking F1 and selecting Dev Containers: Reopen in Container. You can also choose Dev Containers: Rebuild and Reopen in Container in case you want to build the Docker image once again.

<!image/>

This will take some time in building the container and installing the tools. Once completed, you can verify the connection by looking at the left end of the VSCode window. It should display Dev Container: <Container name> @ <Operating system>

<!image/>

Note: The extensions are gone. For your dev container, you'll need to add your extensions once again.

You can also check the python version to confirm if it's installed properly.

> root@2a35d5d14816:/workspaces/devcontainers# python --version
Python 3.9.0

Open your container in a browser

You can also use the power of VSCode from your browser using github.dev editor. You need to host your project on GitHub for this to work. Let's do that.

First, you will need to add git features in devcontainer.json file. This will enable us to use git and github cli from inside the container. You can also enable ssh using the sshd feature. A list of available features can be found at containers.dev/features.

    ...
    "features": {
        "ghcr.io/devcontainers/features/git:1": {},
        "ghcr.io/devcontainers/features/github-cli:1": {},	
        "ghcr.io/devcontainers/features/sshd:1": {}
    },
    ...

Then rebuild the container by pressing F1 -> Rebuild Container. Now you can use git and gh inside your container.

> git init
> git add . 
> git commit -m "Init"
> gh auth login # Here you can create a new ssh key for your container
> gh repo create hello-devcontainer --public --source . --push
✓ Created repository GO-Shubham-Kumar/hello-devcontainer on GitHub
✓ Added remote git@github.com:GO-Shubham-Kumar/hello-devcontainer.git
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 4 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 924 bytes | 71.00 KiB/s, done.
Total 5 (delta 0), reused 0 (delta 0)
To github.com:GO-Shubham-Kumar/hello-devcontainer.git
 * [new branch]      HEAD -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.
✓ Pushed commits to git@github.com:GO-Shubham-Kumar/hello-devcontainer.git

The above procedure creates a new repository and pushes the project commits to it. I can see my project at https://github.com/GO-Shubham-Kumar/hello-devcontainer.

Now we have our code on GitHub. It's time to open a devcontainer using gitub.dev. For this simply visit your GitHub repo and change the URL from github.com to github.dev. This will launch a new VSCode-style window with all your tools available.

<!image/>

Note: github.dev is a lightweight editor which does not support a terminal. If you want to use the terminal, you should switch to GitHub Codespaces.

Creating a dev container using templates available

Above, I showed you how to configure a dev container using a test repository. Now, I'll show you how to use available templates to run services like PostgreSQL along with your container.

In GreyOrange, we have a project that uses SpringBoot and PostgreSQL. We need a list of databases to run the test case for this project. Let's try to configure our dev container to have a PostgreSQL service along with the databases required.

Summary of things we need.

Java 8 and Maven
PostgreSQL 9.6
Populate the databases

You can visit container templates to search for a required template. We will use, Java & Postgres template. This template allows us to configure the Java version, package manager and PostgreSQL version.

In case your template isn't available, you will need to create your Dockerfile & docker-compose files.

From your VSCode editor, Click F1 -> Dev Containers: Add Dev Container Configuration Files.... Click on Show All Definitions and select Java & PostgreSQL. I'm using version 8-bullseye with Maven so I'll select them. After a few moments, VSCode will create a .devcontainer directory with Dockerfile, docker-compose.yml & devcontainer.json.

Looking at the generated code it seems like we require some modifications for this to work properly.

We want to initialize the databases as per the database_creator.sql. This file contains all the databases we require to run the test cases. This can be done by adding an entry to the db -> volume as shown in the docker-compose.yml file.
The Java image should be java:8-bullseye, not java:0-8-bullseye which was generated by the generator.
We need Postgres:9.6, the generated one point to the latest image.

`devcontainer.json`

// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/java-postgres
{
	"name": "Java & PostgreSQL",
	"dockerComposeFile": "docker-compose.yml",
	"service": "app",
	"workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}"
}

`Dockerfile`

# Changed java:0-8-bullseye to java:8-bullseye
FROM mcr.microsoft.com/devcontainers/java:8-bullseye

ARG INSTALL_MAVEN="true"
ARG MAVEN_VERSION=""

ARG INSTALL_GRADLE="false"
ARG GRADLE_VERSION=""

RUN if [ "${INSTALL_MAVEN}" = "true" ]; then su vscode -c "umask 0002 && . /usr/local/sdkman/bin/sdkman-init.sh && sdk install maven \"${MAVEN_VERSION}\""; fi \
    && if [ "${INSTALL_GRADLE}" = "true" ]; then su vscode -c "umask 0002 && . /usr/local/sdkman/bin/sdkman-init.sh && sdk install gradle \"${GRADLE_VERSION}\""; fi

`docker-compose.yml`

version: '3.8'

volumes:
  postgres-data:

services:
  app:
    container_name: javadev
    build: 
      context: .
      dockerfile: Dockerfile
    environment:
      # NOTE: POSTGRES_DB/USER/PASSWORD should match values in db container
        POSTGRES_PASSWORD: postgres
        POSTGRES_USER: postgres
        POSTGRES_DB: postgres
        # We will populate the database using database_creator.sql
        # POSTGRES_HOSTNAME: postgresdb 

    volumes:
      - ../..:/workspaces:cached
    command: sleep infinity

    # Runs app on the same network as the database container, allows "forwardPorts" in devcontainer.json function.
    network_mode: service:db

    # Use "forwardPorts" in **devcontainer.json** to forward an app port locally. 
    # (Adding the "ports" property to this file will not forward from a Codespace.)

  db:
    container_name: postgresdb
    image: postgres:9.6
    restart: always
    volumes:
      # Run the below script as an initialization script 
      - ../misc/database_creator.sql:/docker-entrypoint-initdb.d/database_creator.sql
      # We no longer require this
      # - postgres-data:/var/lib/postgresql/data
    environment:
      # NOTE: POSTGRES_DB/USER/PASSWORD should match values in app container
      POSTGRES_PASSWORD: postgres
      POSTGRES_USER: postgres
      # We will create our databases using misc/database_creator.sql
      # POSTGRES_DB: postgres

    # Add "forwardPorts": ["5432"] to **devcontainer.json** to forward PostgreSQL locally.
    # (Adding the "ports" property to this file will not forward from a Codespace.)

The above configurations will run PostgreSQL along with our app. You can also view this combination in Docker Desktop.

<!image/>

Note: Here I haven't used git and gh features because the image java:8-bullseye already comes with these tools.

Let's verify the Java and Maven versions.

> java -version
openjdk version "1.8.0_352"
OpenJDK Runtime Environment (Temurin)(build 1.8.0_352-b08)
OpenJDK 64-Bit Server VM (Temurin)(build 25.352-b08, mixed mode)
> mvn -version
Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)
Maven home: /usr/local/sdkman/candidates/maven/current
Java version: 1.8.0_352, vendor: Temurin, runtime: /usr/local/sdkman/candidates/java/8.0.352-tem/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.15.49-linuxkit", arch: "aarch64", family: "unix"

Let's compile the project.

> mvn clean install
...
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  35:03 min
[INFO] Finished at: 2023-01-08T00:05:51Z
[INFO] ------------------------------------------------------------------------

It took a while but the build was successful.

Adding fish terminal to the dev container

Right now, the dev container is using bash as it's the default shell. There are a lot of shells that come along with this image.

> echo $SHELL
/bin/bash

> cat /etc/shells
/bin/sh
/bin/bash
/bin/rbash
/bin/dash
/bin/zsh
/usr/bin/zsh

Let's try installing fish terminal which is not available by default. fish can be installed using the features option in devcontainer.json. A list of features is available here.

Modify the devcontainer.json as shown below and rebuild the container.

`devcontainer.json`

{
    ...
    "features": {
		"ghcr.io/meaningful-ooo/devcontainer-features/fish:1": {}
	}
}

Now, you can find fish in the list of available shells. And this feature also sets fish as your default terminal.

> cat /etc/shells
/bin/sh
/bin/bash
/bin/rbash
/bin/dash
/bin/zsh
/usr/bin/zsh
/usr/bin/fish   # This is it

<!image/>

If you don't want to set fish as your default terminal, you can specify the default terminal in devcontainer.json file as follows. This will set bash as your default container.

`devcontainer.json`

{
  ...
	"settings": { 
		"terminal.integrated.defaultProfile.linux": "bash", 
		"terminal.integrated.profiles.linux": { 
			"bash": { 
				"path": "bash" 
				} 
			} 
		}
  ...
}

oh-my-zsh with powerlevel10k

fish is a great shell. But I want the dev container to feel like my local machine that has oh-my-zsh with powerlevel10k. At the time of writing this blog, I wasn't able to find any features to install these things. So I went with configuring the Dockerfile itself to include these features.

Disclaimer: This is a very bad idea to directly change a source controlled dev container file as per your preference

This image already comes with oh-my-zsh. Powerlevel10k is a custom theme that I'll need to install on my own. Same goes for zsh-syntax-highlighting and zsh-autosuggestions plugins.

`Dockerfile`

...
# Install powerlevel10k
RUN git clone --depth=1 https://github.com/romkatv/powerlevel10k.git /home/vscode/.oh-my-zsh/custom/themes/powerlevel10k
# Install zsh-autosuggestions
RUN git clone https://github.com/zsh-users/zsh-autosuggestions /home/vscode/.oh-my-zsh/custom/plugins/zsh-autosuggestions
# Install zsh-syntax-highlighting
RUN git clone https://github.com/zsh-users/zsh-syntax-highlighting.git /home/vscode/.oh-my-zsh/custom/plugins/zsh-syntax-highlighting
# This is my local machine's theme file 
COPY .p10k.zsh /home/vscode
# This is the modified zsh configuration
COPY .zshrc /home/vscode

.p10k.zsh contains the powerlevel10k configurations. .zshrc contains the zsh configurations. These files are located inside the home directory. I just copied them to .devcontainer for an easy transfer to the dev container.

> tree -a .devcontainer
.devcontainer
├── .p10k.zsh
├── .zshrc
├── Dockerfile
├── devcontainer.json
└── docker-compose.yml

If you don't have the .p10k.zsh, you can skip the COPY step while building the container. Once inside the container, you can run p10k configure to generate the .p10k.zsh file. Then the contents of this file can be copied to .devcontainer/.p10k.zsh for future builds.

Below is my .zshrc file. You can verify the ZSH_THEME is set as powerlevel10k/powerlevel10k. I am just using 3 plugins. The git plugin is available by default. zsh-autosuggestions and zsh-syntax-highlighting were downloaded from GitHub.

# Enable Powerlevel10k instant prompt. Should stay close to the top of ~/.zshrc.
# Initialization code that may require console input (password prompts, [y/n]
# confirmations, etc.) must go above this block; everything else may go below.
if [[ -r "${XDG_CACHE_HOME:-$HOME/.cache}/p10k-instant-prompt-${(%):-%n}.zsh" ]]; then
  source "${XDG_CACHE_HOME:-$HOME/.cache}/p10k-instant-prompt-${(%):-%n}.zsh"
fi

export ZSH="$HOME/.oh-my-zsh"

ZSH_THEME="powerlevel10k/powerlevel10k"

plugins=(
	git
	zsh-autosuggestions
	zsh-syntax-highlighting
)

source $ZSH/oh-my-zsh.sh

# To customize prompt, run `p10k configure` or edit ~/.p10k.zsh.
[[ ! -f ~/.p10k.zsh ]] || source ~/.p10k.zsh

<!image/>

If there are multiple developers with their shell preference, modification of Dockerfile for a specific need is not a very good idea.

Make the dev container distributable

You should not change a source controlled Dockerfile to satisfy your need. The current Dockerfile contains the changes I made to suit my preference. I like zsh so I did some customization to Docker to support my theming style. If I commit these changes to the remote, this might become a problem for another developer who prefers some other shell or theme. Plus, if everyone starts making changes to the dev container configuration, this might result in merge conflicts.

In scenarios like this, you can use the git skip-worktree feature. After cloning the base repo, you don't want to make any commits on the dev container. So you can tell git to ignore tracking the .devcontainer directory.

# Tell git to stop tracking these files
> git update-index --assume-unchanged .devcontainer/*

# View the files which are not being tracked
> git ls-files -v | grep "h" | grep dev
h .devcontainer/Dockerfile
h .devcontainer/devcontainer.json
h .devcontainer/docker-compose.yml

Now you can freely configure the dev container as per your need.

Actually, this is not the best way to deal with this problem at all, as this will ignore the .devcontainer directory even in the case of git pull. And if you do the reverse using git update-index --no-assume-unchanged .devcontainer/*, you will need to commit or discard your changes before proceeding.

The best way I can think of to deal with this problem is by keeping the personal installations away from the git repository.

For this, let's create an external script that will install powerlevel10k and other plugins using the docker commands.

First, let's move our .zshrc and .p10k.zsh to a new directory out of our repository. I'm creating a hidden directory in the home directory for this.

> mkdir ~/.personal-devcontainer 
> mv .devcontainer/.zshrc ~/.personal-devcontainer
> mv .devcontainer/.p10k.zsh ~/.personal-devcontainer

Also, let's reset the devcontainer.json, Dockerfile and docker-compose.yml to the unbiased version.

`devcontainer.json`

// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/java-postgres
{
	"name": "Java & PostgreSQL",
	"dockerComposeFile": "docker-compose.yml",
	"service": "app",
	"workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}"
}

`Dockerfile`

# Changed java:0-8-bullseye to java:8-bullseye
FROM mcr.microsoft.com/devcontainers/java:8-bullseye

ARG INSTALL_MAVEN="true"
ARG MAVEN_VERSION=""

ARG INSTALL_GRADLE="false"
ARG GRADLE_VERSION=""

RUN if [ "${INSTALL_MAVEN}" = "true" ]; then su vscode -c "umask 0002 && . /usr/local/sdkman/bin/sdkman-init.sh && sdk install maven \"${MAVEN_VERSION}\""; fi \
    && if [ "${INSTALL_GRADLE}" = "true" ]; then su vscode -c "umask 0002 && . /usr/local/sdkman/bin/sdkman-init.sh && sdk install gradle \"${GRADLE_VERSION}\""; fi

`docker-compose.yml`

version: '3.8'

volumes:
  postgres-data:

services:
  app:
    container_name: javadev
    build: 
      context: .
      dockerfile: Dockerfile
    environment:
      # NOTE: POSTGRES_DB/USER/PASSWORD should match values in db container
        POSTGRES_PASSWORD: postgres
        POSTGRES_USER: postgres
        POSTGRES_DB: postgres
        # We will populate the database using database_creator.sql
        # POSTGRES_HOSTNAME: postgresdb 

    volumes:
      - ../..:/workspaces:cached
    command: sleep infinity

    # Runs app on the same network as the database container, allows "forwardPorts" in devcontainer.json function.
    network_mode: service:db

    # Use "forwardPorts" in **devcontainer.json** to forward an app port locally. 
    # (Adding the "ports" property to this file will not forward from a Codespace.)

  db:
    container_name: postgresdb
    image: postgres:9.6
    restart: always
    volumes:
      # Run the below script as an initialization script 
      - ../misc/database_creator.sql:/docker-entrypoint-initdb.d/database_creator.sql
      # We no longer require this
      # - postgres-data:/var/lib/postgresql/data
    environment:
      # NOTE: POSTGRES_DB/USER/PASSWORD should match values in app container
      POSTGRES_PASSWORD: postgres
      POSTGRES_USER: postgres
      # We will create our databases using misc/database_creator.sql
      # POSTGRES_DB: postgres

    # Add "forwardPorts": ["5432"] to **devcontainer.json** to forward PostgreSQL locally.
    # (Adding the "ports" property to this file will not forward from a Codespace.)

Now, let's create a bash script inside ~/.personal-devcontainer that will change the container configuration as per our preference (personal changes).

> cd ~/.personal-devcontainer
> touch install.sh
> chmod +x install.sh
> vim install.sh

`install.sh`

# This will work on a running dev container
CONTAINER_NAME=$1
docker exec $CONTAINER_NAME git clone --depth=1 https://github.com/romkatv/powerlevel10k.git /home/vscode/.oh-my-zsh/custom/themes/powerlevel10k
docker exec $CONTAINER_NAME git clone https://github.com/zsh-users/zsh-autosuggestions /home/vscode/.oh-my-zsh/custom/plugins/zsh-autosuggestions
docker exec $CONTAINER_NAME git clone https://github.com/zsh-users/zsh-syntax-highlighting.git /home/vscode/.oh-my-zsh/custom/plugins/zsh-syntax-highlighting
docker cp ~/.personal-devcontainer/.p10k.zsh $CONTAINER_NAME:/home/vscode/.p10k.zsh
docker cp ~/.personal-devcontainer/.zshrc $CONTAINER_NAME:/home/vscode/.zshrc
docker exec $CONTAINER_NAME chsh -s $(which zsh)

Now you can just run ./install.sh javadev to install the preferences to the javadev container (which is our development container name). As I'll keep changing the script depending on my projects, I'll publish the installation scripts repo to my git repo.

There are 2 problems with this method -

You will require to run the script on every rebuild.
You cannot directly use this script online with github.dev.

For easier access, let's create an alias for the script. It will take a container name as a parameter and will configure the shell as required.

`.zshrc`

...
# configure-personal-dev-container javadev
function configure-personal-dev-container {
	if [[ -z $1 ]]; then 
		echo Please provide a container name
		exit 1
	else
		~/.personal-devcontainer/install.sh $1
	fi
}
...

Now, I can run configure-personal-dev-container javadev from anywhere.

Conclusion

Dev container helps us to reduce the setup time for a project. We started with creating a simple dev container configuration. We used the power of dev containers to open the project in VSCode and github.dev. Then we created a complex dev container configuration consisting of Java & Maven with PostgreSQL installation and database creation for an ongoing project at GreyOrange. We tried to compile our project inside the dev container which worked as expected. Then we moved to configure the shell inside the dev container. We tried fish and zsh with powerlevel10k which is what I use locally. We also learned to configure the dev container for personal development without disturbing the source-controlled setup configurations.

A note on creating dynamic link previews for your website or blog

Fri, 23 Dec 2022 07:53:55 GMT

import BlogImageWithContext from "../../components/BlogImageWithContext.astro"; import BlogImage from "../../components/BlogImage.astro"; import LinkPreviewDemo from "../../images/posts/2022/info-link-previews/link-preview-demo.png"; import OpenGrapgDebugging from "../../images/posts/2022/info-link-previews/opengraph-debugging.png"; import OpenGrapgDebuggingPreview from "../../images/posts/2022/info-link-previews/opengraph-debugging-preview.png"; import FlaskAppPreviewImage from "../../images/posts/2022/info-link-previews/flask-app-preview-image.png"; import SlackPreview from "../../images/posts/2022/info-link-previews/link-preview-on-slack.png"; import FinalPreview from "../../images/posts/2022/info-link-previews/final-preview.png";

Index

Topic	Description
General Introduction	Understanding link previews
Defining meta tags	Open-graph and Twitter mea tags
Viewing and Debuggig	Using opengraph.xyz to view your previews
Make it dynamic	Create a custom server to generate images
Hosting the server on Heroku	Deploy the server on Heroku and test its working
Conclusion	Final words

General Introduction

Using social networking apps, you can add a preview to the URL you share. As a result, the reader can get a decent notion of the contents.

<!image-context/>

Hosting the server on Heroku

The Flask dev server won't work with Heroku deployments. Let's use Gunicorn for our deployments.

Create a new file with the name Procfile with no extensions and modify its contents as below.

`Procfile`

web: gunicorn gettingstarted.wsgi
web: gunicorn app:app

The first line tells Heroku to use Gunicorn as server. The second line tellsGunicorn to run the app. Gunicorn uses the syntax gunicorn filename:app_name to start a server.

Also, update the requirements.txt file by adding gunicorn.

`requirements.txt`

Flask
pillow
requests
gunicorn

Now you can use Heroku CLI to host the app following the below steps.

You can install the Heroku CLI using the instructions here.

# First you need git (Heroku uses git)
> git init
> git add .
> git commit -m "My first link preview"

# Login to your account
> heroku Login

# This will create an app and add a remote to your git
> heroku create -a <some-unique-name>

> git remote -v
heroku https://git.heroku.com/<some-unique-name>.git

# To deploy just push your code to Heroku remote
> git push heroku main

Let's check it on opengraph.xyz.

<!image/>

This is exactly what we wanted. Now, we can create multiple HTML files with different titles and our server will generate a preview accordingly.

This was a small demo of generating link previews. You can further customize your server to generate previews based on title, description, publish date and much more.

Conclusion

A dynamic link preview can be generated by hosting a server and pointing our og/twitter image to it. Here we used Flask to create a GET route that provides us with the image. You can add more flexibility/features by using html2img module for generating images from HTML directly. We also saw the process of hosting a Gunicorn server on Heroku. This proved to be more useful in testing our deployment on opengraph.xyz as reverse proxy had some problems.

Combining multiple git repositories into a single repository and retaining all the commit histories

Tue, 13 Dec 2022 18:30:00 GMT

Topics	Description
The Problem	Merge multiple git repositories into one while preserving all histories
A bit about git	Introduction to Blob, Tree and Commits
Rewriting history	Uusing filter-branch to rewrite history. Moving a file inside a directory while retaining git histories
Merging multiple repos into one	A step by step guide for combining two git repositories into a larger parent repo while preserving histories
Merging using Shopsys monorepo-tools	Merging two git repos into a larger parent repo using monorepo tools by shopsys
Conclusion	Final words

The Problem

In our company, we had multiple maven projects that were dependent on each other. And for every project, we maintained a separate git repository hosted on Bitbucket.

Due to a certain use-case, we wanted to migrate all our Bitbucket repos to GitHub. And to avoid any circular dependency issues, we also wanted to combine all these related repositories into a mono-repo system.

The idea is to have a parent git repository that will contain all these repositories as a subdirectory.

Things described here will work for any project using git version control, irrespective of the hosting platform.

`Folder structure`

parent
 ├── POM.XML        # Our new parent POM
 ├── .git           # This is our new git that will contain all the histories
 ├── repo1         
 │   ├── POM.XML
 │   ├── ...
 │   └── .git       # This git will be removed
 ├── repo2
 │   ├── POM.XML
 │   ├── ...
 │   └── .git       # This git will be removed
 └── repo3
     ├── POM.XML
     ├── ...
     └── .git       # This git will be removed

Combining the repos is the easier part. As we were using Maven, all we required was to create a master POM.XML that would contain all the sub-repos as submodules.

Again, the steps defined here are not dependent on Maven or any other package manager.

The difficult part was to retain all the histories, which I'll explain in this blog. Because each repository has it's own .git, removing this will delete all the associated histories. After which we won't be able to identify the code changes, commits, and most importantly, the branches.

At GreyOrange, we maintain a develop branch and certain release branches for each repo. We were required to merge all the respective commits on develop and release branches as well. We wanted to merge their histories so that we could have all our commits on the same branch as they were originally. We also had many feature branches, but they were mostly independent, so merging wasn't required for them.

A bit about git

When you run git init, it creates a new .git directory. This directory contains everything git needs to do it's magic.

`objects` directory

The blobs, trees, and commits are all stored in the objects directory within .git. These are the 3 basic elements that define all the functionality of git.

Blobs are a type of data structure that contains compressed information related to the changed files. Each time you run git add . , a new blob is created inside the objects directory. You can view a blob's contents using git cat-file.

Blobs are just the changed files hashed using SHA-1

> ls .git/objects
info/     pack/

# Create a file with it's content as test
> echo "test" >> test.txt
> ls .
    test.txt 

# adding changes creates blob of the changed files
> git add .
> ls .git/objects
0c/   info/    pack/      # 0c is the directory containing the changes

> ls .git/objects/0c
2c5f41c83de09587dfe46d5a5382eddf5bb77f
# The complete hash of the blob is 0c2c5f41c83de09587dfe46d5a5382eddf5bb77f
# Note: The first 2 letters are the directory name

# You can also view the contents of the blob using cat-file utility
> git cat-file blob 0c2c5f41c83de09587dfe46d5a5382eddf5bb77f
test

Multiple blobs are combined together to form a tree data structure. Generally, trees are created during a commit. But you can run git write-tree to generate a tree of recently added blobs. These trees are also stored inside the objects directory.

Trees are like the directories containing the blobs and other trees.

> git write-tree    # This outputs the hash associated with the tree generated
56bac5fc9c69776a5c67daa2225ef9b2e1edd4f6

# Trees are stored in the same manner
> ls .git/objects/56
bac5fc9c69776a5c67daa2225ef9b2e1edd4f6

# You can also view the content of a tree file
# It contains a reference to the blobs or trees
> git cat-file -p 56bac5fc9c69776a5c67daa2225ef9b2e1edd4f6
100644 blob 0c2c5f41c83de09587dfe46d5a5382eddf5bb77f    test.txt
# 0c2c5f... is the hash of the blob created above

So a tree represents the state of the system. Each commit is just 1 tree which is hashed and stored along with other information like author & date. They are also stored in the same manner.

> git commit -m "initial commit"

> ls .git/objects
0c/   52/   56/   info/    pack/    # 52 is newly generated directory

> git cat-file -p 5203c0048b4795669114fcdb261dc5bb4e77a54f
tree 56bac5fc9c69776a5c67daa2225ef9b2e1edd4f6       # This is the hash of the tree we created above
author Shubham Kumar <unresolved.shubham@gmail.com> 1670872587 +0530
committer Shubham Kumar <unresolved.shubham@gmail.com> 1670872587 +0530

A commit just contains the recent tree hash and inforamtion on author, time and committer.

It's not possible to distinguish a blob, a tree and a commit just by looking at the objects directory.

`refs` directory

The refs directory contains the reference to commits. There are 2 types of reference - branch and tags. Tags identify a unique commit while branch points to the latest child along the commit tree.

> ls .git/refs/heads
master               # There is just one branch right now

# Let's create a new branch and see what happens
> git checkout -b dev
Switched to a new branch 'dev'

> ls .git/ref/heads
dev   master        # Now you can see both the branches

> cat .git/ref/heads/dev
5203c0048b4795669114fcdb261dc5bb4e77a54f 
# Points to the latest commit. This is the exact same hash of the commit we created above. 

> git commit -m "commit to dev" --allow-empty
[dev 9d6d972] commit to dev

# Creating a new commit changes the contents of the active branch.
> cat .git/ref/heads/dev
9d6d972066b774e89343e57f2eb053559bf3f22c

`logs` directory

This contains the history of every branch. Everytime you change the branch using git checkout <branch-name> or update the tip, logs/HEAD is updated. logs/refs/heads contains the history of commits for a particular branch. This is a safety net. You can easily retrieve your work even after a rebase.

# To view the logs/HEAD
> git reflog
9d6d972 (HEAD -> dev) HEAD@{0}: checkout: moving from master to dev
5203c00 (master) HEAD@{1}: checkout: moving from dev to master
9d6d972 (HEAD -> dev) HEAD@{2}: commit: commit to dev
5203c00 (master) HEAD@{3}: checkout: moving from master to dev
5203c00 (master) HEAD@{4}: commit (initial): initial commit

Rewriting History

filter-branch can be used to rewrite the entire history of a git repository. This will create a new blob, tree and commit for everything once again. Rewriting history using the filter-branch does not hamper the commit data. Hashes will change but the entire inforamtion about the author, committer, etc remains unchanged.

# For all the commits do this -> mkdir nested; mv test.txt nested/test.txt
> git filter-branch --tree-filter 'mkdir nested; mv test.txt nested/test.txt' HEAD

> tree .
.
├── nested
    └── test.txt

> git cat-file -p d1950a39cbb6b1212e47d0c5be3b19e023051671  # This I got by looking inside the `objects` dir
tree 3ea8798ffe7147c04d66c9bef3ac5109ad6e80b8   # New tree reference
author Shubham Kumar <unresolved.shubham@gmail.com> 1670872587 +0530     # Commit time remained unchanged
committer Shubham Kumar <unresolved.shubham@gmail.com> 1670872587 +0530  

initial commit

# Let's see what's inside this tree
# We moved our 'test.txt' file inside the 'nested' directory. This creates a new tree. 
> git cat-file -p 3ea8798ffe7147c04d66c9bef3ac5109ad6e80b8      
040000 tree 2b297e643c551e76cfa1f93810c50811382f9117    nested  

# Inside the 'nested' directory is our file.
> git cat-file -p 2b297e643c551e76cfa1f93810c50811382f9117
100644 blob 9daeafb9864cf43055ae93beb0afd6c7d144bfa4    test.txt

# And inside the file is the content. 
> git cat-file blob 9daeafb9864cf43055ae93beb0afd6c7d144bfa4
test

Merging multiple repos into one

Merging multiple repositories requires us to take the objects and other entities from one repo and copy it to some other repo. For the sake of explaining, I just created 2 test repos - test-repo-1 and test-repo-2. Both contains a single 'README.MD' file with some content. We'll try merging them into a monorepo system with a parent git and these repos as subdirectories.

> cat test-repo-1/README.md
# test-repo-1

This is a test repository.

> git log --reflog  # To view all the commits
commit f23153640c79ef57849be2baac14f6daa7b96a1c (HEAD -> main, origin/main, origin/HEAD)
Author: Shubham Kumar <35415266+UnresolvedCold@users.noreply.github.com>
Date:   Tue Dec 13 11:02:33 2022 +0530

    Updated README.MD

commit 527ee4e25e8a145448bf799412c369c6cbc8e934
Author: Shubham Kumar <35415266+UnresolvedCold@users.noreply.github.com>
Date:   Tue Dec 13 11:01:57 2022 +0530

    Initial commit

> cat test-repo-2/README.md
# test-repo-2

This is some other test repo.

> git log --reflog  # To view all the commits
commit 69ed70f51232f81a29a537fb6534e91d1d1ac9c2 (HEAD -> main, origin/main, origin/HEAD)
Author: Shubham Kumar <35415266+UnresolvedCold@users.noreply.github.com>
Date:   Tue Dec 13 11:04:00 2022 +0530

    Updated README.MD <repo-2>

commit fe71fde928e17f4adbba0c637a6afc4c503f455a
Author: Shubham Kumar <35415266+UnresolvedCold@users.noreply.github.com>
Date:   Tue Dec 13 11:03:12 2022 +0530

    Initial commit

We have 2 git repos with 2 commits on main branch. Let's try merging them.

`Current folder structure`

.
├── test-repo-1
│   ├── .git        # This will be removed
│   └── README.md
└── test-repo-2
    ├── .git        # This will be removed
    └── README.md

`New folder structure`

parent
├── .git            # New git that will contain all our histories
├── test-repo-1
│   └── README.md
└── test-repo-2
    └── README.md

We will create a new git repository. And include the remotes of the repository we want to merge. We want to fetch everything from these repos. This is done by calling git fetch --all. Then, one by one, we can create a new subdirectory for each repository. And at last, we can manipulate the git history into thinking the codes were present in the subdirectory from the beginning.

> git init  # Create a new repository

# Fetch the repos we want to merge 
> git remote add repo1 git@github.com:UnresolvedCold/test-repo-1.git
> git remote add repo2 git@github.com:UnresolvedCold/test-repo-2.git
> git fetch --all
Fetching repo1
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 6 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (6/6), 1.23 KiB | 180.00 KiB/s, done.
From github.com:UnresolvedCold/test-repo-1
 * [new branch]      main       -> repo1/main
Fetching repo2
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 6 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (6/6), 1.24 KiB | 424.00 KiB/s, done.
From github.com:UnresolvedCold/test-repo-2
 * [new branch]      main       -> repo2/main

Now we have everything updated. We can start our merging process. Checkout the branch of the first repository and modify the index file such that repo1 files are inside the repo1 directory.

# Let's start with the first repo main branch
> git checkout --detach repo1/main

> git ls-files -s   # This is how the file is presently 
100644 142ee887e3184237ac17f918498bda405eeb5fc1 0	README.md

# We want to move README.md to repo1/README.md
> git ls-files -s | sed "s-\t\"*-&repo1/-"      # We want to do this for all the commits
100644 142ee887e3184237ac17f918498bda405eeb5fc1 0	repo1/README.md

# After doing this for all the commits we will update the index file
> git filter-branch --index-filter '
     git ls-files -s | sed "s-\t\"*-&repo1/-" | 
     GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info &&  mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"' 

> ls -a 
./    ../    .git/    repo1/   # A new directory called repo1 is created

> ls repo1/
README.md

# We can see our the first repo contents are inside repo1 directory
> cat README.md
# test-repo-1

This is a test repository.

The above looks good. At least we can see the files related to repo1 in repo1 subdirectory. Let's explore the history. It should be a tree called repo1 with README.md inside it.

# Let's see the git commit list
> git log --pretty=format:"%H"
495e27b84df8e4e6d297b37a03f1c6c0a5fdcc73    # This is the latest commit
8a5f4f3805be9604b1f96b03df5cf1092363ba91

# Let's view the histories for the latest commit
> git cat-file -p 495e27b84df8e4e6d297b37a03f1c6c0a5fdcc73
tree 38eccdd305b8e410a1993391031e948e664d7b1d
parent 8a5f4f3805be9604b1f96b03df5cf1092363ba91
author Shubham Kumar <35415266+UnresolvedCold@users.noreply.github.com> 1670909553 +0530
committer GitHub <noreply@github.com> 1670909553 +0530

Updated README.MD%
> git cat-file -p 38eccdd305b8e410a1993391031e948e664d7b1d
040000 tree d83102ebb26f925b8e087c821e49ca910f1750a6	repo1   # Has a directory called repo1
> git cat-file -p d83102ebb26f925b8e087c821e49ca910f1750a6
100644 blob 142ee887e3184237ac17f918498bda405eeb5fc1	README.md   # repo1 has README.MD

# Let's quickly check the second commit id
> git cat-file -p 8a5f4f3805be9604b1f96b03df5cf1092363ba91 | grep tree
tree 0dc5c3d1b78ed4dcd97120149441a8e3a8d6aefa
> git cat-file -p 0dc5c3d1b78ed4dcd97120149441a8e3a8d6aefa
040000 tree cc92c678191b4d3c4731f5d0f9b212532316ef8c	repo1     # Check
> git cat-file -p cc92c678191b4d3c4731f5d0f9b212532316ef8c
100644 blob 6a642ba8d3e31ba9a02606da98dd4a73a2d554e2	README.md # Check

We can see that both the files and commit histories were successfully migrated to repo1. Now we can do the same process for repo2. If you have more than one branch, you'll need to repeat the process for each one.

But first, we must delete the original references in order to create a new one. And we also want to merge the previous references with the new ones. So let's save the current HEAD and delete the references.

# Our current HEAD (latest commit in our case)
> git rev-parse HEAD
495e27b84df8e4e6d297b37a03f1c6c0a5fdcc73

# Save the current HEAD for merging later on
> REPO1=$(git rev-parse HEAD)

# Delete the original references
> git for-each-ref --format="%(refname)" refs/original/
refs/original/HEAD
> git update-ref -d refs/original/HEAD
> git reset --hard

We are now ready to merge repo2. Again we'll need to reset the HEAD before merging. We'll create a new branch called main and merge both repos to it one by one.

> git checkout --detach repo2/main
> git filter-branch --index-filter '
     git ls-files -s | sed "s-\t\"*-&repo2/-" | 
     GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info &&  mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"' 

# Save the reference
> REPO2=$(git rev-parse HEAD)

# Reset HEAD (again before merging)
> git update-ref -d refs/original/HEAD
> git reset --hard

Now we can begin our merging process.

> git checkout -b main
> git merge --no-commit -q $REPO1 $REPO2 --allow-unrelated-histories
# If no merge conflicts then you can just commit it else you'll need to resolve merge conflicts
> git commit -m "Migrated"

> tree
.
├── repo1
│   └── README.md
└── repo2
    └── README.md

> git log --oneline
d0b5806 (HEAD -> main) Migrated
074ef5b Updated README.MD <repo-2>
025ec51 Initial commit
495e27b Updated README.MD
8a5f4f3 Initial commit

This is the fundamental method for merging two repositories into one while retaining all commit histories. Now, of course, we never want to manually redo this for all the repos and all the branches. A script would be helpful.

Merging using Shopsys monorepo-tools

And as they say,

Don't reinvent the wheel.

There are numerous such merger tools available online, we used monorepo-tools by shopsys. There were several tweaks we did for our use case. One was that by default this tool only merges the master branch. And here we are trying to merge the main branch so, we'll need to update the shopsys repo to do this.

Under the hood, monorepo-tools is using the same commands described above.

# Replace master by main in monorepo files
# Run this inside the shopsys monorepo-tools directory
> for f in $(find *.sh); do c=$(cat $f | sed 's/master/main/g'); echo $c>$f ;done

# sed 's/master/main/g' $f > $f should work on Linux based system 
# but on MAC it just deletes all the contents of my file.

Let's see how you can merge 2 repos using shopsys monorepo-tools now.

# First clone the shopsys repo
> git clone git@github.com:shopsys/monorepo-tools.git

# Create a new repo
> mkdir parent
> cd parent
> git init

# Add the remotes and fetch
> git remote add repo1 git@github.com:UnresolvedCold/test-repo-1.git
> git remote add repo2 git@github.com:UnresolvedCold/test-repo-2.git
> git fetch --all

# Run Shopsys mono-repo tool
> ../monorepo-tools/monorepo_build.sh repo1 repo2

This is how easy it is to merge multiple repos using shopsys monorepo-tools. There are a variety of other things that you try with this tool like splitting a repo into multiple repos (reverse of what we did here). Not exploring this here.

Let's verify if everything worked !!!

> tree .
.
├── repo1
│   └── README.md
└── repo2
    └── README.md   # Both the repositories are under their sub-directories

> git log --oneline
ad3f765 (HEAD -> main) merge multiple repositories into a monorepo
074ef5b Updated README.MD <repo-2>
025ec51 Initial commit
495e27b Updated README.MD
8a5f4f3 Initial commit

# Explore the latest commit
> git cat-file -p ad3f765 | grep tree
tree 630e5e1c57649efcd9929baa790768927783659e
> git cat-file -p 630e5e1c57649efcd9929baa790768927783659e
040000 tree d83102ebb26f925b8e087c821e49ca910f1750a6	repo1   # Check
040000 tree 7801ea011e82b38c3f3de145571ad75536d5bd5c	repo2   # Check
> git cat-file -p d83102ebb26f925b8e087c821e49ca910f1750a6
100644 blob 142ee887e3184237ac17f918498bda405eeb5fc1	README.md   # Check
> git cat-file blob 142ee887e3184237ac17f918498bda405eeb5fc1
# test-repo-1

This is a test repository.
> git cat-file -p 7801ea011e82b38c3f3de145571ad75536d5bd5c
100644 blob 18c9019fe2e7d5e8151db4cb5f1d10307c8547ec	README.md   # Check
This is a test repository.
> git cat-file blob 18c9019fe2e7d5e8151db4cb5f1d10307c8547ec
# test-repo-2

This is some other test repo.

# We can also verify commits 074ef5b and 495e27b to see if README.md is inside their respective folders
> git cat-file -p 074ef5b | grep tree
tree 057f76982f62d051ed841e43c09536d3f3c61980
> git cat-file -p 057f76982f62d051ed841e43c09536d3f3c61980
040000 tree 7801ea011e82b38c3f3de145571ad75536d5bd5c	repo2     # Check
> git cat-file -p 7801ea011e82b38c3f3de145571ad75536d5bd5c
100644 blob 18c9019fe2e7d5e8151db4cb5f1d10307c8547ec	README.md # Check

> git cat-file -p 495e27b | grep tree
tree 38eccdd305b8e410a1993391031e948e664d7b1d
> git cat-file -p 38eccdd305b8e410a1993391031e948e664d7b1d
040000 tree d83102ebb26f925b8e087c821e49ca910f1750a6	repo1     # Check
> git cat-file -p d83102ebb26f925b8e087c821e49ca910f1750a6
100644 blob 142ee887e3184237ac17f918498bda405eeb5fc1	README.md # Check

Everything seems perfect.

Conclusion

The merging of multiple repositories into a monorepo structure can be accomplished by rewriting git histories. This can be done manually by fetching a repo and tweaking the git index to believe it belongs to a subdirectory rather than the main repo itself. Another way is by using open-source tools like shopsys monorepo tools with which merging can be done in a few steps. More tweaks can be done with monorepo-tools to merge multiple branches and create a higher level POM inside every commit.

A simple blogging site for personal use

Sun, 04 Dec 2022 18:30:00 GMT

This is how I quickly hosted the platform where this blog exists that you are reading right now.

There isn't a way to post comments as of now. You'll need to ping me on social platforms.

All I required was a text editor with dev container support (like VS Code), Docker and Github account. For deployment there are many static site hosting platforms. I am going with firebase hosting for deploying my site. As per the advice of one of my close friends, Guruvardhan this is built using Astro (MDX pages for blogging).

Understanding the flow

There are many templates available in astro library. The one I'm using is Lance Ross astro basic template.

Folder structure

src
├── components
├── images
├── layouts
│   ├── BlogLayout.astro
│   └── Layout.astro
└── pages
    ├── 404.astro
    ├── index.astro
    ├── posts
    │   ├── but-why-tho.mdx
    │   ├── how-to-deploy.mdx
    │   └── markdown-styling.mdx
    ├── posts.astro
    └── rss.xml.ts

src/pages/posts is where the blog posts live. All these mdx files are different blogs.

src/pages/*.astro contains the main view components of the website. The layouts are defined at src/layouts.

rss.xml.ts is the configuration for my blog's RSS feed. You can subscribe using Feedly or any other newsreader app.

Publishing the blog

The defaut branch for my blog repository is called publish. Any pushes to this will automatically trigger the rebuilding and publishing of the website.

So I create a new branch for each of my new blogs and when I'm finished, I just merge the changes to publish branch. This triggers the deploy-firebase workflow which pushes the build to firebase hosting. And the updated blog is live within seconds.

How to launch your own instance

You can always fork my repository here. You may need to activate GitHub Actions in your repository. Better way is to use Lance Ross astro basic template.

<!image/>

Click on Use this template -> Create a new repository. This will help you to create a new repository using the template. You can customize your repository as you like.

For firabse hosting, you'll need to create a new app from their website. Update the .firebasesrc with the name of your app. You'll also need firebase-cli to generate the auth token. Simply run firebase login:ci from your terminal to get this token. Save this token as GitHub Action secret as key value with key as 'FIREBASE_TOKEN' and value as the token you generated.

Get a custom domain and link it to firebase hosting

Create an account on Porkbun and buy a domain of your liking.
Go to firebase and click on hosting. There you can find Add custom domain button. <!image/>
Clicking this will ask you to provide your domain (Example: shubham.codes). Don't mention 'https' or 'www'.
In the next steps, it will ask you to update A and TXT records with your domain provider which is porkbun in our case.
1. Go to porkbun and click on the Details button on your domain name.
2. Click on DNS record and update the A and TXT. <!image/>
3. It takes some time to get activated. It's mentioned 24 hours but for me it took around 2 hours.

Shubham's Blog

Migrating blogs to Hugo

Using Jmespath in Emacs

Introduction

Installing JP

Linux

Mac

Adding Jmespath recipe in Emacs

Doom Emacs

Using Jmespath

Read more

Running PMML models in Erlang using NIF and C++

Introduction

Erlang NIF

Prerequisites

Hello Nif program

C/C++ code

Header file

Function Definition

Export functions

Final code

Erlang code

Erlang module

Define your NIF functions

Load the library on module load

Fallback function

Final code

Compiling and Running

Mac OS

Linux

cPMML NIF

C/C++ code

Header files

PMML Parser class

NIF implementation

init function

predict function

Erlang code

Compiling and Running

Conclusion

Running AI/ML predictions in CPP using cPMML library

Introduction

Creating a model file

Dependencies

Imports

The model

Dataset

Test/Train data

Training the model

Saving the pmml file

Using the model file

Installing cPMML

For Mac M1

Running the predictions

Include the library

Load the model

Start predictions

Compilation

Conclusion

Embedded Jetty server with handlers for legacy Java applications

Introduction

Jetty Server Architecture

Using Jetty in a Java application

New Java App

Add Jetty dependency

Main server

Threadpool

Server

Connector

HTTP

HTTPS

Handlers

Final code

Handler class

Extend AbstractHandler

Manage different methods

Extract the query parameters

Generating the result

Final handle method

Connect Handler and MainServer