Posted in Misc, Serverless, Tech

Serverless

Serverless is a cloud-computing execution model that dynamically manages resources. It is categorized as FaaS or Function as a Service Solution.

serverlessThe idea is to create an abstract functionality which is available “on-demand”. Cloud Platforms bill only for the time of execution of the functionality. Serverless model reduces Operation cost, complexity of provisioning servers and maintaining them and handles scaling automatically.

  • Serverless abstracts the functionality.
  • The server side logic is run on stateless computing containers which are triggered on an event.
  • It is fully managed by third-party, relieving the responsibility of managing servers.
  • Provides Increased Reliability
  • Greener Computing: It reduces the necessity of building data centers across the Globe.

Some well known Serverless solutions are Amazon’s AWS Lambda, IBM’s OpenWhisk, Google cloud service, Microsoft Azure Serverless Platform.

Expedia, Coca-cola, Thompson Reuters, EA are some examples of companies using Serverless Architecture.

Ground Rules

Some ground rules that label an Architecture as Serverless are:

  • Granularity: One or more stateless functions having a single purpose, solving a specific problem.
  • Designed as an Event-driven pipeline.
  • Zero Administration: No provisioning or Maintaining or scaling. Using third party services lets you focus on building value-adding customer features.
  • Scaling is automatically managed.
  • Cost-Saving: You can execute code on-demand and pay only for the time of execution. This helps build self-sustaining start-ups regardless of the number of customers it currently has. Scaling is also not a road-block anymore.
  • Rapid Time to Market: Shortens the Time between an Idea evolving into a Product taking away the entire overhead of procuring servers.

 

What is not Serverless and Why?

  • PaaS or Platform as a Service like Heroku, Salesforce or Google App Engine cannot bring the entire application up/down in response to an event unlike how a FaaS works.
  • Container Platform like Docker and Container Hosting Systems like Mesos, Kubernetes need you to manage the size and shape of the cluster. FaaS can automatically handle resource provisioning, allocation and scaling.
  • Stored Procedures as a Service – They often require to be written in a specific Framework or language and is hard to unit test as they are database dependance.

 

AWS Lambda: 

AWS Lambda can run scripts/app on the Amazon’s cloud environment. Amazon charges only when the function is used. Hence you pay as you use regardless of whether your business has a single user or a million. AWS Lambda provides an integrated solution for computing as well as storage, using Amazon S3.

  • AWS Lambda provides a lot of pre-configured templates to choose from instead of writing the Lambda function from scratch.
  • In Python, boto3 is the library used in order to create an Identity and Access Management (IAM) Role to securely access AWS resources. An example given below.
  • A function can be created inline or by uploading a .zip file.
  • Once created, the lambda function can be triggered via an HTTP call, using the “Add API Endpoint” feature.

Here’s an example of a Lambda function:

import boto3, json

lambda_client = boto3.client('lambda')

def lambda_handler(event, context):
   message = 'Hello {} {}!'.format(event['first_name'], 
                                    event['last_name'])  
    return { 
        'message' : message
    } 

lambda_client.create_function(
  FunctionName='exampleLambdaFunction',
  Runtime='python2.7',
  Role=role['Role']['Arn'],
  Handler='main.handler',
  Code=dict(ZipFile=zipped_code),
  Timeout=300
)

The above function can be invoked using the below snippet:

import boto3, json

lambda_client = boto3.client('lambda')

lambda_client.invoke(
FunctionName='exampleLambdaFunction',
InvocationType='Event',
Payload=json.dumps(test_event),
)

 

Drawbacks

The one major drawback is the dependency on the third-party/ vendor whose services are being used, taking away control over system down time, cost changes etc. Also, it is very hard to port from one vendor to another and always involves shifting the entire infrastructure. Privacy and security is also a major concern when the application is built for handling sensitive information.

Posted in Java for Python Developers, Tech

Java for Python Developers 1 – Basics

The intention of this blog is to talk about the similarities and differences between Python and Java. This will be a series of blogposts, this being the first. Through this blog I plan to help myself and other Python developers learn Java easily by relating it to similar concepts in Python. The focus of this post will be on learning Basic concepts and writing some simple code in Java.

Why learn Java?

  • Java is a very popular and a widely used Programming language. It is used in building large systems due to it’s powerful JVM that leads to faster execution. When an Application needs to scale-up, Java is the most popular choice in the industry.
  • It supports Concurrency.
  • Java has a good cross platform support.
  • It has a very strong IDE support making the developer’s life easier.
  • It imposes certain best practices. There is most likely, a single way of doing stuff and hence a bad programmer can only harm so much. :\ But in case of languages like Python, there are several ways of writing a piece of code and the onus is completely on the developer to use the most optimized one.

Classes and Objects:

The concept of Classes, Objects and Object Oriented programming is similar to Python. But here’s the catch:

googleapisjavapython

Each .java file can only have a single public class that contains the main function. This is to know the exact point in the file from which the application is supposed to be launched. The name of this public class should be same as the name of the .java file.

Variables:

Again, the concept is very similar to Python including the naming conventions.

Java cares about Type. An Elephant cannot be put in a basket meant for vegetables. This is very much unlike Python, which decides if we need a Basket or a Cage based on whether the source is an Elephant or a vegetable that we are dealing with. 🙂 Moreover,

  • variables declared as final cannot be reassigned, once assigned to some value.
  • A variable of a particular type can be assigned to another of the same type. Example, a variable of the type Tiger (Tiger t1 = new Tiger()) can be assigned to any another variable of the type Tiger. (Tiger t2 = new Tiger(); t2 = t1;)
  • Java is PASS-BY-VALUE.

Global Variables in Python can be used as global x = 3. Java has no concept of global variables as such. However, a variable marked as public, static and final acts as a globally available constant.

Conditional Statement:

if(var1 == var2) { <do this>
}

Iteration:

WHILE Loop:

Use while loop when we do not know the exact number of times to loop, in advance. In other words, when we have to loop until a condition is satisfied.

while(boolean condn) { <repeat this>
}

FOR Loop:

for(int i=0;i<anArray.length;i++){
<repeat array-length number of times>
}

Enhanced FOR loop: (For each loop)

String[] AnArray = {"John","Bob","Andy","George","Rachel"};
System.out.println("Names: ");
for(String name: AnArray) {
    System.out.println(name);
}
teach-2562905_1280
DO NOT forget the SEMICOLON;

Java Library or Java API

Let’s now writing some simple code in Java to achieve certain redundant tasks using the built-in Libraries.

ArrayList:

ArrayList can perform some frequently used operations on Lists which otherwise takes multiple lines of code and iterations to achieve the same. Similar to the built-in functions for a List in Python, elements can be added to the list, removed, check if an element is present in a list, get the size of the list, get the index of an element etc. as follows:

import java.util.ArrayList;
ArrayList<String> ListPlaces = new ArrayList<String>();
String p1 = new String("Switzerland");
String p2 = new String("Singapore");
ListPlaces.add(p1);
ListPlaces.add(0,p2); //0 - index where p1 should be added
System.out.println(ListPlaces);
System.out.println(ListPlaces.size());
System.out.println(ListPlaces.get(0));
System.out.println(ListPlaces.indexOf(p1));
System.out.println(ListPlaces.remove(1));
System.out.println(ListPlaces.contains(p2));
System.out.println(ListPlaces.isEmpty());

Generating a random number:

int Rand = (int) (Math.random() * 10);

*(int) -> Type casting the return value of the random() function to an integer.

Obtaining user input from command line:

import java.util.Scanner;
System.out.println("Please enter your name: ");
Scanner sc = new Scanner(System.in);
String username = sc.nextLine();

Let’s play Battleship

Here’s an example program, the battleship game written in Java. We will play this on command-line. The game randomly assigns 3 Battleships on a 7×7 board. It allows the user to guess the location. On successful identification of each ship, it says “kill”. When all the ships sink, games ends with a score.

There may be better ways of writing this code but at this stage, we are just trying to get a step closer to learning Java. Here’s an implementation of the battleship game.

This code is also available on Github: https://github.com/shilpavijay/Battleship

import java.util.*;
import java.io.*;

public class Battleship {
    private GameHelper helper = new GameHelper();
    private ArrayList<PlayBattleship> battleList = new ArrayList<PlayBattleship>();
    private int noOfGuesses = 0;

    private void setUpGame() {
        PlayBattleship ship1 = new PlayBattleship();
        ship1.setName("Maximus");
        PlayBattleship ship2 = new PlayBattleship();
        ship2.setName("Aloy");
        PlayBattleship ship3 = new PlayBattleship();
        ship3.setName("Agnes");
        battleList.add(ship1);
        battleList.add(ship2);
        battleList.add(ship3);
        System.out.println(battleList);

        System.out.println("Your goal is to sink the three ships: Maximus, Aloy and Agnes with the least number of guesses.");
        for (PlayBattleship shipToset : battleList) {
            ArrayList<String> newLocation = helper.placeShip(3);
            shipToset.setLocation(newLocation);
        }
    }

    private void startPlaying() {

        while(!battleList.isEmpty()) {
            String userGuess = helper.getUserInput("Enter a guess");
            checkUserGuess(userGuess);
        }
        finishGame();
    }


    private void checkUserGuess(String userGuess) {
        noOfGuesses++;
        String result = "miss";

        for(int i=0;i<battleList.size();i++) {
            result = battleList.get(i).checkTheGuess(userGuess);
            if(result.equals("hit")) {
                break;
            }
            if(result.equals("kill")) {
                battleList.remove(i);
                break;
            }
        }
        System.out.println(result);
    }

    void finishGame() {
        System.out.println("Wow!!! You sunk all the ships! Congrats!");
        if(noOfGuesses <= 18) {
            System.out.println("Score: Congrats, you sunk all ships in " + noOfGuesses + " guesses!");
        }
        else {
            System.out.println("You have exhausted your options! Sorry! Please try again!");
        }
    }

    public static void main(String[] args) {
        Battleship game = new Battleship();
        game.setUpGame();
        game.startPlaying();
    }
}
class PlayBattleship {
    private ArrayList<String> location;
    private String name;

    public void setLocation(ArrayList<String> battleList) {
        location = battleList;
    }

    public void setName(String n){
        name = n;
    }
    public String checkTheGuess(String choice) {
        String result = "miss";
        int index = location.indexOf(choice);
        if(index>=0) {
            location.remove(index);
            if (location.isEmpty()) {
                result = "kill";
                System.out.println("You sunk " + name + ":|");
            }
            else {
                result = "hit";
            }
        }
        return result;
    }
}
class GameHelper {
    private static final String alphabet = "abcdefg";
    private int gridlength = 7;
    private int gridSize = 49;
    private int[] grid = new int[gridSize];
    private int comcount = 0;

    public String getUserInput(String prompt) {
        String inputLine = null;
        System.out.println(prompt);
        try {
            BufferedReader input = new BufferedReader(new InputStreamReader(System.in));
            inputLine = input.readLine();
            if(inputLine.length() == 0) return null;
        }
        catch(IOException e){
            System.out.println("IOException: " + e);
        }
        return inputLine.toLowerCase();
    }

    public ArrayList<String> placeShip(int size) {
        ArrayList<String> cells = new ArrayList<String>();
        String temp = null;
        int [] curr = new int[size];
        int attempts = 0;
        boolean success = false;
        int location = 0;

        comcount++;
        int incr = 1;
        if((comcount % 2) == 0) {
            incr = gridlength;
        }

        while(!success & attempts++ < 200) {
            location = (int) (Math.random() * gridSize);
            int x = 0;
            success = true;
            while(success && x<size) {
                if(grid[location] == 0) {
                    curr[x++] = location;
                    location += incr;
                    if (location >= gridSize) {
                        success = false;
                    }
                    if (x>0 && (location % gridlength == 0)) {
                        success = false;
                    }
                }
                else {
                    success = false;
                }
            }
        }

        int x = 0;
        int row = 0;
        int column = 0;

        while (x < size) {
            grid[curr[x]] = 1;
            row = (int) (curr[x]/gridlength);
            column = curr[x] % gridlength;
            temp = String.valueOf(alphabet.charAt(column));

            cells.add(temp.concat(Integer.toString(row)));
            x++;
//            System.out.print(" co-ord "+x+" = " + cells.get(x-1).toUpperCase());
        }
        return cells;
    }
}

Conclusion:

In this blogpost we learnt how to use Variables, write Classes, conditional statements, loops in Java. We also glanced at some basics APIs or built-in libraries in Java that help perform certain redundant operations easily. We also put these concepts in action by coding a game.

Posted in AI, Tech

Evaluating your Classifier

The previous post: Find your best fit

This post is a continuation of the previous one. This one shall discuss a few more ways of evaluating a Machine Learning Algorithm – a Classification Model, in particular.

Learning Curve

learning_curve

Say we have fit some function to ‘x’ number of training samples. The error of this function keeps increasing, as more and more data samples are added. However, after a certain limit, say ‘n’ number of samples, the error value will plateau.

Plotting a learning curve can assist in taking decisions on collecting more samples to enhance the performance of the classifier on the Test set. It is not going to make much difference to the performance if the learning curve has already reached the plateau.

Precision and Recall

Let’s consider an Email Spam Classifier. The prediction of the classifier vs the actual result, can be one of the following: [Here, 0 -> False, 1 -> True]

prec_recall_table

If the spam classifier predicts an email to be a spam (Predicted class is 1) and it turns out to be actually a spam (i.e. Actual class is also 1) then the result is said to be ‘True Positive’. The classification was accurate. However, if the spam classifier classifies the email to be a spam (Predicted class is 1) and the email is not actually a spam (Actual class is 0) then the result is said to be ‘False Positive’. The classifier seems to have erroneously classified the mail as spam. False Negative and True Negative can be similarly interpreted.

Precision and Recall are further used to calculate F1-Score, that measures the performance of a classification algorithm. Let us see what they mean and how they are calculated.

Precision = True Positive/(True Positive + False Positive)

Precision: Of all the emails that were predicted to be spam, the fraction of those that were actually spam is called the Precision.

Recall = True Positive / (True Positive + False Negative)

Recall: Of all the emails that actually are spam, Recall is the fraction that was correctly predicted as spam.

prec_recall

The relationship between Precision and Recall is illustrated in this graph. They are inversely related. There is a threshold where the classifier performs the best and has the highest precision.

F1-Score

F1-Score is the weighted average of Precision and Recall. It is calculated as:

F1-Score = 2 (Precision * Recall) / (Precision + Recall)

Here’s an example of three different algorithms with their precision, recall and F1 score:

Fscore

The first algorithm has the highest F-Score and hence performs the best.

Evaluating an Algorithm at an early stage results in making a calculative decision on further steps to be taken to improve performance. It is usually suggested to make a very quick and dirty implementation using a simple algorithm initially, and then improvise, by analyzing the algorithm and taking decisions like adding more/reducing features, collecting more samples etc.

Posted in AI, Tech

Find your Best Fit!

Let’s say we have chosen and implemented the best Machine Learning Algorithm, suitable for the data of our choice but ended up figuring out that the algorithm is actually making unacceptably large errors in prediction. What do we do next?

Eyes Cat Domestic Cat Kitten Cat's Eyes Surprised

Let’s discuss some choices that can be made in such a scenario and also get an intuition of what can be expected out of them. These can be used in general, for debugging or analyzing the performance of a Machine Learning Algorithm on a specific implementation.

  1. Add more training examples
  2. Try smaller set of features
  3. Try getting additional features
  4. Add polynomial features
  5. Increasing / decreasing the regularization parameter

The result of these can only make sense after understanding some underlying concepts like Over-fit, Under-fit, Bias, Variance etc. Here is what they mean:

Under-fit and Over-fit:

UnderfitAs illustrated in the graph, if the regression model fits only few data samples, it is said to be under-fitting. Insufficient number of Features might cause under-fitting. This is also called High Bias.

 

Overfit

 

As we can see, the model fits almost all the data samples provided to the algorithm. It may perform well on the training set but will not generalize well enough when new samples are encountered in the test set. This is also called High Variance.

Bias vs Variance:

When the training set error and the test set error are plotted against, say, the size of the data sample, we get a graph similar to that given below.

Bias_Variance

As more number of sample are fed, the algorithm performs excellently on the training set, thus reducing the error drastically. Whereas, on the test set it may performs poorly leading to high error.

[Error is the difference between the prediction made by the model and the actual value. The entire data sample is usually divided into a Training set, a Test set and sometimes a Cross validation set. The predictive model is trained using the training set. Cross validation and test sets are used to validation and Testing the model accordingly.]

High Bias: If both the training set error and the test set error are high, we can incur that the algorithm is suffering a high bias or is under-fitting.

High Variance: In case the training set error is very low and error on test set is very high when compared to the training set, the algorithm is suffering from high variance or is over-fitting.

Back to the diagnosis:

Now that we have the required background, let’s go back to the diagnosis and analyze each of our choices.

  1. Add more training examples:
    • If the test set error is too higher than that of the training set and we conclude the possibility of high variance, we can fix it by collecting more data samples and training the model again.
    • This might not be such a great idea if both the training and test set errors are high.
  2. Try getting additional features:
    • Say the algorithm is under-fitting the samples, the number of features might not be sufficient for the model to make accurate predictions. Collecting some more features may prove to be helpful.
    • For Example, if the model is predicting the price of a hotel, adding more features like number of rooms, room size, floor elevation, balconies, furniture etc can act as additional features.
  3. Try smaller set of features:
    • If the algorithm is over-fitting the data samples; in other words, it performs extremely well on the training set but is highly erroneous on the test set, adding more features might help improve the performance.
  4. Add polynomial features:
    • In some scenarios, we may be confined only to a limited set of features and adding more number of features might not be possible. In such case, an additional term can be added to the model, using an existing feature.
    • A linear model, for example, can be transformed into a quadratic model or a cubic model by adding an additional term which can be the square of the size of the room (or the cube)
    • Adding a polynomial feature can fix high Bias.
  5. Increasing / decreasing the regularization parameter:
    • Cost function is the best possible hypothesis/model to the training set that has the minimum squared error. To minimize the error or optimize the cost function, a regularization parameter called lambda is added.
    • Increasing the regularization parameter penalizes the model vector growing arbitrarily, forcing the optimization function to choose smaller values of the weights (thetha). This results in fitting a better model to the data when the existing one is suffering from high variance/over-fitting.
    • Similarly, decreasing the regularization parameter leads to higher values of thetha, fitting the model better when it is under-fitting. In other words, it fixes high bias.

These choices are not exhaustive but identifying the right step to be taken can save a lot of time and help arrive at the Best Fit, faster!

Posted in Elastic Stack, Misc, Tech

Elastic Stack

Server logs contain some of the most valuable and untapped information. Logs are always unstructured and usually makes little sense. Various opportunities of improvement might unveil by deriving insights from them. Elastic Stack or the ELK Stack is the most widely used solution for Log Analysis. It is Open Source and has a massive community pushing the boundaries by adapting it into various scalable systems. Companies including Microsoft, LinkedIn, Netflix, ebay, SoundCloud, StackOverflow use the Elastic Stack.

Elastic Stack

ELK is an acronym for three open source projects:

  • Elasticsearch:  A search and Analytics Engine. It is an open source, Distributed, RESTFul, JSON based search engine.
  • Logstash: A Server side data Processing Pipeline that can ingest data from multiple sources. It then transforms and send data to Elasticsearch.
  • Kibana: Visualizes data with charts and graphs.
  • Beats: A light-weight single purpose Data Shipper. Beats have a small installation footprint and use limited system resources. It can either directly send data to Elasticsearch or send it via Logstash. Beats is written in Go! 

Logstash along with Beats collects and parses log data from multiple sources. Elasticsearch indexes and stores this information.
Kibana visualizes this information to provide insights.

Elasticsearch

Elasticsearch is worth discussing in-detail. It is widely used for Full Text search. It is written in Java. This powerful search engine is designed to scale-up to millions of search events per second. Elasticsearch is used by Wikipedia, Airbus, ebay and shopify for powering their search for near-real time access. Its powerful features:

  • Scalability
  • Highly Available
  • Multi-tenancy
  • Developer friendly

Logstash

Logstash supports data of many formats coming from various systems. It can ingest data from logs, web applications, Data stores, Network devices, AWS services and REST endpoints. It then parses and transforms data, identifies named fields to build the structure and converts into a common format.logstash

  • Provides around 200 plugins to mix and match and build the data pipeline. It also provides the feature to build a plugin to ingest from a custom application.
  • Pipelines can be very complicated and cumbersome to monitor Load, Performance, Latency, Availability etc. Centralized monitoring is provided by the “monitoring and pipeline viewer” that makes the task easier and understandable.
  • Structures, transforms and enriches data with filter plugins
  • Can emit data to Elasticsearch or other destinations using output plugins like TCP or UDP
  • Logstash is horizontally scalableSecurity: Incoming data from Beats can be encrypted. Logstash also integrates with secured Elasticsearch clusters. 

Kibana

Kibana provides interactive visuals of Elasticsearch data to monitor the behavior, understand the impact of certain data changes and so on.

  • Kibana core comes with histograms, line charts, pie charts, sunburst and many other classics.
  • Plots Geospatial data on any given map.
  • Can perform advanced Time Series analysis.
  • Graph Exploration: Analyzing Relationships with Graphs
  • Build customized canvas, add logos, elements and create a story.
  • Can easily share dashboards across the organization 

Problems ELK can solve:

  • In a distributed system with several nodes, searching through several  log files for certain information, using unix commands is a tedious task. Elasticsearch comes to the rescue by providing faster access along with Logstash+Beats by collecting logs from all the nodes.
  • Ship Reports: Kibana provides faster ways to explore and visualize data. It can schedule and email reports. Can quickly export the results of ad-hoc analysis or saved searches into a CSV file. Alerting can be used to generate data dumps when certain conditions are met, or on a regular interval.
  • Alerting feature can set alerts on data changes, that can be identified using the Elasticsearch query language. Can proactively identify intrusion attempts, trend in social media, peak-hours in network traffic and can also learn from its own Alerting history. It comes with built-in integrations for email, Slack, HipChat etc.
  • Unsupervised Learning: The Machine learning features have the ability to detect different kinds of anomalies, unusual network activities and quick root cause identification.

It can also integrate with Graph APIs to analyze relationships in data. Canvas can be used to build presentations and organize reports. Elastic Stack has been extending its features and exploring many possibilities. 

Useful Resources:

  1. Elastic Stack
  2. Kibana Live Demo
  3. Logstash – Video

Posted in Building REST APIs, Golang, Misc, Tech

Build RESTful API in Golang

Welcome again! After learning some Golang, my next experiment was to build APIs. So, here’s a blog-post for someone interested in trying it out. The only prerequisite to building RESTful APIs is to know the basics of Golang along with some SQL. Link to my blog-post series on Introduction to Golang may come handy.

The code can be broadly divided into two files:

  • migrate.go: Connects to MySQL and creates the necessary tables.
  • base.go: Contains all the API handlers.

Step 1: Establishing Database Connection

We will now be working on the migrate.go file. Certain packages are to be imported before we can start over.

import (
"fmt"
"database/sql"
_ "github.com/go-sql-driver/mysql"
)

The package sql provide an interface for the SQL database. In other words, it equips Golang to work with database and create a connection, executing queries and perform other SQL operations. The sql package works along with a database driver. We have used Go’s mysql driver in this case: go-sql-driver/mysql.

db, err := sql.Open("mysql","root:password@(127.0.0.1:3306)/gotest")
if err != nil {
fmt.Println(err.Error())
} else {
fmt.Println("Connection established successfully")
}
defer db.Close()

sql.Open() opens a database connection to MySQL. This function accepts two parameters – the database used (mysql in this case) and the connection details in the format: username: password@hostname/database_name. Here ‘gotest‘ is the database created to run this project. Appropriate Error message is shown when the connection fails. Defer is used to wait until all the surrounding functions return. Only then is the connection  to the database closed.

err = db.Ping()
if err != nil {
fmt.Println(err.Error())
}

In case the machine is not reachable at all,  sql.Open() does not throw an error. Hence we need to explicitly ping the machine in order to verify the same.

STEP 2: Creating Tables

stmt, err := db.Prepare("CREATE TABLE emp (id int NOT NULL AUTO_INCREMENT, first_name varchar(40), last_name varchar(40), PRIMARY KEY(id));")
if err != nil {
fmt.Println(err.Error())
}
_,err = stmt.Exec()
if err != nil {
fmt.Println(err.Error())
} else {
fmt.Println("emp table migration successful")
}

Now let’s create the table ‘emp‘ using db.Prepare(). Just to recap, ‘db’ is the handler we obtained after connecting to the database. The SQL create statement is provided as parameter. It returns a handler which is then executed using Exec() function. It also returns an error if the function fails. Error handling is coded accordingly in the last few lines.

STEP 3: Building the APIs

We now have everything in-place to start building the APIs. Create the file base.go. We will be importing some new libraries apart from those we have already come across. “Gin” is a HTTP web framework written in golang. The package “bytes” implements functions for manipulating byte slices. “net/http” provides implementations for Client Server Communication.

import (
"fmt"
"database/sql"
"bytes"
"net/http"
"encoding/json"
"github.com/gin-gonic/gin"
_ "github.com/go-sql-driver/mysql"
)

Next, create a database connection using sql.Open() along with implementation of  error handling.

func main() {
db, err := sql.Open("mysql","root:pwd@(127.0.0.1:3306)/gotest")
if err != nil {
fmt.Println(err.Error())
} else {
fmt.Println("Connection established successfully")
}
defer db.Close()
//Checking for connection:
err = db.Ping()
if err != nil {
fmt.Println(err.Error())
}

Create a golang type called Emp which will hold each row from the database and help in processing them. The type is a carbon copy for a typical row in the database containing “id”, “first_name” and “last_name” as its variables.

type Emp struct {
id int
first_name string
last_name string
}

Step 4: The GET request

A simple GET request can be created with the help of Gin which creates a handler to place a http GET request. It takes as input the URL string which is to be matched and function to handle the data returned.

router.GET("/emp/:id", func(c *gin.Context) {
var (
emp Emp
result gin.H
)
id := c.Param("id")
row := db.QueryRow("select id,first_name,last_name from emp where id = ?;",id)
err := row.Scan(&emp.id,&emp.first_name,&emp.last_name)
if err != nil {
//if no results found send Nill
result = gin.H{
"result": nil,
"count": 0,
}} else {
result = gin.H{
"result": gin.H{
"id": emp.id,
"first_name": emp.first_name,
"last_name": emp.last_name,
},
"count": 1,
}}
c.JSON(http.StatusOK, result)
})

 

Here the handler will match /emp/1/ but will not match /emp. The function stores the id, first name and last name into the type Emp by querying the database with the id obtained from the user in the GET request. The result string is constructed and passed as a JSON which serves as the output.

Similarly,

  • A GET request can be constructed to obtain all the employee data in the database.
  • POST request can be used to insert a new record or
  • PUT can modify an existing record.
  • You can similarly delete one or more records using the DELETE request.

Code for each of the above APIs can be found on my github.

Finally, let us see the above APIs in Action.

Here’s an example of a PUT request. We are here updating the first & last name of employee #2. The port in use is 3000. Note the message shown on successful execution.

PUT

Let’s now verify if the name of employee #2 was successfully modified as claimed by the above output. Here’s the GET request:

GET

The entire code can be found on: Github/RESTful-API-with-Golang-and-MySql

Posted in Golang, Tech

Go – Concurrency

Concurrency is when two or more tasks make progress simultaneously. It could be very tedious to get it right in most of the programming languages. However, Go has a rich support to concurrency and makes the task very easy.

Goroutines:

Goroutines are functions or methods that can run concurrently with other methods. They are light-weight threads. The cost of creating a goroutine is very small compared to that of an actual thread. Hence a GO application commonly has thousands of goroutines running concurrently.

  • Goroutines are managed by GO runtimes
  • go exroutine(a,b) starts a new goroutine that runs exroutine(a,b)
  • Here, the evaluation of exroutine,a,b happens in the current goroutine. The execution happens in the new goroutine.
package main

import (
	"fmt"
	"time"
)

func say(s string) {
	for i := 0; i < 10; i++ {
		time.Sleep(100 * time.Millisecond)
		fmt.Println(s)
	}
}

func main() {
	go say("is fun")
	say("learning go")
}

Channels:

  • Channels facilitate sending and receiving values. It is achieved using the channel operator ‘<-‘
  • Sending a to channel chch <- a
  • Receive from ch and assign it to aa := <-ch (Data flows in the direction of the arrow)
  • Channels must be created before using it: ch := make(chan int)
  • Send and Receive blocks implicitly until the other side is ready and hence is synchronized.
package main

import "fmt"

func sum(s []int, c chan int){
   sum := 0
   for _,v := range s{
      sum += v
   }
   c <- sum
}

func main(){
   lst := []int{200, 100, 900, 30, 40, 70, 400, 450}
   c := make(chan int)
   go sum(lst[:len(lst)/2],c)
   go sum(lst[len(lst)/2:],c)
   x,y := <-c, <-c
   fmt.Println(x,y,x+y)
}
  • Channels can also be bufferred. The buffer length should be provided as the second argument to makech := make(chan int, 100)
  • A channel can be closed to indicate that no more values can be sent. close(ch)
  • Only the sender should close a channel (not the receiver).
  • Receivers can check if a channel has been closed using the second parameter:       a, ok := <-ch ok is False if the channel is closed.
  • range can be used to keep receiving values until the channel is closed.
for i := range ch {
   fmt.Println(i)
} 
  • Closing a channel is not mandatory. It is only required to send a message to the receiver that no more values are being sent through the channel. It can also be useful to terminate a range loop.

Select

Select lets a goroutine wait on multiple communication operations. It chooses the case that is ready. If multiple are ready, it randomly makes a choice. Default case is chosen if no other case is ready.

select {
	case c <- x:
		x, y = y, x+y
	case <-quit:
		fmt.Println("quit")
		return
	default:
         	fmt.Println("    .")
	}

Mutex:

Mutex or mutual exclusion ensures that only one goroutine can access a variable at a given time to avoid conflicts. Mutex is provided by the sync library which has two methods: Lock and Unlock.

  • A block of code can be executed in mutual exclusion by enclosing it in a call to lock.
  • Defer ensures the mutex is unlocked when required.
package main

import (
	"fmt"
	"sync"
	"time"
)

type mutexUsage struct {
	v   map[string]int
	mx sync.Mutex
}

func (m *mutexUsage) Incr(key string) {
	m.mx.Lock()
	m.v[key]++
	m.mx.Unlock()
}

func (m *mutexUsage) Value(key string) int {
	m.mx.Lock()
	defer m.mx.Unlock()
	return m.v[key]
}

func main() {
	m := mutexUsage{v: make(map[string]int)}
	for i := 0; i < 100; i++ {
		go m.Incr("somekey")
	}

	time.Sleep(time.Second)
	fmt.Println(m.Value("somekey"))
}

Previous Post: Methods and Interfaces

Posted in Golang, Tech

Go – Methods and Interfaces

Methods

GO has no Classes. However, it has Methods that can be defined on ‘types’. A Method is a function with a special ‘receiver’ argument. In the example below, ‘v‘ is the receiver of the type ‘Square‘, to the method ‘Sqroot‘.

package main

import (
   "math"
   "fmt"
)

type Square struct {
   x, y float64
}

func (v Square) Sqroot() float64 {
   return math.Sqrt(v.x*v.x + v.y*v.y)
}

func main() {
   s := Square{3,4}
   fmt.Println(s.Sqroot())
}
  • Methods can be declared on non-struct types too.
  • A Method can only be declared on a type which is present in the same package as the method.

Pointer Receivers

A method can also be declared with the receiver being a Pointer. A Pointer receiver is very commonly used, as it can modify the underlying value of the type.

There can be two reasons to choose a pointer receiver instead of a value. First is to modify the underlying value and second to avoid copying the value on each method call. If the receiver is large, pointer receiver can prove to be more efficient.

package main

import (
   "math"
   "fmt"
)

type Square struct {
   x, y float64
}

func (v Square) Sqroot() float64 {
   return math.Sqrt(v.x*v.x + v.y*v.y)
}

func (v *Square) Scale(val float64) {
   v.x = v.x * val
   v.y = v.y * val
}

func main() {
   s := Square{3,4}
   s.Scale(100)
   fmt.Println(s.Sqroot())
}

Interfaces

An Interface is a set of method signatures.

type ExInt interface {
	Ifs() float64
}
func(f float64) Ifs() float64 {
   return float64(-f)
}
  • Interfaces are implemented implicitly. No “implements” keyword required.
  • Interfaces can have ‘nil‘ underlying value.
  • An empty interface can hold value of any type interface{}

Errors:

Go uses error values to express error states. An Error type is a built in Interface which looks like this:

type error interface {
    Error() string
}

Functions return an Error value. The calling code should handle errors by testing if the error equals ‘nil

Reader:

  • The reader interface is a part of the ‘io’ package.
  • It helps identify when end of stream of data is reached.
  • The Read method populates the number of bytes of data as specified.
  • The two return values are 1)Number of bytes populated and 2) An error value.
package main

import (
    "fmt"
    "io"
    "strings"
)

func main() {
    r := strings.NewReader("You are using Readers!")

    b := make([]byte, 8)
    for {
        n, err := r.Read(b)
	fmt.Printf("n = %v err = %v b = %v\n", n, err, b)
	fmt.Printf("b[:n] = %q\n", b[:n])
	if err == io.EOF {
	    break
 	    }
        }
}

Output:

n = 8 err =  b = [89 111 117 32 97 114 101 32]
b[:n] = "You are "
n = 8 err =  b = [117 115 105 110 103 32 82 101]
b[:n] = "using Re"
n = 6 err =  b = [97 100 101 114 115 33 82 101]
b[:n] = "aders!"
n = 0 err = EOF b = [97 100 101 114 115 33 82 101]
b[:n] = ""

Previous post: GO – Types (part 2)

Next Post: Go – Concurrency

Posted in Golang, Tech

GO – Types (part 2)

Range

Range iterates over a slice. Range returns two values while iterating over a slice. First is the index of the element and second a copy of the element at that index.

package main

import "fmt"

var t = []int{1, 3, 6, 9, 12, 15, 18, 21}

func main() {
	
	for i,v := range t {
	   fmt.Printf("%d => %d\n", i, v)
	}
}
  • Index or Value can be skipped by assigning it to _.
  • If only indexes are required, the “value” can be dropped entirely.
func main() {
	
	fmt.Printf("Values:\n")
	for _,v := range t {
	   fmt.Printf("%d, ", v)
	}
	
    fmt.Printf("\nIndex:\n")
    for i:= range t {
	   fmt.Printf("%d, ", i)
	}

}

Map

Map is nothing but a key-value pair. ‘Make’ function can be used to initialize a map. Zero value of a map is ‘Nil‘. Keys cannot be added to a ‘nil’ map.

package main

import "fmt"

var Emp map[string]int

func main() {
   Emp = make(map[string]int)
   Emp["John"] = 34
   Emp["Jeff"] = 20
   fmt.Println(Emp)
   fmt.Println(Emp["John"])
}

Map Literals:

package main

import "fmt"

type address struct {
  street, city string
  }

var Emp = map[string]address{
	"John": address{"abc street","London"},
	"Jeff": {"def street","Boston"},
}

The top-level type name (address in this case) can be omitted as well.

Mutating Maps

m = make(map[string]int)

  • Insert/Update” m["Mary"] = 58
  • Retrieve an element: fmt.Println(m["Mary"])
  • delete: delete(m,"Mary")
  • Check if a key is present: val, present := m["Jerry"] If the key is present, val will return the value and present will be True. If the key is not present, val will be 0 and present will be False.

Implementing Word Count

A small exercise to create a Map which will contain the count of each letter in a given word:

package main

import (
	"strings"
	"fmt"
)

func WordCount(s string) map[string]int {
    split := strings.Fields(s)
	wc := make(map[string]int)
	for i:=0;i<len(split);i++ {
	   wc[split[i]] = len(split[i])
	}
	return wc
}

func main() {
	fmt.Println(WordCount("I am learning Go!"))
}

Function values

In GO, Functions can also be passed around, similar to values.

package main

import (
  "fmt"
  "math"
)

func compute(fn func(float64, float64) float64) float64 {
   return fn(4,3)
}

func main() {
   sqrt := func(x,y float64) float64 {
       return math.Sqrt(x*x + y*y)
    }
   fmt.Println(compute(math.Pow))
   fmt.Println(compute(sqrt))
}

Function Closures

Closure is a function that references values from outside its body.

package main

import (
 "fmt"
)

func closure() func(int) int {
 sum := 0
 return func(x int) int {
 sum +=x
 return sum
 }
}
func main() {
 sum1,sum2 := closure(), closure()
 for i:=0;i<10;i++ {
 fmt.Println(sum1(i),sum2(-1*i))
 }
}

Previous post: GO – Types (part 1)

Next Post: Go – Methods and Interfaces

Posted in Golang, Tech

GO – Types (part 1)

Pointers

264px-Pointers.svg

 

  • GO has pointers as in C. A pointer holds the memory address of a value.
  • & operator generates a pointer to its operand
  • ‘*’ operator refers to the underlying value of the pointer

 

package main

import "fmt"

func main() {
  i := 24
  p := &i
  fmt.Println(p)
  *p = *p/2      // This is called "dereferencing" or "indirecting".
  fmt.Println(i)
}

Output:

0xc420082010
12

Structs

Struct is a collection of fields. The fields can be accessed using a dot(.)

package main

import "fmt"

type Emp struct {
   Name string
   Age int
   ID string
}

func main() {
  e1 := Emp{"John",23,"1234"}
  fmt.Println(e1)
  e1.Age = 30
  fmt.Println(e1)
}

Pointers to Structs

Stucts can be accessed through a struct pointer. The struct field (Name) of a struct pointer (sp) can be accessed as sp.Name

package main

import "fmt"

type Emp struct {
   Name string
   Age int
   ID string
}

func main() {
  e1 := Emp{"John",23,"1234"}
  sp := &e1
  fmt.Println(sp.Name)
  sp.Name = "James"
  fmt.Println(sp.Name)
}

Struct Literals

Struct Literal is a newly allocated struct value. It is not required to allocate values to all the fields. Just a few fields can be used. The order of assignment also doesn’t matter. The unassigned values are implicitly assigned the Zero value of its type.

package main

import "fmt"

type strLit struct {
   X int
   Y string
   Z bool
}

var (   //Struct Literals:
  s1 = strLit{}   
  s2 = strLit{X:1,Y:"golang"}
)

func main() {
  fmt.Println(s1)
  fmt.Println(s2)
}

Arrays:

Array is of Fixed size in GO. It is declared as abc [n]T where abc is an Array of n values with the type T.

func main() {
   var a [2]string
   a[0] = "Dobby is a"
   a[1] = "free elf"
   fmt.Println(a)
   fmt.Println(a[0],a[1])
   
   vowels := [6]string{"a","e","i","o","u"}
   fmt.Println(vowels)
}

Slice literal

A slice is a dynamicalled sized Array.

  • A slice does not store any data. Slices are like references to an Array. Modifying an element in the slice results in modifying the underlying array.
  • A slice literal creates an array and builds a slice that references it.
func main() {
   vowels := [6]string{"a","e","i","o","u"}
   var sub []string = vowels[:3]
   fmt.Println(sub)
   
   t := []int{1,2,3,4}
	 fmt.Println(t)
}
  • When slicing, 0 is taken as the default low bound and the length of the slice as the default high bound (similar to Python Arrays). Ex. For the array: t [10]int these expressions are equivalent: t[:], t[:10], t[0:10], t[0:]
  • Slice Length: Number of elements it contains. Can be obtained using the function len(t)
  • Slice capacity: Maximum number of elements the slice can contain. Can be obtained using the function cap(t)
  • Nil Slice: The slice’s len() and cap() is zero. The ‘zero’ value of a slice is ‘Nil’
func main() {
  var s[]int 
  fmt.Println(s, len(s), cap(s))
    if s == nil {
      fmt.Println("nil!")
    }
}

Make – for creating slices

Dynamically sized arrays can be created using the ‘make‘ function. The length and capacity of the slice can be passed as arguments to the ‘make’ function.

package main

import "fmt"

func main() {
  a := make([]int,4,6)
  printSlice("a",a)
	
  b := make([]int,2,6)
  printSlice("b",b)
	
  c:= b[:4]
  printSlice("c",c)
	
  d:= b[1:]
  printSlice("d",d)	
}

func printSlice(s string, x []int) {
  fmt.Printf("%s len=%d cap=%d %v\n",
		s, len(x), cap(x), x)
}

Slices of slices:

Slices can contain other slices too.

package main

import "fmt"

func main() {
	sliceEx := [][]string{
	[]string{"Superman","Batman","Wonder Woman"},
	[]string{"Hulk","Ironman","Spiderman","Captain America"},
	[]string{""},
	}
	fmt.Println(sliceEx)
}

Appending to a slice:

New elements can be appended to a slice using the ‘append‘ function. More than one element can be appended in a single function call. The first parameter is the slice followed by the elements.

func main() {
 var t []int
 
 t = append(t,0)
 t = append(t,1,2,3,4)
}

 

Previous post: Go – Flow Control

Next Post: GO – Types (part 2)