Posted in Docker, Misc, Tech

Docker Compose – Containerizing a Django Application

In the previous blog: Docker – Deep Dive, we created a Dockerfile with a set of instructions, started a service and create a container. Actual Projects are usually much more complex and require more than one services at a time running on multiple containers. Docker Compose handles exactly this.

Docker Compose is a tool to configure and run multiple containers that are inter dependent. It uses a YAML file to configure all the services and you can start/stop all of them with these commands:

Start services: docker-compose -f <filename.yaml> up

Stop services: docker-compose -f <filename.yaml> down

Django Application on Docker

As an example I am trying to run on docker, my Twitter Clone App that is built with Django and uses MySQL as the database: https://github.com/shilpavijay/TwitterClone

We will need a Dockerfile and docker-compose.yml file in the project directory.

Dockerfile:

FROM python:3.7-alpine
RUN apk update && apk add bash
RUN apk add gcc libc-dev linux-headers mariadb-dev
RUN pip install django djangorestframework django-rest-swagger PyJWT==1.7.1 gunicorn mysqlclient
COPY TwitterClone /app/TwitterClone
COPY TUsers /app/TUsers
COPY Tweets /app/Tweets
COPY manage.py /app
WORKDIR /app
EXPOSE 8000
CMD ["gunicorn","TwitterClone.wsgi","--bind=0.0.0.0:8000"]
  • In the above file, we are using a pre-built light weight image called alpine for python.
  • It then install all the required packages. Not specifying the version of the package will install the latest available version.
  • Next, all the project files and directories are copied to /app folder on the created container.
  • The current directory is changed to /app.
  • We also expose the port on which the application will run.
  • The last command will start the application.

docker-compose.yaml

version: '3.4'

services:
  db:
    image: mysql:5.7
    ports:
      - '3306:3306'
    container_name: twitter_db
    command: --default-authentication-plugin=mysql_native_password
    restart: always
    environment:
      MYSQL_DATABASE: 'twitterclone'
      MYSQL_USER: 'root'
      MYSQL_PASSWORD: 'guessme'
      MYSQL_ROOT_PASSWORD: 'guessme'

  twitterapp:
    container_name: twitterclone
    build:
      context: ./
      dockerfile: Dockerfile
    restart: always
    ports:
      - '8080:8000'
    volumes:
      - .:/code
    depends_on:
      - db
  • We have two services, one is the Django App (twitterapp) and the other is the MySQL DB (db). Ensure the names are always in lowercase.
  • The app will be built with the instructions in the Dockerfile.
  • MySQL container will be based on the mysql image available on docker hub. (refer previous blogpost)
  • The “depends_on” clause tells that the application has a dependency on the database.
  • MySQL service takes a while to start. There is usually a lag between the app start and database start time. To ensure a services does not fail if the other has not started yet, “restart: always” is configured.
  • Port forwarding is configured in the “ports:” clause. The host port 8080 is mapped to the container post 8000 on which the application is running. Hence, to access the application from the host system, we will be using port 8080.
  • We will discuss “volumes” in the subsequent section

Next, update settings.py in the Django Project with the host, port, login details to match that given in the Docker Compose file:

DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            'NAME': 'twitterclone',
            'USER': 'root',
            'PASSWORD': 'guessme',
            'HOST': 'db',
            'PORT': '3306',
            },
        }

When running the application for the first time OR if there are any changes to Django migrations, we need to run this explicitly in order to apply the migrations.

docker exec -it  python manage.py migrate

Then, execute the following commands to build and run the docker container:

docker-compose up --build

We are Done!!! The application should now be running on localhost:8080.

The complete code is here: https://github.com/shilpavijay/TwitterClone

Docker Volumes

Docker volumes are used for data persistence. When a container is stopped and re-run, the database starts all over again and the data stored previously is lost. To make it available even after the container is re-run, Docker volumes are made use of.

Here’s how it works. A directory from the host file system is mapped to a directory of the container file system. So, each time a container is started, the data is replicated into the host directory and when the container is restarted, it gets the data automatically from the directory in the host system.

Docker volumes can be created using the ‘docker run’ command. It can also be configured in the Docker Compose file.

There are 3 different types of docker volumes:

  • Host volume: User specifies the host directory where the data is to be replicated.
docker run -v /home/user/dockerdata:/var/lib/mysql/data
  • Anonymous volume: User does not specify the host directory. It is automatically taken care of, by docker. Only the directory on the container is specified.
docker run -v /var/lib/mysql/data
  • Named volume: User specifies a name but not the path. Path is handled by docker. You can reference the volume by the given name This type is most frequently used.
docker run -v name:/var/lib/mysql/data

Conclusion

In this blogpost series, we have leant the basics required to understand and use Docker. We have seen the Evolution of Docker, the need for docker and also how to setup and run an application on docker. We leant about Docker Compose and Docker Volumes too, in this post. Here are the links to previous posts:

Docker – Introduction

Docker – Deep Dive

Github link to the project discussed in this blog: https://github.com/shilpavijay/TwitterClone

You can also refer the official documentation page for more information and updates: https://docs.docker.com/get-started/

Posted in Docker, Misc, Tech

Docker – Deep Dive

Docker – Introduction covered the evolution, the need for Docker and the reason why it is widely used. This post will be a deep dive into the concepts of Docker and it’s practical use cases.

This post covers everything required to create an image and run an application using docker. The OS I have chosen Windows for all the examples, mainly because it is quite challenging. Working with docker on Windows is not as seamless as it is on Linux/Unix OS. Hence I have included a lot of workaround hoping it will be of help those using Windows.

Installation:

https://docs.docker.com/docker-for-windows/install/

You can install from the above link by following a few simple instructions. Once done, add the following to the “PATH” variable. You can verify if it works, with “docker –version” command.

C:\Program Files\Docker\Docker\resources\bin C:\ProgramData\DockerDesktop\version-bin

Let us begin

Now, we will create a simple python program that prints “hello world” and name it main.py.

print('Hello World!!!')

Dockerfile:

Dockerfile is a text document that contains all the instructions required to build an image. It should be named exactly as “Dockerfile” and placed inside the application folder. Most commonly used commands in a dockerfile are:

FROM python
ENV <> 
RUN <>
COPY <>
CMD <>
WORKDIR <>
  • Dockerfile always starts with a FROM command that specifies the pre-existing image (present on dockerhub) to import from.
  • ENV is for setting environment variables in the container. This is optional and usually moved to the Docker compose file
  • RUN to execute linux commands
  • COPY – executes on the host. Can copy files from the host to the container.
  • CMD – is the entry point command for the application execution
  • WORKDIR – points to the directory in the container, where the application code exists.

Create a “Dockerfile” with these three lines:

FROM python:3.7-alpine
COPY helloWorld.py run.py
CMD ["python","run.py"]
  • The first line will set up the environment by installing python. 3.7-alpine which refers to the already available Image on docker hub. You can find more such Images on hub.docker.com. This step takes time when the container is built for the first time as the installations happen. Subsequently, it should take lesser time.
  • The next lines copies the contents of the local file helloWorld.py onto the file on the newly created Image. Here the file name is run.py, but the name can be anything.
  • The third line specifies the Command that is to be run on the Container to execute the program in the file, just copied.

Now, run the following commands to first build the image and then create a container. Replace <sample-image> with a name of your own. You should now see “Hello World!!!” printed on the console.

docker build -t <sample-image> .
docker run <sample-image>

Working with Environment variables

We previously saw how to execute a simple print statement on a docker container. Let’s now see a little more complex example where we pass an environment variable from local as an argument while building the container. This will set the environment variable in the container image. We will verify the same by printing the environment variable in the container.

main.py

import os
def main():
    print(os.environ.get('testenv'))
main()

Dockerfile

FROM python:3.7-alpine
COPY main.py run.py
ARG x
ENV testenv=$x
CMD ["python","run.py"]

Let us set an environment variable in our local machine. Command for windows:

D:\Projects\DockerSample>set whatdoing="docker"
D:\Projects\DockerSample>echo %whatdoing%
 "docker"

Here’s how we run the build command by passing an argument:

docker build --build-arg x=%whatdoing% -t sample-image .
docker run sample-image

The above should print “docker”. %whatdoing% is an env variable in the local machine. In the Dockerfile, we receive an argument “ARG x” and set the same as the value for the environment variable “testenv”. Our python program “main.py” is fetching the environment variable “testenv” and prints the value of the same.

Hence, we can not only install various packages onto the docker container but also set all the required environment variables before-hand, in order to run an application on the newly created container.

Frequently used

Here are some frequently used commands that can come handy while working with docker images and containers:

- Get an image from Docker hub: (Example - redis)
docker pull <name> 

- List all the images:
docker images

- List only image IDs:
docker images -q 
(q stands for quiet. Suppresses other columns)

- Create a container of a given image:
docker run <image>

- Run a container in detached mode:
docker run -d <image>

- Remove a docker image by ID
docker rmi <Image ID>

- List the docker containers in running state: (PS - Process State)
docker ps

- List all containers irrespective of their state:
docker ps -a

- Restart a container:
docker stop <container-id>
docker start <container-id> 

- Bind a port of your host to a port of the container while creating a container:
docker run -p <host-port>:<container-port> <image>

- View container logs: (you can also stream the logs using -f option)
docker logs <container-id/name> -f

- Specify a name for the container:
docker run --name inamedthis <image>

- Get the terminal of the running container:
docker exec -it <container-id/name> /bin/bash

- List the current networks:
docker network ls

- Create a new docker network:
docker network create <name>

Cleaning container images:

Here’s an important workaround while working with Windows. Time and again, I ran into issues with the container not reflecting the current state. This was due to old containers not being stopped and removed correctly before creating new ones when they had the same name. Here’s a batch file that clears old containers:

@ECHO OFF 
FOR /f "tokens=*" %%i IN ('docker ps -aq') DO docker stop %%i
FOR /f "tokens=*" %%i IN ('docker ps -aq') DO docker rm %%i 

You can put the above in a file and add to the path variable so that you can use it whenever you are creating/debugging containers multiple times.

In the next post we will see how we can run multiple services on docker using Docker Compose and also host an application on docker.

Posted in Python, Tech

Django Custom Middleware

A Custom middleware can validate each http request for the set of instructions given in the middleware, before sending back a response. Let us see how we can write a custom Middleware. As an example, we will be implementing a custom Authentication Middleware here.

Django’s out-of-the box Authentication middleware only authenticates requests if the project has made use of the Django provided “User” model or has extended the same. If we have otherwise implemented a model from scratch for the user data, we will have to write a custom middleware for authentication and authorization.

I am implementing this in my twitter clone project, where I have used a custom model for user data. The details of the project can be found in my previous blogpost: Building a Twitter Clone with Django REST Framework – I

First, we will create a file in TUser App. We will call it AuthMiddleware.py. To start off, there are a few basic rules to be followed while defining a Middleware are:

  1. The custom middleware must accept a get_response argument while defining the __init__ method. This is called only once when the web server starts.
  2. Define the __call__() method. This is called each time a request is made.
  3. Include the path of the above file in settings.py – in the “Middleware” section

This is how a sample middleware is defined:

#AuthMiddleware.py
from django.contrib.auth.backends import BaseBackend

class AuthMiddleware(BaseBackend):
    def __init__(self, get_response):
        self.get_response = get_response
    def __call__(self, request):
        return self.get_response(request)
    def CustomMiddleware(self,requst):
        #Steps to perform
        return True

#settings.py
MIDDLEWARE = [
    ...
    #custom middlewares:
    'TUsers.AuthMiddleware.AuthMiddleware'
]

Here’s my version of Custom Authentication Middleware – AuthMiddleware.py

class AuthMiddleware(BaseBackend):
    def __init__(self, get_response):
        self.get_response = get_response
    def __call__(self, request):
        self.auth(request)
        return self.get_response(request)
    def auth(self,request):
        exclusion_list = ['/signup','/login']
        if request.path not in exclusion_list:
            try:
                urlstr = request.path
                user = urlstr.split('/')[1]
                token = request.session.get('authtoken').get('token')
                payload = jwt.decode(token,settings.AUTH_TOKEN)
                userObj = TUser.objects.get(username=user)
                if payload.get('username') == userObj.username:
                    return True
                else:
                    raise PermissionDenied
            except (jwt.ExpiredSignature, jwt.DecodeError, jwt.InvalidTokenError) as e:
                error = {'Error_code': status.HTTP_403_FORBIDDEN,
                                'Error_Message': "Token is Invalid/Expired"}
                logger.error(e)
                raise PermissionDenied(error)
            except Exception as e:
                error = {'Error_code': status.HTTP_403_FORBIDDEN,
                                'Error_Message': "Invalid User"}
                logger.error(e) 
                raise PermissionDenied(error) 
        else:
            return True

This middleware will automatically validates each and every request made. In this project, the validation is performed to check if the token that a user obtains while Login is valid. If yes, the API end-point performs its action and returns the intended response. If not, an exception is raised and the access to the API is forbidden.

The complete file is available on github:

https://github.com/shilpavijay/TwitterClone/blob/main/TUsers/AuthMiddleware.py

You can find the complete project code here:

https://github.com/shilpavijay/TwitterClone

Posted in Python, Tech

Building a Twitter Clone with Django REST Framework – III

Previous posts:

Building a Twitter Clone with Django REST Framework – I

Building a Twitter Clone with Django REST Framework – II

In this post, let’s put some of the APIs to use:

  • Create a new user using the ‘signup’ API end point
  • Login to the application
  • Login with wrong credentials
  • Get all the followers
  • Creating a tweet:
  • Get the user’s Paginated timeline
  • Like a tweet:
  • Search for a tweet:
Posted in Python, Tech

Building a Twitter Clone with Django REST Framework – II

Previous post: Building a Twitter Clone with Django REST Framework – I

User Authentication

We will perform Authentication using JWT or JSON Web Token. JWT is a secret token that is generated and sent to the browser when a user successfully logs in with his credentials. Subsequently, whenever the user accesses any link, the token is sent back to the server in the request header. The server checks for the validity of the token before granting access.

User ————-> Login Successful ——————————–> Server: Generates JWT

User <————- Returns JWT to browser ——————– Server

User ————-> Sends JWT in request header ————> Server validates JWT signature

User <————- Access Granted ——————————– Server sends Response

We will be using Python’s JWT Module – PyJWT here. I did try out Django Rest Framework’s JWT, but I found it hard to get it working with little/unclear documentation. Also, the process of token generation and encryption is better understandable using PyJWT. It is also very flexible for customization. To install:

pip install PyJWT

settings.py

#In settings.py
import uuid

#RANDOM AUTHENTICATION TOKEN GENERATOR
AUTH_TOKEN = uuid.uuid4().hex.upper()

Let’s write the views for Authentication:

import jwt
 
#set Expiration time for the token. Best practise is to set it less than 10 minutes
EXP_TIME = datetime.timedelta(minutes=5)

def GetToken(username):
    '''
    Purpose: Get Access token
    Input: 
    username (mandatory) <str> Account user 
    password (mandatory) <str> Password
    Output: Token that expires in 60 minutes
    '''
    try:
        user = TUser.objects.get(username=username)
        if user:
            try:
                payload = {'id':user.id,'username':user.username,'exp':datetime.datetime.utcnow()+EXP_TIME}
                token = {'token':jwt.encode(payload,settings.AUTH_TOKEN).decode('utf8')}
                                # jwt.encode({'exp': datetime.utcnow()}, 'secret')
                return token
            except Exception as e:
                error = {'Error_code': status.HTTP_400_BAD_REQUEST,
                        'Error_Message': "Error generating Auth Token"}
                logger.error(e)
                return Response(error, status=status.HTTP_403_FORBIDDEN)
        else:
            error = {'Error_code': status.HTTP_400_BAD_REQUEST,
                        'Error_Message': "Invalid Username or Password"}
            return Response(error, status=status.HTTP_403_FORBIDDEN)
    except Exception as e:
        error = {'Error_code': status.HTTP_400_BAD_REQUEST,
                        'Error_Message': "Internal Server Error"}
        logger.error(e) 
        return Response(error,status=status.HTTP_400_BAD_REQUEST) 

def Auth(request,username):
    '''
    Purpose: Login to the Application
    Input: 
    token (mandatory) <str> user token 
    Output: User object of the logged in user
    '''
    try:
        token = request.session.get('authtoken').get('token')
        payload = jwt.decode(token,settings.AUTH_TOKEN)
        user = TUser.objects.get(username=username)
        if payload.get('username') == user.username:
            serializer = TUserSerializer(user)
            return Response(serializer.data, status=status.HTTP_200_OK)
        else:
            error = {'Error_code': status.HTTP_403_FORBIDDEN,
                        'Error_Message': "Invalid User"}
            logger.error(error)
            return Response(error,status=status.HTTP_403_FORBIDDEN) 
    except (jwt.ExpiredSignature, jwt.DecodeError, jwt.InvalidTokenError) as e:
        error = {'Error_code': status.HTTP_403_FORBIDDEN,
                        'Error_Message': "Token is Invalid/Expired"}
        logger.error(e)
        return Response(error,status=status.HTTP_403_FORBIDDEN) 
    except Exception as e:
        error = {'Error_code': status.HTTP_403_FORBIDDEN,
                        'Error_Message': "Internal Server Error"}
        logger.error(e) 
        return Response(error,status=status.HTTP_403_FORBIDDEN) 

@api_view(['POST'])
def Login(request,username=None,password=None):
    '''
    Authenticate if username and password is correct. 
    Input
    Output: return User object or Error 
    '''
    username = request.query_params.get('username')
    password = request.query_params.get('password')
    try:
        user = TUser.objects.get(username=username)
        if user.password == password:
            token = GetToken(username)
            user.token = token['token']
            user.save()
            request.session['authtoken'] = token
            serializer = TUserSerializer(user)
            return Response(serializer.data, status=status.HTTP_200_OK)
        else:
            error = {'Error_code': status.HTTP_400_BAD_REQUEST,
                        'Error_Message': "Invalid Username or Password"}
            return Response(error,status=status.HTTP_400_BAD_REQUEST) 
    except Exception as e:
        error = {'Error_code': status.HTTP_400_BAD_REQUEST,
                        'Error_Message': "Invalid Username"}
        logger.error(e) 
        return Response(error,status=status.HTTP_400_BAD_REQUEST)

App Users – REST APIs

urls.py: https://github.com/shilpavijay/TwitterClone/blob/main/TUsers/urls.py

from django.urls import path
from . import views

urlpatterns = [
    path('signup', views.AccountSignup),
    path('login', views.Login),
    path('<str:username>/account', views.AccountUpdate),
    path('token', views.GetToken),
    path('auth', views.Auth),
    path('<str:loggedin_user>/<str:user>/follow', views.FollowUser),
    path('<str:username>/followers', views.GetFollowers),
    path('<str:username>/following', views.GetFollowing),
    path('<str:username>/block', views.Block_user), 
    path('users',views.users), #debugging
]

serializer.py : https://github.com/shilpavijay/TwitterClone/blob/main/TUsers/serializer.py

from rest_framework import serializers
from TUsers.models import TUser

class TUserSerializer(serializers.ModelSerializer):
    class Meta:
        model = TUser
        fields = '__all__'

views.py : This file content is too large to be placed here and is far more readable on Github. Link: https://github.com/shilpavijay/TwitterClone/blob/main/TUsers/views.py

App Tweets – REST APIs

urls.py: https://github.com/shilpavijay/TwitterClone/blob/main/Tweets/urls.py

from django.urls import path
from . import views

urlpatterns = [
    path('create', views.CreateTweet),
    path('<str:username>/', views.Timeline), 
    path('<int:tweet_id>/delete', views.DeleteTweet),
    path('<int:tweet_id>', views.ShowTweet), 
    path('<int:tweet_id>/reply', views.Reply),
    path('<int:tweet_id>/retweet', views.Retweet),
    path('<int:tweet_id>/like', views.Like),
    path('search', views.Search),
    path('all_tweets', views.all_tweets), #debugging
]

serializer.py: https://github.com/shilpavijay/TwitterClone/blob/main/Tweets/serializer.py

from rest_framework import serializers
from Tweets.models import TCtweets
from django import forms

class TCtweetsSerializer(serializers.ModelSerializer):
    class Meta:
        model = TCtweets
        fields = '__all__'

class TCValidator(forms.Form) :
    username = forms.CharField()
    tweet_text = forms.CharField()

class ReplyValidator(forms.Form):
    username = forms.CharField()
    reply_text = forms.CharField()   

class RetweetValidator(forms.Form):
    username = forms.CharField() 

views.py: https://github.com/shilpavijay/TwitterClone/blob/main/Tweets/views.py

The complete code for this project can be found in my GitHub repository: https://github.com/shilpavijay/TwitterClone

In the next post, we will use some of these API end-points and see how they work.

Next post: Building a Twitter Clone with Django REST Framework – III

Posted in Python, Tech

Building a Twitter Clone with Django REST Framework – I

In this blog-series, I will be sharing my experience building an App with features similar to Twitter, using Django REST.

  • There will be a code snippet for each step. You can find the entire code on GitHub too: https://github.com/shilpavijay/TwitterClone
  • The approach I have taken is TDD – Test Driven Development. There will be a test case for each functionality, followed by the development of that feature.

Setting up the Django Environment

conda create env --name django_env
conda activate django_env

For Django REST Framework:

pip install djangorestframework

# Add to settings.py:
INSTALLED_APPS = [
     ...
    'rest_framework',
]

Let’s begin the Django Project

#To start a Django Project:
django-admin startproject TwitterClone

#Start the Apps:
python manage.py startapp TUsers
python manage.py startapp Tweets

#Add the two Apps to settings.py
INSTALLED_APPS = [
     ...
    'Tweets',
    'TUsers',
]

Writing Test Cases:

  • Create a tests folder inside the App as shown below.
  • This has to be made into a package by creating a ‘__init__.py’.
  • All the files in the package should begin with a test_<>
  • We will be writing test cases that initially fail and then coding the functionality, with the final goal of passing the test cases.

test_users.py and test_tweets.py

from django.test import TestCase
from TUsers.models import TUser

class TUserTestCase(TestCase):
    def setUp(self):      
        TUser.objects.create(username="mark_cuban",
                             password="sharktank",
                             country="USA")
    def test_create(self):
        self.assertEqual("USA",
                      TUser.objects.get(username="mark_cuban").country)
from django.test import TestCase
from Tweets.models import TCtweets
from TUsers.models import TUser

class TweetsTestCase(TestCase):
    def setUp(self):
        user1 = TUser.objects.create(username="lori",password="sharktank",country="USA")
        TCtweets.objects.create(username=user1,tweet_text="Hi there!")
    def test_create_tweet(self):
        user1 = TUser.objects.get(username='lori')
        self.assertEqual("Hi there!",TCtweets.objects.get(username=user1).tweet_text)

Adding the models:

TUser/models.py and Tweets/models.py

from django.db import models
from datetime import datetime

get_cur_time = datetime.now().strftime('%m/%d/%Y %I:%M:%S %p')

class TUser(models.Model):
    username = models.CharField(max_length=100,unique=True)
    password = models.CharField(max_length=100)
    country = models.CharField(max_length=100,null=True)
    modified_time = models.CharField(max_length=50,default=get_cur_time)
    following = models.ManyToManyField('self',related_name='followers',symmetrical=False,blank=True)
    blocked = models.BooleanField(default=False)
    token = models.CharField(max_length=10000,null=True)
from django.db import models
from TUsers.models import TUser
from datetime import datetime

get_cur_time = datetime.now().strftime('%m/%d/%Y %I:%M:%S %p')

class TCtweets(models.Model):
    username = models.ForeignKey(TUser,on_delete=models.CASCADE)
    tweet_text = models.CharField(max_length=100,null=True)
    time = models.CharField(max_length=50,default=get_cur_time)
    retweet = models.ManyToManyField(TUser,related_name="retweeted_users")
    like = models.ManyToManyField(TUser,related_name="liked_users")
    reply = models.ManyToManyField("self")     
    comment = models.CharField(max_length=100,null=True)

Swagger

Let us set up Swagger for API documentation:

pip install django-rest-swagger

#Add to settings.py
INSTALLED_APPS = [
     ...
    'rest_framework_swagger',
]

#Add the following to urls.py
from rest_framework_swagger.views import get_swagger_view

schema_view = get_swagger_view(title='Twitter Clone API')

urlpatterns = [
    path('admin/', admin.site.urls),
    path('docs/',schema_view),
]

If you are using Django 3.x, (which is my case here) there are a few things to be done in order to make Swagger work:

  • Enable CoreAPI Schema: Django Rest Framework 3.x deprecated the CoreAPI based schema generation and introduced the OpenAPI schema generation in its place. Currently to continue to use django-rest-swagger we need to re-enable the CoreAPI schema. To settings.py add:
REST_FRAMEWORK = 
{'DEFAULT_SCHEMA_CLASS': 'rest_framework.schemas.coreapi.AutoSchema' }
  • Change the swagger index.file: {% load staticfiles %} has been deprecated in django 2.x and removed in 3.x. Modify the same in index.html file of swagger in the location: …/Anaconda\envs\dj\Lib\site-packages\rest_framework_swagger\templates\rest_framework_swagger
...
{% load static %}  #Remove {% load staticfiles %}
<!DOCTYPE html>
<html lang="en">
<head>
...
...

And finally, you are done. You should see the Swagger API page.

Test Case for APIs

Too test the login API, first let’s write the test case and then build the code for the same. Here’s a sample:

test_users.py

   def test_api_login(self):
        client = Client()
        response = client.post('/login/', 
                   {'username':'mark_cuban'},format='json')
                    self.assertEqual(200,response.status_code)

In the next blogpost, we will see how to build the REST APIs.

You can find the project code on GitHub: https://github.com/shilpavijay/TwitterClone

Posted in Python, Tech

Python 3.7 – Cool New features

Welcome back, to Python! It’s time to discuss the cool new features shipped with Python 3.7. Here are some of my favorites:

cobra-1287036_1920

  • Data Classes
  • Built-in breakpoint
  • Typing module
  • Importing Data files

Data Classes

What is it? A new decorator: @dataclass

What’s new? It eases writing special methods in the class, like __init__(), __repr__(), and __eq__(. They are added automatically.

Example:

from dataclasses import dataclass, field
@dataclass(order=True)
class Test:
…field1: str
…field2: str

The following does not need an implementation of __repr__() in the class:

>>> t = Test("abc","xyz")
>>> t
Test(field1='abc', field2='xyz')
>>> t.field1
'abc'
>>> t.field2
'xyz'

Doesn’t need an implementation of __eq__ to do this:

>>> t1=Test("abc","xyz")
>>> t==t1
True
>>> t2=Test("a","b")
>>> t==t2
False

Breakpoint

What is it?  A built-in pdb.

What’s new? Not a new feature but simplifies using the debugger. Eliminates the necessity to import pdb.

Example:
def divide(a,b):
…breakpoint()
…return a/b

>>> divide(2,3)
> (3)divide()
(Pdb)

Old way of importing pdb:

def divide(a,b):
….import pdb; pdb.set_trace()
….return a/b

Typing module13541540425_63372041e1

What is it?  Annotations and Type hinting

What’s new? Function Annotations intend to provide a standard way of associating metadata to function arguments and return values.

  • The annotations module and typing module provide hints on arguments and return value of a function.
  • Annotations earlier worked only with names available in the current scope. i.e. Forward referencing was not supported.
  • Annotations are now evaluated when a module is imported.

Example:  

Creating the file py37anno.py:

from __future__ import annotations

class Try:
…def foo(name: str) -> ‘salutation’:
……print(f”Annotations Example for you {name}”)

importing the annotations and typing module:

>>> from py37anno import Try
>>> from __future__ import annotations
>>> Try.foo.__annotations__
{‘name’: ‘str’, ‘return’: “‘salutation'”}
>>> import typing
>>> typing.get_type_hints(Try.foo)
{‘name’: <class ‘str’>, ‘return’: ForwardRef(‘salutation’)}
>>> Try.foo(‘Shilpa’)
Annotations Example for you Shilpa

Importing Data Files

What is it? An optimized, easy and organized API for working with data files in a Project

What’s new? Eliminates the necessity of hard-coding data file-names. The importlib.resources module helps in locating, importing and reading from the data file.

Example:

A file lorem.txt exists in the data directory of the project which also contains the __init__.py file

>>> import os
>>> os.listdir(‘data’)                                     #output: files in the ‘data’ directory
[‘lorem.txt’, ‘__init__.py’, ‘__pycache__’]

>> from importlib import resources
>>> with resources.open_text(“data”,”lorem.txt”) as f:
….print(f.readlines())
….
[‘”Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.” n’]

A note on installing python 3.7:

Creating a new environment:

conda create -n py37 -c anaconda python=3.7

Upgrade in an existing python environment to 3.7 in Anaconda:

conda install -c anaconda python=3.7

Posted in Docker, Misc, Tech

Docker – Introduction

docker.png

Let’s start by looking back on how applications were hosted and how it has evolved. Initially, we had one application running on a single server with its own dedicated memory, CPU and disk space. This model proved to be highly costly in order to host high load applications.

hypervisor.pngVirtual Machine

Next, we moved to Virtual Machines (VM), also called as Hypervisor Virtualization. We could now run multiple applications on each virtual machine, each one being a chunk of the actual physical server and having its own memory, CPU and disk space. However, the drawback was that each Virtual Machine needed its own Operating System and this incurred additional overhead of licensing costs, security patching, driver support, admin time etc.

Docker arrived with a solution to this problem. There is just one OS installed on the server. Containers are created on the OS. Each Container is capable of running separate applications or services. Here’s a comparative illustration of Containers vs Virtual Machines:

docker vs vm

  • Docker is said to be derived from the words: docker + worker -> Docker.
  • It is written using Golang
  • Is Open source with Apache license 2.0.

Let’s look at what the following mean:

Docker Container: It is a software that contains the code and all its dependencies to facilitate an application to run quickly and reliably irrespective of the computing environment

Docker Container Image is a lightweight, standalone, executable package that contains everything to run an application: the code, runtime, system tools, libraries and settings. Container Images become Containers at runtime.

Docker Hub is a public docker repository to store and retrieve docker Images. This is provided by Docker, however there are other third-party registries as well.

Open Container Initiative or OCI  is responsible for standardizing container format and container runtime.

Why is Docker so popular? 

  • A Docker image can run the same way irrespective of the server or the machine. It is hence very portable and consistent. It eliminates the overhead of setting up and debugging environment.
  • Another reason is Rapid Deployment, the deployment time is reduced to a few seconds.
  • They start and stop much faster than Virtual Machines
  • Applications are easy to scale as containers can be easily added from an environment.
  • Helps in converting a monolithic application into Microservices. [To read more, refer Microservices]
  • last but not the least, it is Open Source.
Posted in Scala

Scala – Introduction

Origin

Scala was created by Martin Odersky in 2004. The name ‘Scala’ is derived from the word ‘Scalable’ which signifies that it can grow with the demand of the user. Scala was designed to be both an Object oriented as well as a Functional programming language. i.e. every value is an object and every function is a value. It is designed to exhibit common programming patterns in a concise, type-safe and elegant way. Many say that it was introduced to fulfill the shortcomings of Java.

1_ygv4NHtC9Jao-aZPSCMPYg

Why Scala?

  • Modular, highly efficient and scalable.
  • Is a type-safe JVM language that incorporates both object-oriented and functional programming.
  • Good for exploiting parallelism for multicore and parallel computing.
  • The source code compiles to Java bytecode that runs on the JVM.
  • It is very flexible in defining abstractions
  • Immutability: Scala makes it easy to write code using immutable data
  • Scala can use all Java libraries and Scala code can be imported into Java code interchangeably.
  •  Applications:
    • Writing web applications
    • Distributed applications
    • Streaming Data
    • Parallel processing
    • Analyzing data with Spark

Who’s using Scala?

businesses-using-scala.jpg

Directory structure

SBT or the Source Build Tool is an open-source build tool for Scala/Java Projects. Here’s the directory structure normally followed in Scala projects. The src/ is the Base or the project’s root directory.

src/
   main/
       resources/
         <files to include in main jar here>
       scala/
         <main Scala sources>
       java/
         <main Java sources>
   test/
       resources
         <files to include in test jar here>
       scala/
         <test Scala sources>
       java/
         <test Java sources>

Class, Object, Package and Trait

Class: The concept of classes in Scala is similar to that in Java. They are nothing but templates containing fields and methods. A class can be instantiated using the ‘new’ construct. Classes in Scala cannot have static members.

Object: Object is a named instance with fields and methods.

Package: Packages allows you to modularize programs. Contents of the package can be scattered across many files. Classes defined in a package can be specifically imported.

Trait: Traits are like Interfaces. However, they can even contain field definition or method implementations.

Writing your first Scala Program:

object FirstScalaProg extends App {
    println("I am done! Execute me!")
}

The entry point of the program is defined inside an object. The object is made executable by extending the type ‘App‘ or by adding a “main” method, as follows:

object FirstScalaProg {
    def main(args: Array[String]){
        println("I can be written this way too!")
    }
}

Scala source code is stored in text files with the extension .scala. The Scala compiler compiles .scala files into .class files. Classfiles are binary files containing machine code for the JVM to execute.

Books:

  • Structure and Interpretation of Computer ProgramsHarold Abelson and Gerald J. Sussman. 2nd edition. MIT Press 1996.– [Full text available online].
  • Programming in ScalaMartin Odersky, Lex Spoon and Bill Venners. 2nd edition. Artima 2010. – [Full text of 1st edition available online]
  • Scala for the ImpatientCay Horstmann. Addison-Wesley 2012. – [First part available for download.]
  • Scala in DepthJoshua D. Suereth. Manning 2012.
  • Programming ScalaDean Wampler and Alex Payne. O’Reilly 2009.

Other Learning Sources:

Posted in Java for Python Developers, Tech

Java For Python Developers 2 – OOP

This blog will focus on Object Oriented Programming concepts like Inheritance, Polymorphism and writing Packages, Interfaces and Abstract classes in Java.

Packages

Packages make a project organized. It creates a level of security by letting only the classes within that package have access, and it also provides name-scoping.

  • Some of them we have already come across are System (System.out.println), String, Math. They are from the package java.lang that is automatically included without having to import.
  • An import statement is written at the beginning of the code and is accessible throughout. An alternate way is to type the full name on each use in the code., i.e <pkg_name>.<class_name>
  • Java doesn’t compile the imported class. An import statement only saves time typing the full name of every class.

Inheritance

Public, Private, Protected and Default are called access controls. They control whether a variables or a methods can be inheritance or not.

  • Public members are inherited.
  • Private members are not inherited
  • Protected members can be access only by subclasses
  • Default can be accessed only within the same class. It doesn’t require a keyword.

Java does not support multiple inheritance, as it leads to the “Deadly Diamond of Death” problem. Lets see what that means.

180px-Diamond_inheritance.svg

The diamond problem is an ambiguity that arises when two classes B and C inherit from A and then when D inherits from B and C. Now, if there is a method in A that has been overridden by both B and C then which version of the method does D inherit? that of B or that of C? To avoid this problem, Java does not allow multiple inheritance.

Few other important aspects of inheritance:

  • A non-public class can be sub-classed only by the classes in the same package.
  • A class declared as final cannot be sub-classed anymore.
  • A class which has all private constructors cannot be sub-classed.
  • The keyword used is “extends”

Polymorphism

Polymorphism is having many forms of a method.

It is not a valid overloading if only the return type of a given method has been changed. To overload a method, the argument list has to be changed.

Abstract Class

  • Abstract Classes cannot be instantiated. The compiler will restrict instantiation of an abstract class. Whereas, concrete classes can be instantiated.
  • Abstract Methods must be overridden by the inherited class. Abstract methods do not have a body. It keeps a check to ensure that the inherited class implement them.
  • An abstract method should always be put in an abstract class.

Lets see how they are implemented in Java:

abstract class Pan extends Utensils { 
    public abstract void Material(); //an abstract method.
} 

Pan p; //works
p = new Spoon; //works
p = new Pan; //Compiler throws an error: Pan is abstract; cannot be instantiated

It is similar to that in Python. Lets see the implementation:

from abc import ABCMeta, abstractmethod #IMPORTANT

class animal:
     __metaclass__ = ABCMeta
    def delegate(self):
        self.action()

    @abstractmethod
    def action(self): pass

class tiger(animal): pass

A=animal()
t = tiger()

Interfaces

An interface can be presumed to be an extra feature that are not inherited from a parent class. This kind of helps overcome the drawback of not being able to inherit multiple classes.

  • An interface contains all “abstract” methods that the subclass is forced to implement. It can be perceived as a 100% abstract class
  • The keyword used is “implements”
  • Interfaces are always “public”. Methods in the interface are always “abstract”
  • A class can implement multiple interfaces which allows adding a number of features to a class making it more meaningful, in other words more real-word than the parent class.

Here’s an example:

interface Pet {      // all are the abstract methods.      abstractvoid wagTail(int a);      abstractvoid RunAround(int a);      abstractvoid EatFood(int a);  }

class Dog extends Animal implements Pet {}

Conclusion:

In this blogpost we learnt some Object Oriented Programming concepts in Java. We also learnt how to implement Packages, Abstract Classes and Interfaces. In the next blog will be a deep dive into Memory management and Garbage Collection.

The previous blog was about getting started with basic concepts like using Variables, write Classes, conditional statements, loops and built-in libraries.                                     Java for Python Developers 1 – Basics