Python

Certain non-alphanumeric characters are defined to perform a specified operation. Such characters are called operators. For example the characters +, -, * and / are defined to perform arithmetic operations on two numeric operands. Similarly <, > == and != perform comparison of two numeric operands by default.

Some of built-in classes of Python allow certain operators to be used with non-numeric objects too. For instance the + operator acts as concatenation operator with two strings. We say that + operator is overloaded. In general overloading refers to attaching additional operation to the operator.

>>> #default addition operation of +
>>> 2+5
7
>>> #+operator overloaded as concatenation operator
>>> 'Hello'+'Python'
'HelloPython'
>>> [1,2,3]+[4,5,6]
[1, 2, 3, 4, 5, 6]
>>> #default multiplication operation of *
>>> 2*5
10
>>> #overloaded * operator as repetition operation with sequences
>>> [1,2,3]*3
[1, 2, 3, 1, 2, 3, 1, 2, 3]
>>> 'Hello'*3
'HelloHelloHello'

For user defined classes, operator overloading is achieved by overriding relevant magic methods (methods with two underscores before and after name) from the object class. For example, if a class contains overridden definition of __add__() method, it is implemented when + operator is used with objects of that class. Following table shows magic methods and the arithmetic operator they implement:

Following script contains a class that overrides __add__() method. It causes + operator overloading.

>>> class MyClass:
def __init__(self, x,y):
self.x=x
self.y=y
def __add__(self, obj):
x=self.x+obj.x
y=self.y+obj.y
print ('x:{} y:{}'.format(x,y))

We now have two objects of above class and use + operator with them. The __add__() method will implement overloaded behaviour of + operator as below:

>>> m1=MyClass(5,7)
>>> m2=MyClass(3,8)
>>> m1+m2
x:8 y:15

Similarly comparison operators can also be overloaded by overriding following magic methods:

Complex number z=x+yj is a Cartesian (also called rectangular) representation. It is internally represented in polar coordinates with its modulus r (as returned by built-in abs() function) and the phase angle θ (pronounced as theta) which is counter clockwise angle in radians, between the x axis and line joining x with the origin. Following diagram illustrates polar representation of complex number:

Functions in cmath module allow conversion of Cartesian representation to polar representation and vice versa.

polar() : This function returns polar representation of a Cartesian notation of complex number. The return value is a tuple consisting of modulus and phase.

>>> import cmath
>>> a=2+4j
>>> cmath.polar(a)
(4.47213595499958, 1.1071487177940904)

Note that the modulus is returned by abs() function

>>> abs(a)
4.47213595499958

phase(): This function returns counter clockwise angle between x axis and segment joining a with origin. The angle is represented in radians and is between π and -π

>>> cmath.phase(a)
1.1071487177940904

rect(): This function returns Cartesian representation of complex number represented in polar form i.e. in modulus and phase

>>> cmath.rect(4.47213595499958, 1.1071487177940904)
(2.0000000000000004+4j)

Function is said to be recursive if it calls itself. Recursion is used when a process is defined in terms of itself. A body of recursive function which is executed repeatedly is a type of iteration.

The difference between a recursive function and the one having a loop such as ‘while’ or ‘for’ loop is that of memory and processor overhead. Let us try to understand with the help of iterative and recursive functions that calculate factorial value of a number.

Iterative factorial function

>>> def factorial(x):
f=1
for i in range(1, x+1):
f=f*i
return f
>>> factorial(5)
120

Recursive factorial function

>>> def factorial(n):    
   if n == 1:
       print (n)
       return 1    
   else:
       return n * factorial(n-1)
>>> factorial(5)
120

For a function that involves a loop, it is called only once. Therefore only one copy is created for all the variables used in it. Also, as function call uses stack to return function value to calling environment, only limited stack operations are performed.

In case of recursion, repeated calls to same function with different arguments are initiated. Hence that many copies of local variables are created in the memory and that many stack operations are needed.  

Obviously, a recursive call by function to itself causes an infinite loop. Hence recursive call is always conditional. Recursive approach provides a very concise solution to complex problem having many iterations.

In case of iterative version of factorial function, it is a straightforward case of cumulative multiplication of numbers in a range. However, in case of recursive function it is implementation of mathematical definition of factorial as below:

n! = n X (n-1)!

Here we have used factorial itself to define factorial. Such problems are ideally suited to be expressed recursively.

We can perform this calculation using a loop as per following code:

Now we shall try to write recursive equivalent of above loop. Look at mathematical definition of factorial once again.

n!=nX(n-1)!

Substitute n with 5. The expression becomes

5!=5X4! = 5X4X3! = 5X4X3X2! = 5X4X3X2X1! = 5X4X3X2X1 = 120

Let us see graphically the step by step process of computing factorial value of 5.

The diagram illustrates how successively performing factorial calculation of number by decrementing the number till it reaches 1. This involves more cost of computing.

Hence it can be seen that recursion is more expensive in terms of resource utilization. It is also difficult to write a recursive function as compared to a straightforward repetitive or iterative solution.

Having said that, recursion is sometimes preferred especially in cases where iterative solution becomes very big, complex and involves too many conditions to keep track of. There are some typical applications where we have to employ recursive solution. Traversal of binary tree structure, sorting algorithms such as heap sort, finding traversal path etc are typical applications of recursion.

On a lighter note, to be able to write a recursive solution gives a lot of satisfaction to the programmer!

‘assert’ is one of 33 keywords in Python language. Essentially it checks whether given boolean expression is true. If expression is true, Python interpreter goes on to execute subsequent code. However, if the expression is false, AssertionError is raised bringing execution to halt.

To demonstrate usage of assert statement, let us consider following function:

>>> def testassert(x,y):
assert y!=0
print ('division=',x/y)

Here, the division will be performed only if boolean condition is true, but AssertionError raised if it is false

>>> testassert(10,2)
division= 5.0
>>> testassert(10,0)
Traceback (most recent call last):
 File "<pyshell#8>", line 1, in <module>
   testassert(10,0)
 File "<pyshell#6>", line 2, in testassert
   assert y!=0
AssertionError

AssertionError is raised with a string in assert statement as custom error message

>>> def testassert(x,y):
assert y!=0, 'Division by Zero not allowed'
print ('division=',x/y)
>>> testassert(10,0)
Traceback (most recent call last):
 File "<pyshell#11>", line 1, in <module>
   testassert(10,0)
 File "<pyshell#10>", line 2, in testassert
   assert y!=0, 'Division by Zero not allowed'
AssertionError: Division by Zero not allowed

Instead of letting the execution terminate abruptly, the AssertionError is handled with try – except construct as follows:

>>> def testassert(x,y):
try:
assert y!=0
print ('division', x/y)
except AssertionError:
print ('Division by Zero not allowed')
>>> testassert(10,0)
Division by Zero not allowed

An object that can invoke a certain action or process is callable. The most obvious example is a function. Any function, built-in or user defined or a method in built-in or user defined class is an object of Function class

>>> def myfunction():
print ('Hello')
>>> type(myfunction)
<class 'function'>
>>> type(print)
<class 'builtin_function_or_method'>

Python’s standard library has a nuilt-in callable() function that returns true if object is callable. Obviously a function returns true.

>>> callable(myfunction)
True
>>> callable(print)
True

A callable object with parentheses (having arguments or not) invokes associated functionality. In fact any object that has access to __call__() magic method is callable. Above function is normally called by entering myfunction(). However it can also be called by using __call_() method inherited from function class

>>> myfunction.__call__()
Hello
>>> myfunction()
Hello

Numbers, string, collection objects etc are not callable because they do not inherit __call__() method

>>> a=10
>>> b='hello'
>>> c=[1,2,3]
>>> callable(10)
False
>>> callable(b)
False
>>> callable(c)
False

In Python, class is also an object and it is callable.

>>> class MyClass:
def __init__(self):
print ('called')
>>> MyClass()
called
<__main__.MyClass object at 0x7f37d8066080>
>>> MyClass.__call__()
called

<__main__.MyClass object at 0x7f37dbd35710>

However, object of MyClass is not callable.

>>> m=MyClass()
called
>>> m()
Traceback (most recent call last):
 File "<pyshell#23>", line 1, in <module>
   m()
TypeError: 'MyClass' object is not callable

In order that object of user defined be callable, it must override __call__() method

>>> class MyClass:
def __init__(self):
print ('called')
def __call__(self):
print ('this makes object callable')
>>> m=MyClass()
called
>>> m()
this makes object callable

In Python, single and double asterisk (* and **) symbols are defined as multiplication and exponentiation operators respectively. However, when prefixed to a variable, these symbols carry a special meaning. Let us see what that means.

A Python function is defined to receive a certain number of arguments. Naturally the same number of arguments must be provided while calling the function, otherwise Python interpreter raises TypeError as shown below.

>>> def add(x,y):
return x+y
>>> add(11,22)
33
>>> add(1,2,3)
Traceback (most recent call last):
 File "<pyshell#8>", line 1, in <module>
   add(1,2,3)
TypeError: add() takes 2 positional arguments but 3 were given

This is where the use of formal argument prefixed with single * comes. In a function definition, an argument prefixed with * is able to receive variable number of values from the calling environment and stores the values in a tuple object.

>>> def add(*numbers):
s=0
for n in numbers:
s=s+n
return s
>>> add(11,22)
33
>>> add(1,2,3)
6
>>> add(10)
10
>>> add()
0

Python allows a function to be called by using the formal arguments as keywords. In such a case, the order of argument definition need not be followed.

>>> def add(x,y):
return x+y
>>> add(y=10,x=20)
30

However, if you want to define a function which should be able to receive variable number of keyword arguments, the argument is prefixed by ** and it stores the keyword: values passed as a dictionary.

>>> def add(**numbers):
s=0
for k,v in numbers.items():
s=s+v
return s
>>> add(a=1,b=2,c=3,d=4)
10

In fact, a function can have positional arguments, variable number of arguments, keyword arguments with defaults and variable number of keyword arguments. In order to use both variable arguments and variable keyword arguments, **variable should appear last in the argument list.

>>> def add(x,y,*arg, **kwarg):
print (x,y)
print (arg)
print (kwarg)
>>> add(1,2,3,4,5,6,a=10,b=20,c=30)
1 2
(3, 4, 5, 6)
{'a': 10, 'b': 20, 'c': 30}

Calling a function by value is found to be used in C/C++ where a variable is actually a named location in computer memory. Hence the passed expression is copied into the formal argument which happens to be a local variable of the called function. Any manipulation of that local variable doesn’t affect the variable that was actually passed.

The following C program has square() function. It is called from the main by passing x which is copied to the local variable in square(). The change inside called function doesn’t have impact on x in main()

#include <stdio.h>
int main()
{
   int square(int);
int x=10;
printf("\nx before passing: %d",x);
square(x);
printf("\nx after passing: %d",x);
}
int square(int x)
{
x=x*x;
}
result:
x before passing: 10
x after passing: 10

In Python though, a variable is just a label of an object in computer memory. Hence formal as well as actual argument variables in fact are two different labels of same object – in this case 10. This can be verified by id() function.

>>> def square(y):
print (id(y))
>>> x=10
>>> id(x)
94172959017792
>>> square(x)
94172959017792

Hence we can infer that Python calls a function by passing reference of actual arguments to formal arguments. However, what happens when the function manipulates the received object depends on the type of object.

When a function makes modifications to the immutable object received as an argument,  changes do not reflect in the object that was passed.

>>> def square(y):
print ('received',id(y))
y=y**2
print ('changed',id(y),'y=',y)
>>> x=10
>>> id(x)
94172959017792
>>> square(x)
received 94172959017792
changed 94172959020672 y= 100
>>> x
10

It can be seen that x=10 is passed to the square() function. Inside, y is changed. However, y becomes a label of another object 100, which is having different id(). This change doesn’t reflect in x – it being an immutable object. The same thing happens to a string or tuple.

However, if we pass a mutable object, any changes by called function will also have effect on that object after returning from the function.

In the example below, we pass a list (it is a mutable object) to a function and then add some more elements to it. After returning from the function, the original list is found to be modified.

>>> def newlist(list):
print ('received', list, id(list))
list.append(4)
print ('changed', list, id(list))
>>> num=[1,2,3]
>>> id(num)
139635530080008
>>> newlist(num)
received [1, 2, 3] 139635530080008
changed [1, 2, 3, 4] 139635530080008
>>> num
[1, 2, 3, 4]

Hence we can say that a mutable object, passed to a function by reference gets modified by any changes inside the called function.

A must-know for anyone looking for top Python interview questions, this is one of the frequently asked Python programming interview questions.

As per the principles of object oriented programming, data encapsulation is a mechanism by which instance attributes are prohibited from direct access by any environment outside the class. In C++ / Java, with the provision of access control keywords such as public, private and protected it is easy to enforce data encapsulation. The instance variables are usually restricted to have private access, though the methods are publicly accessible.

However, Python doesn’t follow the doctrine of controlling access to instance or method attributes of a class. In a way, all attributes are public by default. So in a strict sense, Python doesn’t support data encapsulation. In the following example, name and age are instance attributes of User class. They can be directly manipulated from outside the class environment.

>>> class User:
def __init__(self):
self.name='Amar'
self.age=20
>>> a=User()
>>> a.name
'Amar'
>>> a.age
20
>>> a.age=21

Python even has built-in functions getattr() and setattr() to fetch and set values of an instance attribute.

>>> getattr(a, 'name')
'Amar'
>>> setattr(a,'name','Ravi')
>>> a.name
'Ravi'

Having said that, Python does have means to emulate private keywords in Java/C++. An instance variable prefixed with ‘__’ (double underscore) behaves as a private variable – which means direct access to it will raise an exception as below:

>>> class User:
def __init__(self):
self.__name='Amar'
self.__age=20
>>> a=User()
>>> a.__name
Traceback (most recent call last):
 File "<pyshell#18>", line 1, in <module>
   a.__name
AttributeError: 'User' object has no attribute '__name'

However, the double underscore prefix doesn’t make price truly private (as in case of Java/C++). It merely performs name mangling by internally renaming the private variable by adding the "_ClassName" to the front of the variable. In this case, variable named "__name" in Book class will be mangled in “_User__name” form.

>>> a._User__name
'Amar'

Protected member is (in C++ and Java) accessible only from within the class and its subclasses. Python accomplishes this behaviour by convention of prefixing the name of your member with a single underscore. You’re telling others “don’t touch this, unless you’re a subclass”.

The def keyword is used to define a new function with a user specified name. Lambda keyword on the other hand is used to create an anonymous (un-named) function. Usually such a function is created on the fly and meant for one-time use. The general syntax of lambda is as follows:

lambda arg1, arg2… : expression

A lambda function can have any number of arguments but there’s always a single expression after : symbol. The value of the expression becomes the return value of the lambda function. For example

>>> add=lambda a,b:a+b

This is an anonymous function declared with lambda keyword. It receives a and b and returns a+b.. The anonymous function object is assigned to identifier called add. We can now use it as a regular function call

>>> print (add(2,3))
5

The above lambda function is equivalent to a normal function declared using def keyword as below:

>>> def add(a,b):
return a+b
>>> print (add(2,3))
5

However, the body of the lambda function can have only one expression. Therefore it is not a substitute for a normal function having complex logic involving conditionals, loops etc.

In Python, a function is a called a first order object, because function can also be used as argument, just as number, string, list etc. A lambda function is often used as argument function to functional programming tools such as map(), filter() and reduce() functions.

The built-in  map() function subjects each element in the iterable to another function which may be either a built-in function, a lambda function or a user defined function and returns the mapped object. The map() function needs two arguments:

map(Function, Sequence(s))

Next example uses a lambda function as argument to map() function. The lambda function itself takes two arguments taken from two lists and returns the first number raised to second. The resulting mapped object is then parsed to output list.

>>> powers=map(lambda x,y: x**y, [10,20,30], [1,2,3])
>>> list(powers)
[10, 400, 27000]

The filter() function also receives two arguments, a function with Boolean return value and an iterable. Only those items for whom the function returns True are stored in a filter object which can be further cast to a list.

Lambda function can also be used as a filter. The following program uses lambda function that filters all odd numbers from a given range.

>>> list(filter(lambda x: x%2 == 1,range(1,11)))
[1, 3, 5, 7, 9]

The intention of ‘with’ statement in Python is to make the code cleaner and much more readable. It simplifies the management of common resources like file streams.

The ‘with’ statement establishes a context manager object that defines the runtime context. The context manager handles the entry into, and the exit from, the desired runtime context for the execution of the block of code. Context managers are normally invoked using the ‘with’ statement.

Here is a typical use of with statement. Normally a file object is declared with built-in open() function. After performing file read/write operations the file object must be relieved from memory by its close() method failing which the disk file/stream underneath the object may be corrupted.

>>> f=open('file.txt','w')
>>> f.write('Hello')
>>> f.close()

Use of ‘with’ statement sets up a context for the following block of statements. As the block finishes, the context object also automatically goes out of scope. It need not be explicitly destroyed.

>>> with open('file.txt','w') as f:
f.write('Hello')

Note that the object f is no longer alive after the block and you don’t have to call close() method as earlier. Hence, the code becomes clean and easy to read.

The file object acts as a context manager object by default. To define a custom context manager, the class must have __enter__() and __exit__() methods.

class Mycontext:
   def __init__(self):
       print ('object initialized')
   def __enter__(self):
       print ('context manager entered')
   def __exit__(self, type, value, traceback):
       print ('context manager exited')
with Mycontext() as x:
   print ('context manager example')

Output:

object initialized
context manager entered
context manager example
context manager exited

A module in Python is a Python script that may have definitions of class, functions, variables etc. having certain similar nature or intended for a common purpose. For example math module in Python library contains all frequently required mathematical functions. This modular approach is taken further by the concept of package. A package may contain multiple modules or even subpackages which may be a part of resources required for a certain purpose. For example Python library’s Tkinter package contains various modules that contain functionality required for building GUI.

Hierarchical arrangement of modules and subpackages is similar to the operating system’s file system. Access to a certain file in the file tree is obtained by using ‘/’. For example a file in a certain folder may have its path as c:\newdir\subdir\file.py. Similarly a module in newdir package will be named as newdir.subdir.py

For Python to recognize any folder as a package, it must have a special file named __init__.py which may or may not be empty and also serves as a packaging list by specifying resources from its modules to be imported.

Modules and their contents in a package can be imported by some script which is at the same level. If we have a.py, b.py and c.py modules and __init__.py file in a test folder under newdir folder, one or more modules can be imported by test.py in newdir as under:

#c:/newdir/test.py
from test import a
a.hello()
#to import specific function
from test.a import hello
hello()

The __init__.py file can be customized to allow package level access to functions/classes in the modules under it:

#__init__.py
from .a import hello
from .b import sayhello

Note that hello() function can now be access from the package instead of its module

#test.py
import test
test.hello()

The test package is now being used by a script which is at the same level as the test folder. To make it available for system-wide use, prepare the following setup script:

#setup.py
from setuptools import setup
setup(name='test',
     version='0.1',
     description='test package',
     url='#',
     author='....',
     author_email='...@...',
     license='MIT',
     packages=['test'],
     zip_safe=False)

To install the test package:

C:\newdir>pip install .

Package is now available for import from anywhere in the file system. To make it publicly available, you have to upload it to PyPI repository.

Python’s built-in open() function returns a file object representing a disk file although it can also be used to open any stream such as byte stream or network stream. The open() function takes two arguments:

f=open(“filename”, mode)

Default value of mode parameter is ‘r’ which stands for reading mode. To open a file for storing data, the mode should be set to ‘w’

f=open(‘file.txt’,’w’)
f.write(‘Hello World’)
f.close()

However, ‘w’ mode causes overwriting file.txt if it is earlier present, thereby the earlier saved data is at risk of being lost. You can of course check if the file already exists before opening, using os module function:

import os
if os.path.exists('file.txt')==False:
   f=open('file.txt','w')
   f.write('Hello world')
   f.close()

This can create a race condition though because of simultaneous conditional operation and open() functions. To avoid this ‘x’ mode has been introduced, starting from Python 3.3. Here ‘x’ stands for exclusive. Hence, a file will be opened for write operation only if it doesn’t exist already. Otherwise Python raises FileExistsError

try:
   with open('file.txt','x') as f:
       f.write("Hello World")
   except FileExistsError:
       print ("File already exists")

Using ‘x’ mode is safer than checking the existence of file before writing and guarantees avoiding accidental overwriting of files.

Python offers more than one way to check if a certain class is inherited from the other. In Python each class is a subclass of object class. This can be verified by running built-in issubclass() function on class A as follows:

>>> class A:
pass
>>> issubclass(A,object)
True

We now create B class that is inherited from A. The issubclass() function returns true for A as well as object class.

>>> class B(A):
pass
>>> issubclass(B,A)
True
>>> issubclass(B,object)
True

You can also use __bases__ attribute of a class to find out its super class.

>>> A.__bases__
(<class 'object'>,)
>>> B.__bases__
(<class '__main__.A'>,)

Finally the mro() method returns method resolution order of methods in an inherited class. In our case B is inherited from A, which in turn is a subclass of object class.

>>> B.mro()
[<class '__main__.B'>, <class '__main__.A'>, <class 'object'>]

Depending upon where in the program a certain variable is declared, it is classified as a global, local or nonlocal variable. As the name suggests, any variable accessible to any function is a global variable. Any variable declared before a call to function (or any block of statements) is having global scope.

def square():
   print (x*x)
x=10
square()
output:
100

Here, x is a global variable hence it can be accessed from anywhere, including from square() function. If we modify it in the function, its effect is persistent to global variable.

def square():
   x=20
   print ('in function',x)
x=10
square()
print ('outside function',x)
output:
in function 20
outside function 10

However, if we try to increment x in the function, it raises UnBoundLocalError

def square():
   x=x+1
   print ('in function',x)
x=10
square()
print ('outside function',x)
output:
   x=x+1
UnboundLocalError: local variable 'x' referenced before assignment

A local variable is one which is declared inside any block of statements such as a function. Such a variable has only local presence which means, if we try to access outside its scope, an error will be reported.

def square():
   x=10
   x=x**2
   print ('in function',x)
square()
print ('outside function',x)
output:
   print ('outside function',x)
NameError: name 'x' is not defined

Note that if we use a variable of the same name as that available in the global namespace inside a function, it becomes a local variable as far as the local namespace is concerned. Hence, it will not modify the global variable.

def square():
   x=5
   x=x**2
   print ('in function',x)
x=100
square()
print ('outside function',x)
output:
in function 25
outside function 100

If however, you insist that your function needs to access global and local variables of the same name, you need to refer the former with ‘global’ keyword.

def square():
   global x
   x=x**2
   print ('global x in function',x)
x=10
square()
print ('global x outside function',x)
output:
global x in function 100
global x outside function 100

Python's global() function returns a dictionary object of all global variables keys and their respective values.

def square():
   print ('global x in function',globals()['x'])
x=10
square()
output:
global x in function 10

Python 3 introduced nonlocal keyword. It should be used to refer to a variable in the scope of an outer function from an inner function. Python does allow definition of nested functions. This nonlocal variable is useful in this context.

In following program function2 is nested inside function1 which is having var1 as its local variable. For function2 to refer to var1 it has to use a nonlocal keyword because it is neither a local nor a global variable.

def function1():
   var1=20
   def function2():
       nonlocal var1
       var1=50
       print (var1)
   print ('nonlocal var1 before inner function',var1)
   function2()
   print ('nonlocal var1 after inner function',var1)
var1=10
function1()
print ('global var1 after outer function unchanged',var1)
output:
nonlocal var1 before inner function 20
50
nonlocal var1 after inner function 50
global var1 after outer function unchanged 10

Format specification symbols (%d, %f, %s etc) are popularly used in C/C++ for formatting strings. These symbols can be used in Python also. Just as in C, a string is constructed by substituting these symbols with Python objects.

In the example below, we use the symbols %s and %d in the string and substitute them by values of objects in tuple outside the string, prefixed by % symbol

>>> name='Ravi'
>>> age=21
>>> string="Hello. My name is %s. I am %d years old" %(name,age)
>>> string
'Hello. My name is Ravi. I am 21 years old'

In numeric formatting symbols, the width of number can be specified before and/or after a decimal point.

>>> price=50
>>> weight=21.55
>>> string="price=%3d weight=%3.3f" %(price,weight)
>>> string
'price= 50 weight=21.550'

Since Python 3.x a new format() method has been added in built-in string class which is more efficient and elegant, and is prescribed to be the recommended way of formatting with variable substitution.

Instead of  % formatting operator, {} symbols are used as place holders.

>>> name='Ravi'
>>> age=21
>>> string="Hello. My name is {}. I am {} years old".format(name,age)
>>> string
'Hello. My name is Ravi. I am 21 years old'

Format specification symbols are used with : instead of %. So to insert a string in place holder use {:s} and for integer use {:d}. Width of numeric and string variables is specified as before.

>>> price=50
>>> weight=21.55
>>> string="price={:3d} quantity={:3.3f}".format(price,weight)
>>> string
'price= 50 quantity=21.550'

At first, it gives the impression that given expression is invalid because of railing comma. But it is perfectly valid and Python interpreter doesn’t raise any error.

As would be expected, expression without railing comma declares an int object.

>>> num=100
>>> type(num)
<class 'int'>

Note that use of parentheses in the representation of tuple is optional. Just comma separated values form a tuple.

>>> x=(10,20)
>>> type(x)
<class 'tuple'>
>>> #this is also a tuple even if there are no parentheses
... x=10,20
>>> type(x)
<class 'tuple'>

Hence a number (any object for that matter) followed by comma, and without parentheses forms a tuple with single element.

>>> num=100,
>>> type(num)
<class 'tuple'>

As mentioned above omitting comma is a regular assignment of int variable.

While the two names sound similar, they are in fact entirely different.

Python is a general purpose programming language that is on top of the popularity charts among programmers worldwide. IPython, on the other hand, is an interactive command-line terminal for Python. 

Standard Python runs in the interactive mode by invoking it from the command terminal to start a REPL (Read, Evaluate, Print and Loop) with Python prompt >>>

IPython is an enhanced Python REPL with much more features as compared to standard Python shell. Some of the features are given below:

• Syntax highlighting
• Stores the history of interactions
• Offers features such as Tab completion of keywords, variables and function names
• Has object introspection feature
• Has magic command system for controlling Python environment
• Can be embedded in other Python programs
• Acts as a main kernel for Jupyter notebook.

IPython has been developed by Fernando Perez in 2001. Its current version is IPython 7.0.1, that runs on Python 3.4 version or higher. IPython has spun off into Project Jupyter that provides a web based interface for the IPython shell.

IPython is bundled with Anaconda distribution. If you intend to install IPython with standard Python, use pip utility

pip install ipython

Or better still, you can install Jupyter. IPython, being one of its dependencies, will be installed automatically.

pip install jupyter

The following diagram shows the IPython console.

The Jupyter notebook frontend of IPython looks as below:

In Python, range is one of the built-in objects. It is an immutable sequence of integers between two numbers separated by a fixed interval. The range object is obtained from built-in range() function.

The range() function has two signatures as follows:

range(stop)
range(start, stop[, step])

The function can have one, two or three integer arguments named as start, stop and step respectively. It returns an object that produces a sequence of integers from start (inclusive) to stop (exclusive) by step.

In case only one argument is given it is treated as stop value and start is 0 by default, step is 1 by default.

>>> #this will result in range from 0 to 9
>>> range(10)
range(0, 10)

If it has two arguments, first is start and second is stop. Here step is 1 by default.

>>> #this will return range of integers between 1 to 9 incremented by 1
>>> range(1,10)
range(1, 10)

If the function has three arguments, they will be used as start, stop and step parameters. The range will contain integers from start to step-1 separated by step.

>>> #this will return range of odd numbers between 1 to 9
>>> range(1,10,2)
range(1, 10, 2)

Elements in the range object are not automatically displayed. You can cast range object to list or tuple.

>>> r=range(1,10,2)
>>> list(r)
[1, 3, 5, 7, 9]

The most common use of range is to traverse and process periodic sequence of integers using ‘for’ loop. The following statement produces the square of all numbers between 1 to 10.

>>> for i in range(11):
print ('square of {} : {}'.format(i,i**2))
square of 0 : 0
square of 1 : 1
square of 2 : 4
square of 3 : 9
square of 4 : 16
square of 5 : 25
square of 6 : 36
square of 7 : 49
square of 8 : 64
square of 9 : 81
square of 10 : 100

The range object has all methods available to a sequence, such as len(), index() and count()

>>> r=range(1,10,2)
>>> len(r)
5
>>> r.index(7)
3
>>> r.count(5)
1

It is also possible to obtain an iterator out of the range object and traverse using next() function.

>>> it=iter(r)
>>> while True:
try:
print(next(it))
except StopIteration:
break
1
3
5
7
9

The acronym JSON stands for JavaScript Object Notation based on a subset of the JavaScript language. It is a lightweight data interchange format which is easy for humans to use and easy for machines to parse.

Python library provides json module for converting Python objects to JSON format and vice versa. It contains dumps() and loads() functions to serialize and de-serialize Python objects.

>>> import json
>>> #dumps() converts Python object to JSON
>>> data={'name':'Raju', 'subjects':['phy', 'che', 'maths'], 'marks':[50,60,70]}   
>>> jso=json.dumps(data)
>>> #loads() converts JSON string back to Python object
>>> Pydata=json.loads(jso)
>>> Pydata
{'name': 'Raju', 'subjects': ['phy', 'che', 'maths'], 'marks': [50, 60, 70]}

The json module has dump() and load() functions that perform serialization and deserialization of Python object to a File like object such as disk file or network stream. In the following example, we store JSON string to a file using load() and retrieve Python dictionary object from it. Note that File object must have relevant ‘write’ and ‘read’ permission.

>>> data={'name':'Raju', 'subjects':['phy', 'che', 'maths'], 'marks':[50,60,70]}   
>>> file=open('testjson.txt','w')
>>> json.dump(data, file)   
>>> file.close()

The ‘testjson.txt’ file will show following contents:

{"name": "Raju", "subjects": ["phy", "che", "maths"], "marks": [50, 60, 70]}

To load the json string from file to Python dictionary we use load() function with file opened in ‘r’ mode.

>>> file=open('testjson.txt','r')
>>> data=json.load(file)
>>> data
{'name': 'Raju', 'subjects': ['phy', 'che', 'maths'], 'marks': [50, 60, 70]}
>>> file.close()

The json module also provides object oriented API for the purpose with the help of JSONEncode and JSONDecoder classes.

The module also defines the relation between Python object and JSON data type for interconversion, as in the following table:

A Python script has .py extension and can have definition of class, function, constants, variables and even some executable code. Also, any Python script can be imported in another script using import keyword.

However, if we import a module having certain executable code, it will automatically get executed even if you do not intend to do so. That can be avoided by identifying __name__ property of module.

When Python is running in interactive mode or as a script, it sets the value of this special variable to '__main__'. It is the name of the scope in which top level is being executed.

>>> __name__
'__main__'

From within a script also, we find value of __name__ attribute is set to '__main__'. Execute the following script.

#example.py
print ('name of module:',__name__)
C:\users>python example.py
name of module: __main__

However, for imported module this attribute is set to the name of the Python script. (excluding the extension .py)

>>> import example
>>> messages.__name__
'example'

If we try to import example module in test.py script as follows:

#test.py
import example
print (‘name of module:’,__name__)
output:
c:\users>python test.py
name of module:example
name of module: __main__

However we wish to prevent the print statement in the executable part of example.py. To do that, identify the __name__ property of the module. If it is __main__ then only the print statement will be executed if we modify example.py as below:

#example.py
if __name__==”__main__”:
     print (‘name of module:,__name__)

Now the output of test.py will not show the name of the imported module.

Python’s import mechanism makes it possible to use code from one script in another. Any Python script (having .py extension) is a module. Python offers more than one way to import a module or class/function from a module in other script.

Most common practice is to use ‘import’ keyword. For example, to import functions in math module and call a function from it:

>>> import math
>>> math.sqrt(100)
10.0

Another way is to import specific function(s) from a module instead of populating the namespace with all contents of the imported module. For that purpose we need to use ‘from module import function’ syntax as below:

>>> from math import sqrt, exp
>>> sqrt(100)
10.0
>>> exp(5)
148.4131591025766

Use of wild card ‘*’ is also permitted to import all functions although it is discouraged; instead, basic import statement is preferred.

>>> from math import *

Python library also has __import__() built-in function. To import a module using this function:

>>> m=__import__('math')
>>> m.log10(100)
2.0

Incidentally, import keyword internally calls __import__() function.

Lastly we can use importlib module for importing a module. It contains __import__() function as an alternate implementation of built-in function. The import_module() function for dynamic imports is as below:

>>> mod=input('name of module:')
name of module:math
>>> m=importlib.import_module(mod)
>>> m.sin(1.5)
0.9974949866040544

Polymorphism is an important feature of object oriented programming. It comes into play when there are commonly named methods across subclasses of an abstract base class. 

Polymorphism is the ability to present the same interface for differing underlying forms. This allows functions to use objects of any of these polymorphic classes without needing to be aware of distinctions across the classes.

If class B inherits from class A, it doesn’t have to inherit everything about class A, it can do some of the things that class A does differently. It is most commonly used while dealing with inheritance. Method in Python class is implicitly polymorphic. Unlike C++ it need not be made explicitly polymorphic.

In the following example, we first define an abstract class called shape. It has an abstract method area() which has no implementation.

import abc
class Shape(metaclass=abc.ABCMeta):
    def __init__(self,tp):
        self.shtype=tp
        @abc.abstractmethod
        def area(self):
                pass

Two subclasses of shape - Rectangle and Circle are then created. They provide their respective implementation of inherited area() method.

class Rectangle(Shape):
        def __init__(self,nm):
            super().__init__(nm)
            self.l=10
            self.b=20
        def area(self):
                return self.l*self.b
class Circle(Shape):
    def __init__(self,nm):
        super().__init__(nm)
        self.r=5
    def area(self):
        import math
        return math.pi*pow(self.r,2)

Finally we set up objects with each of these classes and put in a collection. 

r=Rectangle('rect')
c=Circle('circle')
shapes=[r,c]
for sh in shapes:
    print (sh.shtype,sh.area())

Run a for loop over a collection. A generic call to area() method calculates the area of respective shape.

Output:
rect 200
circle 78.53981633974483

IP address is a string of decimal (or hexadecimal) numbers that forms a unique identity of each device connected to a computer network that uses Internet Protocol for data communication. The IP address has two purposes: identifying host and location addressing.

To start with IP addressing scheme uses 32 bits comprising of four octets each of 8 bits having a value equivalent to a decimal number between 0-255. The octets are separated by dot (.). For example 111.91.41.196

This addressing scheme is called Ipv4 scheme.  As a result of rapid increase in the number of devices directly connected to the internet, this scheme is being gradually replaced by Ipv6 version.

An IPv6 address uses hexadecimal digits to represent a string of unique 128 bit number. Each position in an IPv6 address represents four bits with a value from 0 to f. The 128 bits are divided into 8 groupings of 16 bits each separated by colons.

Example: 2001:db8:abcd:100::1/64

The IPv6 address always uses CIDR notation to determine how many of the leading bits are used for network identification and rest for host/interface identification.

Python's ipaddress module provides the capability to create, manipulate and operate on IPv4 and IPv6 addresses and networks. Following factory functions help to conveniently create IP addresses, networks and interfaces:

ip_address():

This function returns an IPv4Address or IPv6Address object depending on the IP address passed as argument. 

>>> import ipaddress
>>> ipaddress.ip_address('192.168.0.1')
IPv4Address('192.168.0.1')
>>> ipaddress.ip_address('2001:ab7::')
IPv6Address('2001:ab7::')

ip_network():

This function returns an IPv4Network or IPv6Network object depending on the IP address passed as argument.

>>> ipaddress.ip_network('192.168.100.0/24')
IPv4Network('192.168.100.0/24')
>>> ipaddress.ip_network('2001:db8:abcd:100::/64')
IPv6Network('2001:db8:abcd:100::/64')

ip_interface():

This function returns IPv4Interface or IPv6Interface object .
>>> ipaddress.ip_interface('192.168.100.10/24')
IPv4Interface('192.168.100.10/24')
>>> ipaddress.ip_interface('2001:db8:abcd:100::1/64')
IPv6Interface('2001:db8:abcd:100::1/64')

The pip is a very popular package management utility used to install and manage Python packages especially those hosted on Python’s official package repository called Python Package index(PyPI).

The pip utility is distributed by default in standard Python’s distributions with versions 2.7.9 or later or 3.4 or later. For other Python versions, you may have to obtain it by first downloading get-pip.py from https://bootstrap.pypa.io/get-pip.py and running it.

First, check the installed version of pip on your system.

$ pip –version

Note that for Python 3.x, the pip utility is named as pip3.

Most common usage of pip is to install a certain package using the following syntax:

$ pip install nameofpackage
#for example
$ pip install flask

In order to install a specific version of the package, version number is followed by name

$ pip install flask==1.0

Use --upgrade option to upgrade the local version of the package to the latest version on PyPi.

$ pip --upgrade flask

Many a time, a project needs many packages and they may be having a lot of dependencies. In order to install all of them in one go, first prepare a list of all the packages needed in a file called ‘requirements.txt’ and ask pip to use it for batch installation.

$ pip -r requirements.txt

Uninstalling a package is easy

$ pip uninstall packagename

If the package needs to be installed from a repository other than PyPI, use –index-url option.

$ pip --index-url path/to/package/ packagename

There is ‘search’ option to search for a specific package available.

$ pip search pyqt

The show option displays additional information of a package

$ pip show sqlalchemy
Name: SQLAlchemy
Version: 1.1.13
Summary: Database Abstraction Library
Home-page: http://www.sqlalchemy.org
Author: Mike Bayer
Author-email: [email protected]
License: MIT License
Location: /home/lib/python3.6/site-packages

The freeze option shows installed packages and their versions.

$ pip freeze

Unit testing is one of the various software testing methods where individual units/ components of software are tested. Unit testing refers to tests that verify the functionality of a specific section of code, usually at the function level. 

The unittest module is a part of Python’s standard library. This unit testing framework was originally inspired by Junit. Individual units of source code, such as functions, methods, and class are tested to determine whether they are fit for use. Intuitively, one can view a unit as the smallest testable part of an application. Unit tests are short code fragments created by programmers during the development process.

Test case is the smallest unit of testing. This checks for a specific response to a particular set of inputs. unittest provides a base class, TestCase, which may be used to create new test cases.

test suite is a collection of test cases, test suites, or both and is used to aggregate tests that should be executed together. Test suites are implemented by the TestSuite class.

Following code is a simple example of unit test using TestCase class.

First let us define a function to be tested. In the following example, add() function is to be tested.

Next step is to Create a testcase by subclassing unittest.TestCase.

Inside the class, define a test as a method with name starting with 'test'.

Each test call assert function of TestCase class. The assertEquals() function compares result of add() function with arg2 argument and throws assertionError if comparison fails.

import unittest
def add(x,y):
   return x + y
class SimpleTest(unittest.TestCase):
   def testadd1(self):
      self.assertEquals(add(10,20),30)
if __name__ == '__main__':
   unittest.main()

Run above script. Output shows that the test is passed.

Ran 1 test in 0.060s
OK

Possible outcomes are:

• OK: test passes
• FAIL: test doesn’t pass and AssertionError exception is raised.
• ERROR: test raises exception other than AssertionError

The unittest module also has a command line interface to be run using Python’s -m option.

Python -m unittest example.py

Python script containing a definition of function, class or module usually contains a ‘docstring’ which appears as a comment but is treated as a description of the corresponding object.

The doctest module uses the docstring. It searches docstring text for interactive Python sessions and verifies their output if it is exactly as the output of the function itself.

The doctests are useful to check if docstrings are up-to-date by verifying that all interactive examples still work as documented. They also perform regression testing by verifying interactive examples from a test file or a test object. Lastly they serve as an important tool to write tutorial documentation for a package.

Following code is a simple example of doctest. The script defines add() function and embeds various test cases in docstring in interactive manner.  The doctest module defines testmod() function which runs the interactive examples and checks output of function for mentioned test cases with the one mentioned in docstring. If the output matches, test is passed otherwise it is failed. 

def add(x,y):
   """Return the factorial of n, an exact integer >= 0.
   >>> add(10,20)
   30
   >>> add('aa','bb')
   'aabb'
   >>> add('aaa',20)
   Traceback (most recent call last):
   ...
   TypeError: must be str, not int
   """
   return x+y
if __name__ == "__main__":
   import doctest
   doctest.testmod()

Save the above script as example.py and run it from command line. To get verbose output of doctests use -v option.

$ python example.py ##shows no output
$ python example.py -v
Trying:
    add(10,20)
Expecting:
    30
ok
Trying:
    add('aa','bb')
Expecting:
    'aabb'
ok
Trying:
    add('aaa',20)
Expecting:
    Traceback (most recent call last):
    ...
    TypeError: must be str, not int
ok
1 items had no tests:
    __main__
1 items passed all tests:
   3 tests in __main__.add
3 tests in 2 items.
3 passed and 0 failed.
Test passed

Test examples can be put in a separate textfile and subjected as argument to testfile() function instead of using testmod() function as above.

There are three distinct situations in Python script where ‘as’ keyword is appropriately used.

1. to define alias for imported package, module or function.

Alias for module in a package
>>> from os import path as p
alias for a module
>>> import math as m
>>> m.sqrt(100)
10.0
alias for imported function
>>> from math import sqrt as underroot
>>> underroot(100)
10.0

1. to define a context manager variable:

>>> with open("file.txt",'w') as f:
f.write("Hello")

1. as argument of exception:

When an exception occurs inside the try block, it may have an associated value known as the argument defined after exception type with ‘as’ keyword. The type of the argument depends on the exception type. The except clause may specify a variable after the exception name. The variable is bound to an exception instance with the arguments stored in instance.args.

>>> def divide(x,y):
try:
z=x/y
print (z)
except ZeroDivisionError as err:
print (err.args)
>>> divide(10,0)
('division by zero',)

Errors detected during execution are called exceptions. In Python, all exceptions must be instances of a class that derives from BaseException. Python library defines many subclasses of BaseException. They are called built-in exceptions. Examples are TypeError, ValueError etc.

When user code in try: block raises an exception, program control is thrown towards except block of corresponding predefined exception type. However, if the exception is different from built-in ones, user can define a customized exception class. It is called user defined exception.

Following example defines MyException as a user defined exception.

class MyException(Exception):
   def __init__(self, num):
      self.num=num
   def __str__(self):
      return "MyException: invalid number"+str(self.num)

Built-in exceptions are raised implicitly by the Python interpreter. However, exceptions of user defined type must be raised explicitly by using raise keyword.

The try: block accepts a number from user and raises MyException if the number is beyond the range 0-100. The except block then processes the exception object as defined.

try:
   x=int(input('input any number: '))
   if x not in range(0,101):
      raise MyException(x)
   print ("Number is valid")
except MyException as m:
   print (m)

Output:

input any number: 55
Number is valid
input any number: -10
MyException: invalid number-10

This, along with other Python advanced interview questions, is a regular feature in Python interviews. Be ready to tackle it with the approach mentioned.

Obviously, print() is most commonly used Python statements and is used to display the value of one or more objects. However, it accepts other arguments other than objects to be displayed.

print(*objects, sep=' ', end='\n', file=sys.stdout, flush=False)

The function can have any number of objects as positional arguments, all separated by commas. Other arguments are optional but must be used as keyword arguments.

By default, the non-keyword arguments that is the list of objects is displayed with whitespace as a separator. However, you can define separator symbol with the help of sep argument.

>>> x=10
>>> y=20
>>> z=30
>>> print (x,y,z)
10 20 30
>>> #using comma as separator
>>> print (x,y,z, sep=',')
10,20,30

Second keyword argument controls the end of line character. By default output of each print() statement terminates with newline character or ‘\n’. If you wish you can change it to any desired character.

print ("Hello World")
print ("Python programming")
#this will cause output of next statement in same line
print ("Hello World", end=' ')
print ("Python programming")
output:
Hello World
Python programming
Hello World Python programming

The file argument decides the output stream to which result of print() is directed. By default it is sys.stdout which is standard output

• device – the primary display device of the system. It can be changed to any file like object such as a disk file, bytearray, network stream
• etc – any object possessing write() method.

The buffer argument is false by default, but if the flush keyword argument is true, the stream is forcibly flushed.

SQL injection is the technique of inserting malicious SQL statements in the user input thereby gaining unauthorized access to database and causing harm in an interactive data driven application.

Most computer applications including Python applications that involve database interaction use relational database products that work on SQL queries. Python has standardized database adapter specifications by DB-API standard. Python standard distribution has sqlite3 module which is a DB-API compliant SQLite driver. For other databases the module must be installed. For example, if using MySQL, you will have to install PyMySQL module.

Standard procedure of executing SQL queries is to first establish connection with database server, obtain cursor object and then execute SQL query string. Following describes the sequence of steps for SQLite database

con=sqlite3.connect(‘testdata.db’)
cur=con.cursor()
cur.execute(SQLQueryString)

For example, following code displays all records in products table whose category is ‘Electronics’

qry=”SELECT * from products where ctg==’Electronics’;”

However, if price is taken as user input then the query string will have to be constructed by inserting user input variable.

catg=input(‘enter category:’)
qry=”select * from prooducts where ctg={};”.format(catg)

if input is Electronics, the query will be built as select * from products where ctg=’Electronics’But a malicious input such as following is given enter category:electronics;drop table products;Now this will cause the query to be as follows:

• select * from products where ctg=’Electronics’;drop table products;
• Subsequent cur.execute(qry) statemnt will inadvertently delete products table as well!
• To prevent this DB-API recommends use of parameterized queries using ? Placeholder.
• cur.execute(‘select * from products where ctg=?’, (catg))
• In some database adapter modules such as mysql, other placeholder may be used
• cur.execute(‘select * from products where ctg=%s’, (catg))

In such prepared statements, database engine checks validity of interpolated data and makes sure that undesirable query is not executed, preventing SQL injection attack.

The dataclasses module is one of the most recent modules to be added in Python's standard library. It has been introduced since version 3.7 and defines @dataclass decorator that automatically generates following methods in a user defined class:

constructor method __init__()
string representation method __repr__()
__eq__() method which overloads == operator

The dataclass function decorator has following prototype:

@dataclass(init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False)

All the arguments are of Boolean value. Each decides whether a corresponding magic method will be automatically generated or not.

The 'init' parameter is True by default which will automatically generate __init__() method for the class.

Let us define Employee class using dataclass decorator as follows:

#data_class.py
from dataclasses import dataclass
@dataclass
class Employee(object):
     name : str
     age : int
     salary : float

The auto-generated __init__() method is like:

def __init__(self, name: str, age: int, salary: float):
       self.name = name
       self.age = age
       self.salary = salary

Auto-generation will not happen if the class contsins explicit definition of __init__() method.

The repr argument is also true by default.  A __repr__() method will be generated. Again the repr string will not be auto-generated if class provides explicit definition.

The eq argument forces __eq__() method to be generated. This method gets called in response to equals comparison operator (==). Similarly other operator overloading magic methods will be generated if the order argument is true (the default is False), __lt__(), __le__(), __gt__(), and __ge__() methods will be generated, they implement comparison operators < <= > ans >= respectively.

If unsafe_hash argument is set to False (the default), a __hash__() method is generated according to how eq and frozen are set.

frozen argument: If true (the default is False), it emulates read-only frozen instances.

>>> from data_class import Employee
>>> e1=Employee('XYZ', 21, 2150.50)
>>> e2=Employee('xyz', 20, 5000.00)
>>> e1==e2
False

Other useful functions in this module are as follows:

asdict():

This function converts class instance into a dictionary object.

>>> import dataclasses
>>> dataclasses.asdict(e1)
{'name': 'XYZ', 'age': 21, 'salary': 2150.5}

astuple():

This function converts class instance into a tuple object.

>>> dataclasses.astuple(e2)
('xyz', 20, 5000.0)

make_dataclass():

This function creates a new dataclass from the list of tuples given as fields argument.

>>> NewClass=dataclasses.make_dataclass('NewClass', [('x',int),('y',float)])
>>> n=NewClass(10,20)
>>> n
NewClass(x=10, y=20)

Function is a reusable block of statements. Whenever called, it performs a certain process on parameters or arguments if passed to it. Following function receives two integers and returns their division

>>> def division(x,y):
return x/y
>>> division(10,2)
5.0

While calling the function, arguments must be passed in the same sequence in which they have been defined. Such arguments are called positional arguments. If it is desired to provide arguments in any other order, you must use keyword arguments as follows:

>>> def division(x,y):
return x/y
>>> division(y=2, x=10)
5.0

The formal names of arguments are used as keywords while calling the function. However, using keyword arguments is optional. Question is how to define a function that can be called only by using keywords arguments?

To force compulsory use of keyword arguments, use '*' as one parameter in parameter list. All parameters after * become keyword only parameters. Any parameters before * continue to be positional parameters.

>>> def division(*,x,y):
return x/y
>>> division(10,2)
Traceback (most recent call last):
 File "<pyshell#7>", line 1, in <module>
   division(10,2)
TypeError: division() takes 0 positional arguments but 2 were given
>>> division(y=2, x=10)
5.0
>>> division(x=10, y=2)
5.0

In above example, both arguments are keyword arguments. Hence keyword arguments must be used even if you intend to pass values to arguments in the same order in which they have been defined.

A function can have mix of positional and keyword arguments. In such case, positional arguments must be defined before ‘*’

In following example, the function has two positional and two keyword arguments. Positional arguments may be passed using keywords also. However, positional arguments must be passed before keyword arguments.

>>> def sumdivide(a,b,*,c,d):
return (a+b)/(c-d)
>>> sumdivide(10,20,c=10, d=5)
6.0
>>> sumdivide(c=10, d=5,a=10,b=20)
6.0
>>> sumdivide(c=10, d=5,10,20)
SyntaxError: positional argument follows keyword argument

Expect to come across this, one of the most important Python interview questions for experienced professionals in data science, in your next interviews.

When you install Python and add Python executable to PATH environment variable of your OS, it is available for use from anywhere in the file system. Thereafter, while building a Python application, you may need Python libraries other than those shipped with the standard distribution (or any other distribution you may be using – such as Anaconda). Those libraries are generally installed using pip installer or conda package manager. These packages are installed in site-packages directory under Python’s installation folder.

However, sometimes, requirement of a specific version of same package may sometimes be conflicting with other application’s requirements. Suppose you have application A that uses library 1.1 and application B requires 1.2 version of the same package which may have modified or deprecated certain functionality of its older version. Hence a system-wide installation of such package is likely to break application A.

To avoid this situation, Python provides creation of virtual environment. A virtual environment is a complete but isolated Python environment having its own library, site-packages and executables. This way you can work on dependencies of a certain application so that it doesn’t clash with other or system wide environment.

For standard distribution of Python, virtual environment is set up using following command:

c:\Python37>python -m venv myenv

Here myenv is the name of directory/folder (along with its path. If none, it will be created under current directory) in which you want this virtual environment be set up. Under this directory, include, lib and scripts folders will be created. Local copy of Python executable along with other utilities such as pip are present in scripts folder. Script older also has activate and deactivate batch files.

To activate newly created environment, run activate.bat in Scripts folder.

C:\myenv>scripts\activate
(myenv)c:\python37>

Name of the virtual environment appears before command prompt. Now you can invoke Python interpreter which create will local copy in the new environment and not system wide Python.

To deactivate the environment, simply run deactivate.bat

If using Anaconda distribution, you need to use conda install command.

conda create --name myenv

This will create new environment under Anaconda installation folder’s envs directory.

Python’s built-in function library contains exec() function that is useful when it comes to dynamic execution of Python code. The prototype syntax of the function is as follows:

exec(object, globals, locals)

The object parameter can be a string or code object. The globals and locals parameters are optional. If only globals is provided, it must be a dictionary, which will be used for both the global and the local variables. If both are given, they are used for the global and local variables, respectively. The locals can be any mapping object.

In first example of usage of exec(), we have a string that contains a Python script.

>>> string='''
x=int(input('enter a number'))
y=int(input('enter another number'))
print ('sum=',x+y)
'''
>>> exec(string)
enter a number2
enter another number4
sum= 6

Secondly, we have a Python script saved as example.py

#example.py
x=int(input('enter a number'))
y=int(input('enter another number'))
print ('sum=',x+y)

The script file is read using built-in open() function as a string and then exec() function executes the code

>>> exec(open('example.py').read())
enter a number1
enter another number2
sum= 3

As a third example, the string containing Python code is first compiled to code object using compile() function

>>> string='''
x=int(input('enter a number'))
y=int(input('enter another number'))
print ('sum=',x+y)
'''
>>> obj=compile(string,'example', 'exec')
>>> exec(obj)
enter a number1
enter another number2
sum= 3

Thus we can see how exec() function is useful for dynamic execution of Python code.

Python is an interpreter based language. Hence, a Python program is run by executing one statement at a time. Object version of the entire script is not created as in C or C++. As a result, Python interpreter must be available on the machine on which you intend to run Python program. On a Windows machine, a program is executed by following the command line:

C:\users>python example.py

The Python executable must be available in the system’s path.

Although the same command line can be used for running a program on Linux, there is one more option. On Linux, the Python script can be given execute permission by invoking chmod. However, for that to work, the first line in the script must specify the path to Python executable. This line, though it appears as a comment, is called shebang.

#!/usr/bin/python
def add(x,y):
   return x+y
x=10
y=20
print ('addition=',add(x,y))

The first line in above script is the shebang line and mentions the path to the Python executable. This script is now made executable by chmod command:

$ chmod +x example.py

The Python script now becomes executable and doesn’t need executable’s name for running it.

$ ./example.py

However, the system still needs Python installed. So in that sense this is not self-executable. For this purpose, you can use one of the following utilities:

py2exe

cx-freeze

py2app

pyinstaller

The pyinstaller is very easy to use. First install pyinstaller with the help of pip

$ pip install pyinstaller

Then run the following command from within the directory in which your Python script is present:

~/Desktop/example$ pyinstaller example.py

In the same directory (here Desktop/example) a dist folder will be created. Inside dist, there is another directory with name of script (in this case example) in which the executable will be present. It can now be executed even if the target system doesn’t have Python installed.

~/Desktop/example/dist/example$ ./example

This is how a self-executable Python application is bundled and distributed. The dist folder will contain all modules in the form of libraries.

Instructions written in C/C++ are translated directly in a hardware specific machine code. On the other hand, Java/Python instructions are first converted in a hardware independent bytecode and then in turn the virtual machine which is specific to the hardware/operating system converts them in corresponding machine instructions.

The official distribution of Python hosted by https://www.python.org/ is called Cpython. The Python interpreter (virtual machine) software is written in C. We can call it as a C implementation of Python.

There are many alternatives to this official implementation. They are explained as below:

• Jython: Jython is a JRE implementation of Python. It is written in Java. It is designed to run on Java platform. Just as a regular Python module/package, a Jython program can import and use any Java class. Jython program also compiles to bytecode. One of the main advantages is that a user interface designed in Python can use GUI elements of AWT, swing or SWT package.
• Jython follows closely the standard Python implementation. Jython was created in 1997 by Jim Hugunin. Jython 2.0 was released in 1999. Current Jython 2.7.0 released in May 2015, corresponds to CPython 2.7. Development of Jython 3.x is under progress. It can be downloaded from https://www.jython.org/downloads.html
• IronPython: IronPython is a .NET implementation of Python. It is completely written in C#. Power of .NET framework can be easily harnessed by Python programs. Python’s rapid development tools are also easily embeddable in .NET applications. Similar to JVM, IronPython runs on Microsoft’s Common Language Runtime (CLR)
• IronPython’s current release, version 2.7.9 can be downloaded from https://ironpython.net/download/
• PyPy: PyPy is a fast, compliant alternative implementation of the Python language. It uses Just-in-Time compiler, because of which Python programs often run faster on PyPy. PyPy programs also consume less space than they do in CPython. PyPy is highly compatible with existing Python code.  PyPy comes by default with support for stackless mode, providing micro-threads for massive concurrency.
• Both CPython 2.x and 3.x compatible versions of PyPy are available for download on https://www.pypy.org/download.html

Although Python is predominantly an object oriented programming language, it does have important functional programming capabilities included in functools module.

One such function from the functools module is partial() function which returns a partial object. The object itself behaves like a function. The partial() function receives another function as argument and freezes some portion of a function’s arguments resulting in a new object with a simplified signature.

Let us consider the signature of built-in int() function which is as below:

int(x, base=10) 

This function converts a number or string to an integer, or returns 0 if no arguments are given.  If x is a number, returns x.__int__(). For floating point numbers, this truncates towards zero. If x is not a number or if base is given, then x must be a string.

>>> int(20)
20
>>> int('20', base=8)
16

We now use partial() function to create a callable that behaves like the int() function where the base argument defaults to 8.

>>> from functools import partial
>>> partial_octal=partial(int,base=8)
>>> partial_octal('20')
16

In the following example, a user defined function power() is used as argument to a partial function square() by setting a default value on one of the arguments of the original function.

>>> def power(x,y):
return x**y
>>> power(x=10,y=3)
1000
>>> square=partial(power, y=2)
>>> square(10)
100

The functools module also defines partialmethod() function that returns a new partialmethod descriptor which behaves like partial except that it is designed to be used as a method definition rather than being directly callable.

We normally come across login page of web application where a secret input such as a password or PIN is entered in a text field that masks the actual keys. How do we accept user input in a console based program? This is where the getpass module proves useful.

The getpass module provides a portable way to handle such password inputs securely so that the program can interact with the user via the terminal without showing what the user types on the screen.

The module has the getpass() function which prompt the user for a password without echoing. The user is prompted using the string prompt, which defaults to 'Password: '.

import getpass
pwd=getpass.getpass()
print ('you entered :', pwd)

Run above script as example.py from command line:

$ python example.py
Password:
you entered: test

You can change the default prompt by explicitly defining it as argument to getpass() function

pwd=getpass.getpass(prompt='enter your password: ')
$ python example.py
enter your password:
you entered test

The getpass() function can also receive optional stream argument to redirect the prompt to a file like object such as sys.stderr – the default value of stream is sys.stdout.

There is also a getuser() function in this module. It returns the “login name” of the user by checking the environment variables LOGNAME, USER, LNAME and USERNAME.

The mechanism of execution of a Python program is similar to Java. A Python script is first compiled to a bytecode which in turn is executed by a platform specific Python virtual machine i.e. Python interpreter. However, unlike Java, bytecode version is not stored as a disk file. It is held temporarily in the RAM itself and discarded after execution.

However, things are different when a script imports one or more modules. The imported modules are compiled to bytecode in the form of .pyc file and stored in a directory named as __pycache__.

Following Python script imports another script as module.

#hello.py
def hello():
   print ("Hello World")
#example.py
import hello
hello.hello()

Run example.py from command line

Python example.py

You will find a __pycache__ folder created in current working directory with hello.cpython-36.pyc file created in it.

The advantage of this compiled file is that whenever hello module is imported, it need not be converted to bytecode again as long as it is not modified.

As mentioned above, when executed, .pyc file of the .py script is not created. If you want it to be explicitly created, you can use py_compile and compileall modules available in standard library.

To compile a specific script

import py_compile
py_compile.compile(‘example.py’, ‘example.pyc’)

This module can be directly used from command line

python -m py_compile example.py

To compile all files in a directory

import compileall
compileall.compile_dir(‘newdir’)

Following command line usage of compileall module is also allowed:

python -m compileall newdir

Where newdir is a folder under current working directory.

Don't be surprised if this question pops up as one of the top Python interview questions and answers in your next interview. It has all the elements of a standard interview question.

Garbage collection is the mechanism of freeing up a computer’s memory by identifying and disposing of those objects that are no longer in use. C and C++ language compilers don’t have the capability of automatically disposing memory contents. That’s why disposal of memory resources has to be manually performed. Runtime environments of Java and Python have automatic garbage collection feature hence they provide efficient memory management.

Garbage collection is achieved by using different strategies. Java uses a tracing strategy wherein it determines which objects should be added to the garbage, by tracing the objects that are reachable by a chain of references from certain root objects. Those that are not reachable are considered as garbage and collecting them.

Python on the other hand uses a reference counting strategy. In this case, Python keeps count of the number of references to an object. The count is incremented when a reference to it is created, and decremented when a reference is destroyed. Garbage is identified by having a reference count of zero. When the count reaches zero, the object's memory is reclaimed.

For example 0 is stored as int object and is assigned to variable a, thereby incrementing its reference count to 1. However, when a is assigned to another object, the reference count of 10 becomes 0 and is eligible for garbage collection.

Python’s garbage collector works periodically. To identify and collect garbage manually, we have gc module in Python’s library.  You can also set the collection threshold value. Here, the default threshold is 700. This means when the number of allocations vs. the number of deallocations is greater than 700, the automatic garbage collector will run.

>>> import gc
>>> gc.get_threshold()
(700, 10, 10)

The collect() function identifies and collects all objects that are eligible.

The term Monkey patching is used to describe a technique used to dynamically modify the behavior of a piece of code at run-time. This technique allows us to modify or extend the behavior of libraries, modules, classes or methods at runtime without actually modifying the source code.

Monkey patching technique is useful in the following scenarios:

1. When we want to extend or modify the behavior of third-party or built-in libraries or methods at runtime without actually modifying the original code.

1. It is often found to be convenient when writing unit tests.

In Python, classes are just like any other mutable objects. Hence we can modify them or their attributes including functions or methods at runtime.

In the following demonstration of monkey patching, let us first write a simple class with a method that adds two operands.

#monkeytest.py
class MonkeyTest:
   def add(self, x,y):
       return x+y

We can call add() method straightaway

from monkeytest import MonkeyTest
a=MonkeyTest()
print (a.add(1,2))

However, we now need to modify add() method in the above class to accept a variable number of arguments. What do we do? First define a newadd() function that can accept variable arguments and perform addition.

def newadd(s,*args):
   s=0
   for a in args:s=s+int(a)
   return s

Now assign this function to the add() method of our MonkeyTest class. The modified behaviour of add() method is now available for all instances of your class.

MonkeyTest.add=newadd
a=MonkeyTest()
print (a.add(1,2,3))

Python doesn’t implement data encapsulation principle of object oriented programming methodology in the sense that there are no access restrictions like private, public or protected members. Hence a traditional method (as used in Java) of accessing instance variables using getter and setter methods is of no avail in Python.

Python recommends use of property object to provide easy interface to instance attributes of a class. The built-in property() function uses getter , setter and deleter methods to return a property attribute corresponding to a specific instance variable.

Let us first write a Student class with name as instance attributes. We shall provide getname() and setname() methods in the class as below:

class User:
   def __init__(self, name='Test'):
       self.__name=name
   def getname(self):
       return self.__name
   def setname(self,name):
       self.__name=name

We can declare its object and access its name attribute using the above methods.

>>> from example import User
>>> a=User()
>>> a.getname()
'Test'
>>> a.setname('Ravi')

Rather than calling setter and getter function explicitly, the property object calls them whenever the instance variable underlying it is either retrieved or set. The property() function uses the getter and setter methods to return the property object.

Let us add name as property object in the above User class by modifying it as follows:

class User:
   def __init__(self, name='Test'):
       self.__name=name
   def getname(self):
       print ('getname() called')
       return self.__name
   def setname(self,name):
       print ('setname() called')
       self.__name=name
   name=property(getname, setname)

The name property, returned by property() function hides private instance variable __name.  You can use property object directly. Whenever, its value is retrieved, it is internally called getname() method. On the other hand when a value is assigned to name property, internally setname() method is called.

>>> from example import User
>>> a=User()
>>> a.name
getname() called
'Test'
>>> a.name='Ravi'
setname() called

The property() function also can have deleter method. A docstring can also be specified in its definition.

property(getter, setter, deleter, docstring)

Python also has @property decorator that encapsulates property() function.

Python’s built-in function library provides input() and print() functions. They are default functions for console IO. The input() function receives data received from standard input stream i.e. keyboard and print() function sends data to standard output stream directed to standard output device i.e. display monitor device.

In addition to these, we can make use of standard input and output stream objects defined in sys module. The sys.stdin object is a file like object capable of reading data from a console device. It in fact is an object of TextIOWrapper class capable of reading data. Just as built-in File object, read() and readline() methods are defined in TextIOWrapper.

>>> import sys
>>> a=sys.stdin
>>> type(a)
<class '_io.TextIOWrapper'>
>>> data=a.readline()
Hello Python
>>> data
'Hello Python\n'

Note that unlike input() function, the trailing newline character (\n) is not stripped here.

On the other hand sys.stdout object is also an object of TextIOWrapper class that is configured to write data in standard output stream and has write() and writelines() methods.

>>> b.write(data)
Hello Python
13

The eval() function evaluates a single expression embedded in a string. The exec() function on the other hand runs the script embedded in a string. There may be more than one Python statement in the string.

Simple example of eval():

>>> x=5
>>> sqr=eval('x**2')
>>> sqr
25

Simple example of exec()

>>> code='''
x=int(input('enter a number'))
sqr=x**2
print (sqr)
'''
>>> exec(code)
enter a number5
25

eval() always returns the result of evaluation of expression or raises an error if the expression is incorrect. There is no return value of exec() function as it just runs the script.

Use of eval() may pose a security risk especially if user input is accepted through string expression, as  any harmful code may be input by the user.

Both eval() and exec() functions can perform operations on a code object returned by another built-in function compile(). Depending on mode parameter to this function, the code object is prepared for evaluation or execution.

According to Python’s standard documentation, compile() function returns code object from a string containing valid Python statements, a byte string or AST object.

A code object represents byte-compiled executable version of Python code or simply a bytecode. AST stands for Abstract syntax tree. The AST object is obtained by parsing a string containing Python code.

Syntax of compile() function looks as follows:

compile(source, filename, mode, flags=0, dont_inherit=False, optimize=-1)

The source parameter can be a string, bytestring or AST object. Filename parameter is required if you are reading code from a file. The code object can be executed by exec() or eval() functions which is mode parameter. Other parameters are optional.

Thee following example compiles given string to a code object for it to be executed using exec() function.

>>> string='''
x=10
y=20
z=x+y
print (z)
'''
>>> obj=compile(string,'addition', 'exec')
>>> exec(obj)
30

The following example uses eval as mode parameter in compile() function. Resulting code object is executed using eval() function.

>>> x=10
>>> obj=compile('x**2', 'square','eval')
>>> val=eval(obj)
>>> val
100

The AST object is first obtained by parse() function which is defined in ast module. It in turn is used to compile a code object.

>>> import ast
>>> astobj=parse(string)
>>> astobj=ast.parse(string)
>>> obj=compile(astobj,'addition', 'exec')
>>> exec(obj)
30

Whereas print() is a built-in function that displays value of one or more arguments on the output console, pprint() function is defined in the pprint built-in module. The name stands for pretty print. The pprint() function produces an aesthetically good looking output of Python data structures. Any data structure that is properly parsed by the Python interpreter is elegantly formatted. It tries to keep the formatted expression in one line as far as possible, but generates into multiple lines if the length exceeds the width parameter of formatting. 

In order to use it, first it must be imported from pprint module.

from pprint import pprint

One of the unique features of pprint output is that it automatically sorts dictionaries before the display representation is formatted. 

>>> from pprint import pprint
>>> ctg={'Electronics':['TV', 'Washing machine', 'Mixer'], 'computer':{'Mouse':200, 'Keyboard':150,'Router':1200},'stationery':('Book','pen', 'notebook')}
>>> pprint(ctg)
{'Electronics': ['TV', 'Washing machine', 'Mixer'],
 'computer': {'Keyboard': 150, 'Mouse': 200, 'Router': 1200},
 'stationery': ('Book', 'pen', 'notebook')}

The pprint module also has definition of PrettyPrinter class, and pprint() method belongs to it. 

>>> from pprint import PrettyPrinter
>>> pp=PrettyPrinter()
>>> pp.pprint(ctg)
{'Electronics': ['TV', 'Washing machine', 'Mixer'],
 'computer': {'Keyboard': 150, 'Mouse': 200, 'Router': 1200},
 'stationery': ('Book', 'pen', 'notebook')}

One of the most frequently posed Python technical interview questions, be ready for this conceptual question.

CGI, which stands for Common Gateway Interface, is a set of standards for the HTTP server to render the output of executable scripts. CGI is one of the earliest technologies to be used for rendering dynamic content to the client of a web application. All major http web server software, such as Apache, IIS etc are CGI aware. With proper configuration, executable scripts written C/C++, PHP, Python, Perl etc can be executed on the server and the output is dynamically rendered in the HTML form to the client’s browser.

One important disadvantage of CGI is that the server starts a new process for every request that it receives from its clients. As a result, the server is likely to face severe bottlenecks and is slow.

WSGI (Web Server Gateway Interface) on the other hand prescribes a set of standards for communication between web servers and web applications written in Python based web application frameworks such as Django, Flask etc. WSGI specifications are defined in PEP 3333 (Python enhanced Proposal)

WSGI enabled servers are designed to handle multiple requests concurrently. Hence a web application hosted on a WSGI server is faster.

There are many self-contained WSGI containers available such as Gunicorn and Gevent. You can also configure any Apache server for WSGI by installing and configuring mod_wsgi module. Many shared hosting services are also available for deploying Python web applications built with Django, Flask or other web frameworks. Examples are Heroku, Google App Engine, PythonAnywhere etc.

FieldStorage class is defined in cgi module of Python’s standard library. 

A http server can be configured to run executable scripts and programs stored in the cgi-bin directory on the server and render response in HTML form towards client. For example a Python script can be executed from cgi-bin directory.

However, the input to such a script may come from the client in the form of HTTP request either via its GET method or POST method.

In following HTML script, a form with two text input elements are defined. User data is sent as a HTTP request to a script mentioned as parameter of ‘action’ attribute with GET method.

<form action = "/cgi-bin/test.py" method = "get">
Name: <input type = "text" name = "name">  <br />
address: <input type = "text" name = "addr" />
<input type = "submit" value = "Submit" />
</form>

In order to process the user input, Python’s CGI script makes use of FieldStorage object. This object is actually a dictionary object with HTML form elements as key and user data as value.

#!/usr/bin/python
# Import CGI module 
import cgi
# Create instance of FieldStorage 
fso = cgi.FieldStorage()

The fso object now contains form data which can be retrieved using usual dictionary notation.

nm = fso[‘name’]
add=fso[‘addr’]

FieldStorage class also defines the following methods:

• getvalue(): This method retrieves value associated with form element. 
• getfirst(): This method returns only one value associated with form field name. The method returns only the first value in case more values were given under such name.
• getlist():This method always returns a list of values associated with form field name.

Both are built-in functions in standard library. Both functions return sequence objects. However, bytes() returns an immutable sequence, whereas bytearray is a mutable sequence.

The bytes() function returns a bytes object that is an immutable sequence of integers between 0 to 255.

>>> #if argument is an int, creates an array of given size, initialized to null
>>> x=10
>>> y=bytes(x)
>>> y
b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
>>> type(y)
<class 'bytes'>

If string object is used with bytes() function, you should use encoding parameter, default is ‘utf-8’

>> string='Hello'
>>> x='Hello'
>>> y=bytes(x,'utf-8')
>>> y
b'Hello'

Note that the bytes object has a literal representation too. Just prefix the string by b

>>> x=b'Hello'
>>> type(x)
<class 'bytes'>

On the other hand bytearray() returns a mutable sequence. If argument is an int, it initializes the array of given size with null.

>>> x=10
>>> y=bytearray(x)
>>> y
bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')
>>> type(y)
<class 'bytearray'>

However, if the argument is an iterable sequence, array size is equal to iterable count and items are initialized by elements of iterable. Following example shows bytearray object obtained from a list.

>>> x=[2,4,6]
>>> y=bytearray(x)
>>> y
bytearray(b'\x02\x04\x06')

Unlike bytes object, a bytearray object doesn’t have literal representation.

Both bytes and bytearray objects support all normal sequence operations like find and replace, join and count etc. 

>>> x=b'Hello'
>>> x.replace(b'l', b'k')
b'Hekko'

Both objects also support logical functions such as isalpha(), isalphanum(), isupper(), islower() etc.

>>> x=b'Hello'
>>> x.isupper()
False

Case conversion functions are also allowed.

>>> x=b'Hello'
>>> x.upper()
b'HELLO'

A class having one or more abstract methods is an abstract class. A method on the other hand is called abstract if it has a dummy declaration and carries no definition. Such a class cannot be instantiated and can only be used as a base class to construct a subclass. However, the subclass must implement the abstract methods to be a concrete class.

Abstract base classes (ABCs) introduce virtual subclasses. They don’t inherit from any class but are still recognized by isinstance() and issubclass() built-in functions.

For example, numbers module in standard library contains abstract base classes for all number data types. The int, float or complex class will show numbers. Number class is its super class although it doesn’t appear in the __bases__ attribute.

>>> a=10
>>> type(a)
<class 'int'>
>>> int.__bases__
(<class 'object'>,)
#However, issubclass() function returns true for numbers.Number
>>> issubclass(int, numbers.Number)
True
#also isinstance() also returns true for numbers.Number
>>> isinstance(a, numbers.Number)
True

The collections.abc module defines ABCs for Data structures like Iterator, Generator, Set, mapping etc.

The abc module has ABCMeta class which is a metaclass for defining custom abstract base class. In the following example Shape class is an abstract base class using ABCMeta. The shape class has area() method decorated by @abstractmethod decorator.

import abc
class Shape(metaclass=abc.ABCMeta):
        @abc.abstractmethod
        def area(self):
                pass

We now create a rectangle class derived from shape class and make it concrete by providing implementation of abstract method area()

class Rectangle(Shape):
        def __init__(self, x,y):
                self.l=x
                self.b=y
        def area(self):
                return self.l*self.b
r=Rectangle(10,20)
print ('area: ',r.area())

If the abstract base class has more than one abstract method, the child class must implement all of them otherwise TypeError will be raised.

Hashing algorithm is a technique of obtaining an encrypted version from a string using a certain mathematical function. Hashing ensures that the message is securely transmitted if it is intended for a particular recipient only.

Several hashing algorithms are currently in use. Secure hash algorithms SHA1, SHA224, SHA256, SHA384, and SHA512 are defined by Federal Information Processing Standard (FIPS). 

RSA algorithm defines md5 algorithm.

The hashlib module in standard library provides a common interface to many different secure hash and message digest algorithms. 

hashlib.new(name,[data]) function in the module is a generic constructor that takes the string name of the desired algorithm as its first parameter.

>>> import hashlib
>>> hash=hashlib.new('md5',b'hello')
>>> hash.hexdigest()
'5d41402abc4b2a76b9719d911017c592'

The hexdigest() function returns a hashed version of string in hexadecimal symbols.

Instead of using new() as generic constructor, we can use a named constructor corresponding to hash algorithm. For example for md5 hash,

>>> hash=hashlib.md5(b'hello')
>>> hash.hexdigest()
'5d41402abc4b2a76b9719d911017c592'

The hashlib module also defines algorithms_guaranteed and algorithms_available attributes.

The algorithms_guaranteed returns a set containing the names of the hash algorithms guaranteed to be supported by this module on all platforms. 

algorithms_available is a set containing the names of the hash algorithms that are available in the running Python interpreter.

>>> hashlib.algorithms_guaranteed
{'sha3_512', 'md5', 'sha3_256', 'sha224', 'sha384', 'sha3_224', 'sha256', 'shake_128', 'blake2b', 'sha1', 'sha3_384', 'blake2s', 'shake_256', 'sha512'}
>>> hashlib.algorithms_available
{'mdc2', 'SHA512', 'SHA384', 'dsaWithSHA', 'md4', 'shake_256', 'SHA224', 'sha512', 'DSA', 'RIPEMD160', 'md5', 'sha224', 'sha384', 'sha3_224', 'MD5', 'ripemd160', 'whirlpool', 'MDC2', 'sha', 'blake2b', 'SHA', 'sha1', 'DSA-SHA', 'sha3_512', 'sha3_256', 'sha256', 'SHA1', 'sha3_384', 'ecdsa-with-SHA1', 'shake_128', 'MD4', 'SHA256', 'dsaEncryption', 'blake2s'}

The vars() function can be operated upon those objects which have __dict__ attribute. It returns the __dict__ attribute for a module, class, instance, or any other object having a __dict__ attribute. If the object doesn’t have the attribute, it raises a TypeError exception. 

The dir() function returns list of the attributes of any Python object. If no parameters are passed it returns a list of names in the current local scope.  For a class , it returns a list of its attributes as well as those of the base classes recursively. For a module, a list of names of all the attributes, contained is returned.

When you access an object's attribute using the dot operator, python does a lot more than just looking up the attribute in that objects dictionary.

Let us take a look at the following class:

class myclass:
    def __init__(self):
        self.x=10
    def disp(self):
        print (self.x)

The vars() of above class is similar to its __dict__  value

>>> vars(myclass)
mappingproxy({'__module__': '__main__', '__init__': <function myclass.__init__ at 0x7f770e520950>, 'disp': <function myclass.disp at 0x7f770e5208c8>, '__dict__': <attribute '__dict__' of 'myclass' objects>, '__weakref__': <attribute '__weakref__' of 'myclass' objects>, '__doc__': None})
>>> myclass.__dict__
mappingproxy({'__module__': '__main__', '__init__': <function myclass.__init__ at 0x7f770e520950>, 'disp': <function myclass.disp at 0x7f770e5208c8>, '__dict__': <attribute '__dict__' of 'myclass' objects>, '__weakref__': <attribute '__weakref__' of 'myclass' objects>, '__doc__': None})

However, vars() of its object just shows instance attributes

>>> a=myclass()
>>> vars(a)
{'x': 10}
>>> x.__dict__
{'x': 10}

The dir() function returns the attributes and methods of myclass as well as object class, the base class of all Python classes.

>>> dir(myclass)
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'disp']

Whereas dir(a), the object of myclass returns instance variables of the object, methods of its class and object base class

>>> a=myclass()
>>> dir(a)
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'disp', 'x']

Python 3 was introduced in 2008. It was meant to be backward incompatible although some features have later been backported to Python 2.7. Still because of vast differences in specifications, Python 2.x code won’t run on Python 3.x environment. 

However, Python 3 library comes with a utility to convert Python 2.x code. This module is called 2to3. Command Line usage of this tool is as below:

$ 2to3 py2.py

To test this usage, save following code as py2.py

def sayhello(nm):
    print "Hello", nm
nm=raw_input("enter name")
sayhello(nm)

Above code uses Python 2.x syntax. The print statement doesn’t require parentheses. Also raw_input() function is not available in Python 3.x. Hence when we convert above code, print() function will have parentheses and raw_input will change to input()

As we run above command, a diff against the original source file is printed as below:

$ 2to3 py2.py
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Refactored py2.py
--- py2.py (original)
+++ py2.py (refactored)
@@ -1,5 +1,5 @@
 def sayhello(nm):
-    print "Hello", nm
+    print("Hello", nm)
-nm=raw_input("enter name")
+nm=input("enter name")
 sayhello(nm)
RefactoringTool: Files that need to be modified:
RefactoringTool: py2.py

2to3 can also write the needed modifications right back to the source file which is enabled with the -w flag. Changed version of script will be as shown.

$ 2to3 -w py2.py
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Refactored py2.py
--- py2.py (original)
+++ py2.py (refactored)
@@ -1,5 +1,5 @@
 def sayhello(nm):
-    print "Hello", nm
+    print("Hello", nm)
-nm=raw_input("enter name")
+nm=input("enter name")
 sayhello(nm)
RefactoringTool: Files that were modified:
RefactoringTool: py2.py

The py2.py will be modified to be compatible with python 3.x syntax

def sayhello(nm):
    print("Hello", nm)
nm=input("enter name")
sayhello(nm)

People often get confused between assertions and exceptions and often use the latter when the former should be used in a program (and vice versa).

The fundamental difference lies in the nature of the error to be caught and processed. Assertion is a statement of something that MUST be true. If not, then the program is terminated and you cannot recover from it. Exception on the other hand is a runtime situation that can be recovered from.

Python’s assert statement checks for a condition. If the condition is true, it does nothing and your program just continues to execute. But if it evaluates to false, it raises an AssertionError exception with an optional error message.

>>> def testassert(x,y):
try:
assert y!=0
print ('division', x/y)
except AssertionError:
print ('Division by Zero not allowed')
>>> testassert(10,0)
Division by Zero not allowed

The proper use of assertions is to inform developers about unrecoverable errors in a program. They’re not intended to signal expected error conditions, like “file not found”, where a user can take corrective action or just try again. The built-in exceptions like FileNotFoundError are appropriate for the same.

You can think of assert as equivalent to a raise-if statement as follows:

if __debug__:
    if not expression1:
        raise AssertionError(expression2)

__debug__ constant is true by default. 

Python’s assert statement is a debugging aid, not a mechanism for handling run-time errors for which exceptions can be used.

Assertion can be globally disabled by setting __debug__ to false or starting Python with -O option.

$python -O
>>> __debug__
False

Exceptions are generally fatal as the program terminates instantly when an unhandled exception occurs. Warnings are non-fatal. A warning message is displayed but the program won’t terminate. Warnings generally appear if some deprecated usage of certain programming element like keyword/function/class etc. occurs.

Python’s builtins module defines Warnings class which is inherited from Exception itself. However, custom warning messages can be displayed with the help of warn() function defined in built-in warnings module.

There are a number of built-in Warning subclasses. User defined subclass can also be defined.

Following code defines WarnExample class. When method1() is called, it issues a depreciation warning.

# warningexample.py 
import warnings
class WarnExample:
    def __init__(self):
        self.text = "Warning"
    def method1(self):
        warnings.warn(
            "method1 is deprecated, use new_method instead",
            DeprecationWarning
if __name__=='__main__':        
        e=WarnExample()
        e.method1()

Output:

$ python -Wd warningexample.py
warningexample.py:10: DeprecationWarning: method1 is deprecated, use new_method instead
  DeprecationWarning

Counter class is defined in collections module, which is part of Python’s standard library. It is defined as a subclass of built-in dict class. Main application of Counter object is to keep count of hashable objects. Counter object is a collection of key-value pairs, telling how many times a key has appeared in sequence or dictionary.

The Counter() function acts as a constructor. Without arguments, an empty Counter object is returned. When a sequence is given as argument, it results in a dictionary holding occurrences of each item.

>>> from collections import Counter
>>> c1=Counter()
>>> c1
Counter()
>>> c2=Counter('MALAYALAM')
>>> c2
Counter({'A': 4, 'M': 2, 'L': 2, 'Y': 1})
>>> c3=Counter([34,21,43,32,12,21,43,21,12])
>>> c3
Counter({21: 3, 43: 2, 12: 2, 34: 1, 32: 1})

Counter object can be created directly by giving a dict object as argument which itself should be k-v pair of element and count.

>>> c4=Counter({'Ravi':4, 'Anant':3, "Rakhi":3})

The Counter object supports all the methods that a built-in dict object has. One important method is elements() which returns a chain object and returns an iterator of element and value repeating as many times as value.

>>> list(c4.elements())
['Ravi', 'Ravi', 'Ravi', 'Ravi', 'Anant', 'Anant', 'Anant', 'Rakhi', 'Rakhi', 'Rakhi']

Python supports data compression using various algorithms such as zlib, gzip, bzip2 and lzma. Python library also has modules that can manage ZIP and tar archives.

Data compression and decompression according to zlib algorithm is implemented by zlib module. The gzip module provides a simple interface to compress and decompress files just as very popular GNU utilities GZIP and GUNZIP.

Following example creates a gzip file by writing compressed data in it.

>>> import gzip
>>> data=b'Python is Easy'
>>> with gzip.open("test.txt.gz", "wb") as f:
f.write(data)

This will create “test.txt.gz” file in the current directory.

In order to read this compressed file:

>>> with gzip.open("test.txt.gz", "rb") as f:
data=f.read()
>>> data
b'Python is Easy'

Note that the gz file should be opened in wb and rb mode respectively for writing and reading.

The bzip2 compression and decompression is implemented by bz2 module. Primary interface to the module involves following three functions:

1. Open(): opens a bzip2 compressed file and returns a file object. The file can be opened as binary/text mode with read/write permissions. 
2. write(): the file should be opened in ‘w’ or ‘wb’ mode. In binary mode, it writes compressed binary data to the file. In normal text mode, the file object is wrapped in TetIOWrapper object to perform encoding.
3. read(): When opened in read mode, this function reads it and returns the uncompressed data.

Following code writes the compressed data to a bzip2 file

>>> f=bz2.open("test.bz2", "wb")        
>>> data=b'SureCertificate Solutions Private Limited'        
>>> f.write(data)
 >>> f.close()

This will create test.bz2 file in the current directory. Any unzipping tool will show a ‘test’ file in it. To read the uncompressed data from this test.bz2 file use the following code:

>>> f=bz2.open("test.bz2", "rb")        
>>> data=f.read()        
>>> data        
b'SureCertificate Solutions Private Limited'

The Lempel–Ziv–Markov chain algorithm (LZMA) performs lossless data compression with a higher compression ratio than other algorithms. Python’s lzma module consists of classes and convenience functions for this purpose.

Following code is an example of lzma compression/decompression:

>>> import lzma
>>> data=b"SureCertificate Solutions Private Limited"
>>> f=lzma.open("test.xz","wb")
>>>f.write(data)
>>>f.close()

A ‘test.xz’ file will be created in the current working directory. To fetch uncompressed data from this file use the following code:

>>> import lzma
>>> f=lzma.open("test.xz","rb")
>>> data=f.read()
>>> data
b'SureCertificate Solutions Private Limited'

The ZIP is one of the most popular and old file formats used for archiving and compression. It is used by famous PKZIP application.

The zipfile module in Python’s standard library provides ZipFile() function that returns ZipFile object. Its write() and read() methods are used to create and read archive.

>>> import zipfile
>>> newzip=zipfile.ZipFile('a.zip','w')
>>> newzip.write('abc.txt')
>>> newzip.close()

To  read  data from a particular file in the archive

>>> newzip=zipfile.ZipFile('a.zip','r')
>>> data=newzip.read('abc.txt')
>>> data

Finally, Python’s tarfile module helps you to create a tarball of multiple files by applying different compression algorithms.

Following example opens a tar file for compression with gzip algorithm and adds a file in it.

>>> fp=tarfile.open("a.tar.gz","w:gz")        
>>> fp.add("abc.txt")        
>>> fp.close()

Following code extracts the files from the tar archive, extracts all files and puts them in current folder. 

>>> fp=tarfile.open("a.tar.gz","r:gz")        
>>> fp.extractall()        
>>> fp.close()

Duck typing is a special case of dynamic polymorphism and a characteristic found in all dynamically typed languages including Python. The premise of duck typing is a famous quote: "If it walks like a duck and it quacks like a duck, then it must be a duck".

In programming, the above duck test is used to determine if an object can be used for a particular purpose. With statically typed languages, suitability is determined by an object's type. In duck typing, an object's suitability is determined by the presence of certain methods and properties, rather than the type of the object itself. Here, the idea is that it doesn't actually matter what type data is - just whether or not it is possible to perform a certain operation.  

Take a simple case of addition operation. Normally addition of similar objects is allowed, for example the addition of numbers or strings. In Python, addition of two integers, x+y actually does int.__add__(x+y) under the hood.

>>> x=10
>>> y=20
>>> x+y
30
>>> int.__add__(x,y)
30

Hence, addition operation can be done on any objects whose class supports __add__() method. Python’s data model describes protocols of data types. For example sequence protocol stipulates that any class that supports iterator, len() and __getitem__() methods is a sequence. 

The simple example of duck typing given below demonstrates how any object may be used in any context, up until it is used in a way that it does not support:

class Duck:
    def fly(self):
        print("Duck flies")
class Airplane:
    def fly(self):
        print("Airplane flies")
class Kangaroo:
    def swim(self):
        print("Kangaroo runs")
d=Duck()
a=Airplane()
k=Kangaroo()
for b in [d,a,k]:
   b.fly()
output:
Duck flies
Airplane flies
Traceback (most recent call last):
  File "example.py", line 18, in <module>
    b.fly()
AttributeError: 'Kangaroo' object has no attribute 'fly'

Duck typing is possible in certain statically typed languages by providing extra type annotations  that instruct the compiler to arrange for type checking of classes to occur at run-time rather than compile time, and include run-time type checking code in the compiled output.

Python allows a function to be defined inside another function. Such functions are called nested functions. In such case, a certain variable is located in the order of local, outer, global and lastly built-in namespace order. The inner function can access variable declared in nonlocal scope.

>>> def outer():
data=10
def inner():
print (data)
inner()
>>> outer()
10

Whereas the inner function is able to access the outer scope, inner function is able to use variables in the outer scope through closures. Closures maintain references to objects from the earlier scope. 

A closure is a nested function object that remembers values in enclosing scopes even if they are not present in memory.

Following criteria must be met to create a closure in Python:

• We must have a nested function (function inside a function).
• The nested function must refer to a value defined in the enclosing function.
• The enclosing function must return the nested function.

Following code is an example of closure.

>>> def outer():
data=10
def inner():
print (data)
return inner
>>> closure=outer()
>>> closure()
10

A memory view object is returned by memoryview() a built-in function in standard library. This object is similar to a sequence but allows access to the internal data of an object that supports the buffer protocol without copying.

Just as in the C language, we access the memory using pointer variables; in Python; we use memoryview to access its referencing memory.

Buffer Protocol allows exporting an interface to access internal memory (or buffer). The built-in types bytes and bytearray supports this protocol and objects of these classes are allowed to expose internal buffer to allow to directly access the data.

Memory view objects are required to access the memory directly instead of data duplication.

The memoryview() function has one object that supports the buffer protocol as argument.

>>> obj=memoryview(b'HEllo Python')
>>> obj
<memory at 0x7ffa12e2b048>

This code creates a memoryview of bytesarray

>>> obj1=memoryview(bytearray("Hello Python",'utf-8'))
>>> obj1
<memory at 0x7ffa12e2b108>

A memoryview supports slicing and indexing to expose its data.

>>> obj[7]
121

Slicing creates a subview

>>> v=obj[6:]
>>> bytes(v)
b'Python'

This, along with other Python advanced interview questions, is a regular feature in Python interviews. Be ready to tackle it with the approach mentioned.