Lecture
19 Advanced Python Topics I
1. Iterators
Iterators
are everywhere in Python. They are elegantly implemented within for
loops, comprehensions, generators etc. but hidden in plain sight.
Iterator in Python is simply an object that can be iterated upon. An
object which will return data, one element at a time. Technically
speaking, Python iterator object must implement two special methods,
__iter__() and __next__(), collectively called the iterator protocol.
An object is called iterable if we can get an iterator from it. Most of
built-in containers in Python like: list,
tuple, string etc. are iterables. The iter() function
(which in turn calls the __iter__() method) returns an iterator from
them.
We
use the next() function to manually iterate through all the items of an
iterator. When we reach the end and there is no more data to be
returned, it will raise StopIteration. Following is an example.
A
more elegant way of automatically iterating is by using the for loop.
Using this, we can iterate over any object that can return an iterator,
for example list, string, file etc.
1.1 How for loop actually
works?
As
we see in the above example, the for loop was able to iterate
automatically through the list. In fact the for loop can iterate over
any iterable. Let's take a closer look at how the for loop is actually
implemented in Python.
Is actually implemented as.
So
internally, the for loop creates an iterator object, iter_obj by
calling iter() on the iterable. Ironically, this for loop is actually
an infinite while loop. Inside the loop, it calls next() to get the
next element and executes the body of the for loop with this value.
After all the items exhaust, StopIteration is raised which is
internally caught and the loop ends. Note that any other kind of
exception will pass through.
1.2 Building Your Own
Iterator in Python
Building an iterator from scratch is easy in Python. We just have to
implement the methods __iter__() and __next__(). The __iter__() method
returns the iterator object itself. If required, some initialization
can be performed. The __next__() method must return the next item in
the sequence. On reaching the end, and in subsequent calls, it must
raise StopIteration. Here, we show an example that will give us next
power of 2 in each iteration. Power exponent starts from zero up to a
user set number.
We can also use a for loop to iterate over our iterator class.
2. Python Generators
You'll
learn how to create iterations easily using Python generators, how is
it different from iterators and normal functions, and why you should
use it.
What are generators in Python? There is a lot of overhead in building
an iterator in Python; we have to implement a class with __iter__() and
__next__() method, keep track of internal states, raise StopIteration
when there was no values to be returned etc. This is both lengthy and
counter intuitive. Generator comes into rescue in such situations.
Python generators are a simple way of creating iterators. All the
overhead we mentioned above are automatically handled by generators in
Python. Simply speaking, a generator is a function that returns an
object (iterator) which we can iterate over (one value at a time).
How
to create a generator in Python? It is fairly simple to create a
generator in Python. It is as easy as defining a normal function with
yield statement instead of a return statement. If a function contains
at least one yield statement (it may contain other yield or return
statements), it becomes a generator function. Both yield and return
will return some value from a function. The difference is that, while a
return statement terminates a function entirely, yield statement pauses
the function saving all its states and later continues from there on
successive calls.
Differences between Generator function and a
Normal function
Here is how a generator function differs from a normal
function.
Generator function contains one or more yield statement.
* When
called, it returns an object (iterator) but does not start execution
immediately.
* Methods like __iter__()
and __next__() are implemented
automatically. So we can iterate through the items using next().
* Once
the function yields, the function is paused and the control is
transferred to the caller.
* Local variables and
their states are
remembered between successive calls.
* Finally, when the
function
terminates, StopIteration is raised automatically on further calls.
Here is an example to illustrate all of the points stated above. We
have a generator function named my_gen() with several yield statements.
One interesting thing to note in the above example is that, the value
of variable n is remembered between each call. Unlike normal functions,
the local variables are not destroyed when the function yields.
Furthermore, the generator object can be iterated only once. To restart
the process we need to create another generator object using something
like a = my_gen(). Note: One final thing to note is that we can use
generators with for loops directly. This is because, a for loop takes
an iterator and iterates over it using next() function. It
automatically ends when StopIteration is raised. Check here to know how
a for loop is actually implemented in Python.
Python Generators with a
Loop
The above example is of
less use and
we studied it just to get an idea of what was happening in the
background. Normally, generator functions are implemented with a loop
having a suitable terminating condition. Let's take an example of a
generator that reverses a string.
Generator
expression produces one item at a time. They are kind of lazy,
producing items only when asked for. For this reason, a generator
expression is much more memory efficient than an equivalent list
comprehension.
We havne done the following example using a list and a for loop:
However,
using the 'Generator Expression as follows, the generator expression
did not produce the required result immediately. Instead, it returned a
generator object with produces items on demand.
Take note of the parens on either side of the second line denoting a
generator expression, which, for the most part, does the same thing
that a list comprehension does, but does it lazily:
Why generators are used
in Python?
There are several reasons which make generators an attractive
implementation to go for.
1.
Easy to Implement Generators can be implemented in a clear and concise
way as compared to their iterator class counterpart. Following is an
example to implement a sequence of power of 2's using iterator class.
This was lengthy. Now lets do the same using a generator function.
Since, generators keep track of details automatically, it was concise
and much cleaner in implementation.
3. Python Closure
Before we do Closure, we need to know Python nested functions:
Defining a Closure Function
In
the example above, what would happen if the last line of the function
print_msg() returned the printer() function instead of calling it? This
means the function was defined as follows.
That's unusual.
The
print_msg() function was called with the string "Hello" and the
returned function was bound to the name another. On calling another(),
the message was still remembered although we had already finished
executing the print_msg() function. This technique by which some data
("Hello") gets attached to the code is called closure in Python.
This
value in the enclosing scope is remembered even when the variable goes
out of scope or the function itself is removed from the current
namespace. Try running the following in the Python shell to see the
output.
Here is a simple example of a Closure:
4. Python Decorators
Decorators
in Python make an extensive use of closures as well. Next, we will
learn Decorators:
What are decorators in Python? Python has an interesting feature called
decorators to add functionality to an existing code.
Here is an example:
When
you run the code, both functions first and second gives same output.
Here, the names first and second refer to the same function object. Now
things start getting weirder. Functions can be passed as arguments to
another function.
Such function that take other functions as
arguments are also called higher order functions. Here is an example of
such a function.
Even more weird, a function can return another function.
Here,
is_returned() is a nested function which is defined and returned, each
time we call is_called(). Finally, we must know about closures in
Python. We learned Closure from the previous section.
Getting
back to Decorators Functions and methods are called callable as they
can be called. In fact, any object which implements the special method
__call__() is termed callable. So, in the most basic sense, a decorator
is a callable that returns a callable. Basically, a decorator takes in
a function, adds some functionality and returns it.
In the example shown above, make_pretty() is a
decorator.
In the assignment step: pretty = make_pretty(ordinary), The function
ordinary() got decorated and the returned function was given the name
pretty.
We can see that the decorator function added some new
functionality to the original function. This is similar to packing a
gift. The decorator acts as a wrapper. The nature of the object that
got decorated (actual gift inside) does not alter. But now, it looks
pretty (since it got decorated). Generally, we decorate a function and
reassign it as,:
ordinary = make_pretty(ordinary)
This is
a common construct and for this reason, Python has a syntax to simplify
this. We can use the @ symbol along with the name of the decorator
function and place it above the definition of the function to be
decorated. For example,
This didn't work since I called the functions from the decorator. I
should call the inner function instead.
The wrapper (the decorator) will never be the most important part but
the gift (inner function) is.
One more example:
5. Chaining Decorators in
Python
Multiple
decorators can be chained in Python. This is to say, a function can be
decorated multiple times with different (or same) decorators. We simply
place the decorators above the desired function.
Swap it:
Tasks:
1.
What are the method(s) that iterator object must implement?
A. __iter__()
B. __iter__() and __next__()
C. __iter__() and __super__() __
D. iter__(), __super__() and __next__()
2. How can you create an iterator object from a list?
A. By passing the given list to the iter() function.
B. By using a for loop.
C. By using a while loop.
D. You cannot create an iterable object from the list.
3. If a function contains at least one yield statement, it becomes :
A. an iterable
B. a generator function
C. an anonymous function
D. None of the above
4. Use the 'one-liner' for loop and the 'next()' function to print
out the square of the first two elements of the following list.
5. What are the criterias that must be met to create closure in Python?
A. Program Must have a function inside a function.
B. The nested function must refer to a value defined in the enclosing
function.
C. The enclosing function must return the nested function.
D. All of the above.
6. What is the output of the following code?
7. What is the output of the following code?
8. Which of the following statement is true?
A. You cannot chain multiple decorators in Python.
B. Decorators doesn’t work with functions that take parameters.
C. The @ symbol doesn’t have any use while using decorators.
D. None of the above
9. The following Closure can generate the results as shown below:
Of course, you don't have to assign 'nf1=f(1), nf2=f(3)' but just do
this:
Convert this into the format that using the '@' symbol to decorate
function g() using the function f(). (as long as you use the '@' symbol
to decorate the function and get the same result, it if flexible how
you do this)
10. Design a function 'G(x)' to do the following calculation. This
function only passes the parameter 'x'. Use a closure function
F(a,b,c) to pass 'a, b, and c' to the function. Print the results.