Blog

A set data structure may be unfamiliar to a Python beginner. A set is used for sequence data structures. Therefore, you can have an image against a set like a list, tuple, and dictionary.

First, let’s look at a list as an example. The sample list “sample_list” has six elements, however, whose unique elements are three kinds. The elements are four “apple”, one “orange”, and one “grape”.

sample_list = [  
               "apple",
               "orange",
               "grape",
               "apple",
               "apple",
               "apple" 
              ]
print(sample_list)

>>  ['apple', 'orange', 'grape', 'apple', 'apple', 'apple']

You might have encountered the situation that you would like to know the unique elements of a list. Such a situation is the time to use the set() function in Python. Note that the set() is included in Python as a standard module, so you don’t need to import any external module.

It is easy. You just pass the list to the set() function as follows.

sample_list_unique = set(sample_list)
print(sample_list_unique)

>>  {'apple', 'orange', 'grape'}

You have found out the unique elements of the sample list.

Like the output of the above cell, a set is created by the curly braces {}.

set_sample_without_overlaps = {"apple", "orange", "grape"}
print(set_sample_without_overlaps)

>>  {'apple', 'orange', 'grape'}

Of course, if there are overlaps when we define the elements, these are ignored. Let’s put several “apple” elements in the set when defining. You will see that additional “apple” elements are ignored.

set_sample_with_overlaps = {"apple", "orange", "grape", "apple", "apple", "apple"}
print(set_sample_with_overlaps)

>>  {'apple', 'orange', 'grape'}

Application example

A set is so useful in data science. This is because we have many situations to confirm the overlaps between datasets.

Here is an example. We will check the overlaps of the id column between training and validation datasets. We can do this easily by the set() function and the “intersection” method. The intersection method returns the overlaps elements between two set-type data.

id_train = ["01", "02", "03", "04", "05"]
id_validation = ["04", "05", "06", "07", "08"]

# Into a set data structure
id_train_set = set(id_train)
id_validation_set = set(id_validation)

# Check an overlaps
id_overlap = id_train_set.intersection(id_validation_set)
print(id_overlap)

>>  {'05', '04'}

We have known that the elements of “05” and “04” coexist in the training and validation datasets. From this fact, we should perform a preprocessing, for example dropping overlap data.

Mixing the same information as the training data with the validation data is called data leakage, which leads to an overestimation of accuracy.

Summary

As a sequence data structure, a list, tuple, dictionary are famous. However, a set is practical when we treat unique elements.

You will surely come across a situation where you want to know the unique element of sequence-type data. Recall that there is a set data structure in the Python standard module s at that time.

This post is the next post of Part 1 and 2. A series of posts is intended to see the basics of Python.

The following contents were already introduced in the previous posts, Part 1 and 2.

• variables
• comment
• arithmetic operations
• boolean
• comparison operator
• list
• dictionary
• if statement
• for loop
• function

In this Part 3, we will learn the following contents.

• object
• class
• instance

Note) The sample code in this post is supposed to run on Jupyter Notebook.

object

Python is an object-oriented programming language. An object-oriented style makes it possible to write a more readable and flexible code. Therefore, to understand the concept of an object is highly important.

However, as we have seen in Part 1 and 2, an object-oriented style doesn’t appear. But, it is just Python has hidden object orientation. But from here on, let’s take advantage of object-orientation and take it one step further. This will make your code more functional and maintainable.

Everything that Python deals with is an object. For example, variables, list, function, ..etc. An object is often likened to a thing. In Python, we call a concrete object an instance. The concept of an instance is unfamiliar to beginners. However, please note that it is needed especially when creating a machine learning model.

Example to understand object

Let’s imagine an object with some examples.

First, how about the variable $x$, whose value is $1$.

x = 1
x

>>  1

You may be thinking that $x$ is a variable with the value of $1$. Or, $x$ equals $1$. However, recall the following fact.

type(x)

>>  int

Actually, the variable $x$ has a value of $1$ and the information of data type of “int“. We don’t usually think about the above. However, $x$ is a variable object that has value and data-type information.

We’re not aware of it because we just gave $x$ a value of $1$. But, in behind, Python also gives variable attribute information.

Next, let’s see another example of list.

a = [1, 2, 3]
type(a)

>>  list

The list $a$ has the values of [1, 2, 3] and the information of list. Here, please recall that we can add new element by the append() method.

a.append(10)
a

>>  [1, 2, 3, 10]

When be aware of object, the list object $a$ has the method append() and we called it by $a$.append(). In other words, the append() method was originally included in the list object $a$. The list object has values, information of data type, and functions.

Short summary of object

Could you have imagined an object from the above example? An object is a thing including values, information, and functions. Note that variables, lists, functions, etc. are objects that Python has as standard. We were unknowingly calling and using it.

From here, you will create your own objects with a next topic called “class”. Especially, when creating a machine learning model, we need to create our original object by class(). This is due to designing machine learning models by giving the objects model structure, training, and predictive functions.

class

Here, let’s create our own object by class. The sample code is below. We define a class by “class (class name)”. In the following, we created the class “MyClass()”. And, “__init__()” is for initializing an argument $x$ when we create an instance from the class. Although it is unfamiliar to beginners, a function in class must receive the own argument “self”, which is just an object itself. Then, each function in the class also receives “self”, making it possible to use the variables(self.x) and the functions(func1, func2, func3).

class MyClass():
  def __init__(self, x):
    self.x = x
  
  # f(x) = x
  def func1(self):
    return self.x

  # f(x) = x^2
  def func2(self):
    return self.x*self.x

  # f(x) = 10*x
  def func3(self):
    return 10*self.x

instance

An instance is a thing created from a class. Here, let’s create an instance with name of “instance” from the class “MyClass()”.

x = 5
instance = MyClass(x)

This instance has three functions(func1, func2, func3) defined in “MyClass()”. These functions can be called in the form of methods.

instance.func1()  # f(x) = x
>>  5

instance.func2()  # f(x) = x^2
>>  25

instance.func3()  # f(x) = 10*x
>>  50

Here, the “__call__()” method is introduced. This method is called without the form of “.method()”. Let’s take the following example. The shaded area is where “__call__()” was added.

class MyClass_updated():
  def __init__(self, x):
    self.x = x

  def __call__(self):
    if self.x < 4:
      return self.func1()
    elif self.x < 8:
      return self.func2()
    else:
      return self.func3()
  
  def func1(self):
    return self.x

  def func2(self):
    return self.x*self.x
    
  def func3(self):
    return 10*self.x

Then, create an instance from the class “MyClass_updated()” and call the instance. The point is that the “__call__()” is called at “instance()”.

x = 5
instance = MyClass_updated(x)
instance()  # __call__() is called

>>  25

At this point, we can convert $x$, which will vary from $1$ to $10$ in order, with the function $f(x)$. Note that $f(x)$ changes dependent on the range of $x$, whose conditional branching is shown in the figure below.

for x in range(1, 11):
  instance = MyClass_updated(x)
  print( instance() )  # __call__() is called

>>  1
>>  2
>>  3
>>  16
>>  25
>>  36
>>  49
>>  80
>>  90
>>  100

Summary

We have seen an object, class, and instance in Python. These topic may be unfamiliar to beginners. However, these are important especially for a data scientist. The basics of Python are covered in the series of posts of Part 1 – 3.

The next step is to learn the external Python library, such as NumPy, Pandas, and scikit-learn, for data science and machine learning. By calling these libraries from Python, you can take advantage of various functions. For example, NumPy makes it easier to perform numerical calculations. Pandas is useful for the treatment of table data. And, we can create a machine-learning model with low-codes by using scikit-learn.

Note that what you call from an external library is just a class someone created. You have already basic knowledge. And you can also create your own external library.

The author hopes this blog helps readers a little.

This post is the next post of Part 1.

A series of posts is intended to see the basics of Python. In the previous post, the following contents were introduced.

• Variables
• Comment “#”
• Arithmetic operations
• Boolean
• Comparison operator
• List

In this post, we will learn the following contents.

• dictionary
• if statement
• for loop
• function
• instance
• class

Note) The sample code in this post is supposed to run on Jupyter Notebook.

dictionary

A dictionary stores pairs of “key” and “value”. We can access the “value” of an element by the “key”.

Here, let’s see the example of a dictionary, which stores fruit names and their numbers. A dictionary is defined by “{ }”. However, when we access the value of the dictionary, we use “[ ]” whose argument is its key.

dic_fruit = {"apple":1, "orange":3, "grape":5}
dic_fruit["grape"]

>>  5

When we pass a key that is NOT included in a dictionary, Python returns the error, where “nut” does NOT exist in the dictionary “dic_fruit”.

dic_fruit["nut"]  # dic_fruit = {"apple":1, "orange":3, "grape":5}

>>  ---------------------------------------------------------------------------
>>  KeyError                                  Traceback (most recent call last)
>>  <ipython-input-11-2c89a279e528> in <module>()
>>  ----> 1 dic_fruit["nut"]
>>  
>>  KeyError: 'nut'

It is easy to add a new element as follows.

dic_fruit["nut"] = 100
print(dic_fruit)

>>  {'apple': 1, 'orange': 3, 'grape': 5, 'nut': 100}

The element of “nut” was added.

if statement

“if statement” is for a conditional statement, whether True or False. For example, if x is greater than zero, output “x > 0”. On the other hand, if x is less than zero, output “x < 0”. In Python, the above conditional statements are written as follows.

x = 5

if x > 0:
  print( "x > 0" )
elif x < 0:
  print( "x < 0" )
else:
  print( "x = 0")

>>  x > 0

In the above example, when x = 5, x > 0 is True. On the other hand, x < 0 is False. Note that we sometimes forget to add the colon “:”.

How about the input of “x = 0”? You can confirm the output “x = 0”, too.

The if statement branches according to the conditions. Python runs line by line from top to bottom lines.

----------------------------------------------
# condition1, condition2 are True or False.

if condition1:
  Processing 1  # When, condition1 is True
elif condition2:
  Processing 2  # When, condition2 is True
else:
  Processing 3  # When, otherwise
----------------------------------------------

An indent is required at the beginning of the “processing” line. Python recognizes it as processing from whether an indent exists. Note that we sometimes forget to add the colon “:” at the ending of the “if condition”.

It is the same when the variable type is “str”.

x = "orange"

if x == "aplle":
  print( "x is apple." )
else:
  print( "x is NOT apple" )

>>  x is NOT apple

By if statement, we can judge whether the element is included in the list. We use the “in” and “not in” operators.

words = ["orange", "grape", "peach"]

if "apple" in words:
  print( "words includes apple." )
elif "apple" not in words:
  print( "words NOT include apple." )

>>  words NOT include apple.

The point is that the “if(elif)” part is executed when True, and the “else” part is executed when the above does not apply.

Python is an intuitive language so that we can easily confirm the conditions of the above example. From the following example, you can easily understand that if statement branches according to True or False.

"apple" in words  # words = ["orange", "grape", "peach"]
>>  False

"orange" in words  # words = ["orange", "grape", "peach"]
>>  True

x > 0  # x = 5
>>  True

for loop

Repeat the same operation. We use “for loop” in such a case. Here, let’s see the example of printing numbers from 0 to 4.

for i in range(5):
  print(i)

>>  0
>>  1
>>  2
>>  3
>>  4

In the “for loop”, the variable “i” changes in order from 0 to 4.

Note that, in Python, an index starts from 0. And, “range(5)” indicated the five consecutive integers starting from 0. Then, when you would like to print the values from 1 to 5, you should set them as follows.

for i in range(1, 6):
  print(i)

>>  1
>>  2
>>  3
>>  4
>>  5

If you want to skip one, the code is below.

for i in range(1, 6, 2):
  print(i)

>>  1
>>  3
>>  5

It is also possible to output the elements of the list storing the strings in order.

for name in words:  # words = ["orange", "grape", "peach"]
  print(name)

>>  orange
>>  grape
>>  peach

function

A function is like a converter. For example, we can imagine the typical mathematical function, $f(x) = x^{2}$. The $f(x)$ converts the input $x$ into the output $x^{2}$, whose image is below. And, let’s see the code of this image.

def func(x):
  return x*x

print( func(5) )

>>  25

We defined the function by “def func(x):”, whose name is “func”. And, “x” is an argument, namely an input. And, “return x*x” means this function returns the value of $x^{2}$. Of course, the return value is not always necessary. In the following example, Python just executes the command written in the function “func_print()”.

def func_print():
  print("This is a function")
  print("without a returned value.")

func_print()

>>  This is a function
>>  without a returned value.

Why we use a function? This is because we can understand a code more readable. For example, if we write a set of codes as a function of $f(x)=x^{2}$, we can recognize it as mathematical calculus. In other words, by using a function, the blueprint of your code becomes clearer.

Let’s see the following example for the above description. Here, we consider converting $x$, which will vary from 1 to 10 in order, with the function $f(x)$. Note that $f(x)$ changes dependent on the range of $x$, whose conditional branching is shown in the figure below.

First, we will see the sample code without function.

for x in range(1, 11):
  if x < 4:   # x < 4
    y = x
  elif x < 8: # 4 <= x < 8
    y = x*x
  else:
    y = 10*x  # 9 <= x
  print("x=", x, "was converted into y=", y)

>>  x= 1 was converted into y= 1
>>  x= 2 was converted into y= 2
>>  x= 3 was converted into y= 3
>>  x= 4 was converted into y= 16
>>  x= 5 was converted into y= 25
>>  x= 6 was converted into y= 36
>>  x= 7 was converted into y= 49
>>  x= 8 was converted into y= 80
>>  x= 9 was converted into y= 90
>>  x= 10 was converted into y= 100

Next, the sample code with function is below. We will see that the above code becomes more readable. Especially, it should be easier to understand what is going on in the for loop.

"""
    y = f(x)
"""
def func(x):
  if x < 4:   # x < 4
    return x
  elif x < 8: # 4 <= x < 8
    return x*x
  else:
    return 10*x  # 9 <= x


for x in range(1, 11):
  y = func(x)
  print("x=", x, "was converted into y=", y)

>>  x= 1 was converted into y= 1
>>  x= 2 was converted into y= 2
>>  x= 3 was converted into y= 3
>>  x= 4 was converted into y= 16
>>  x= 5 was converted into y= 25
>>  x= 6 was converted into y= 36
>>  x= 7 was converted into y= 49
>>  x= 8 was converted into y= 80
>>  x= 9 was converted into y= 90
>>  x= 10 was converted into y= 100

In programming, it is very important to write a script with a combination of smaller functional codes. This will make your code easier to read and maintain. In other words, the clearer the blueprint of the code, the better the code.

One more thing you need to know about a function is a local or global variable. A local variable can be accessed just in a function. On the other hand, a global variable can be accessed anytime. Besides, the information of a variable defined in a function is lost after exiting the function.

Let’s understand this explanation through an example. Try to understand how the value of “a” changes with each step.

a = 5
##-- Step 1
print("Step 1: a =", a)
>>  Step 1: a = 5


def func():
  a = 10
  ##-- Step 2
  print("Step 2: a =", a)
 
func()
>>  Step 2: a = 10


##-- Step 3
print("Step 3: a =", a)
>>  Step 3: a = 5

The above example indicates that the variables of “a” are different between outside and inside of the function “func()”.

Summary

Until here, the core syntax in Python has been introduced. In particular, the main topics (if statement, for loop, and function) are often used, so even beginners of Python should master them.

In the next post, we will try to learn more complex contents.

Python is free and one of the most popular programming languages. You can find that python is simple, flexible, readable, and with rich functions. Python can use external libraries so that we can utilize it in a wide range of fields. Especially, the most popular field is in machine learning and deep learning.

Through some posts, we will see the basics of python from scratch. Let’s take a look at the basic concepts and grammar, keeping in mind the use in data science. Python runs a script line by line. So Python will tell you where you went wrong. So, don’t be afraid to make a mistake. Rather, you can make more and more mistakes and write the code from small stacks with small modifications.

In this post, we will learn the following contents.

• Variables
• Comment “#”
• Arithmetic operations
• Boolean
• Comparison operator
• List

Learning programming is boring as it is basic. However, creative working awaits beyond that !!

Note) The sample code in this post is supposed to run on Jupyter Notebook.

Variables

A variable is like a box for storing information. We assign a name to a variable, such as “x” and “y”. And, we put a value or a string into the variable. Let’s take a glance.

We first put “1” into “x“. And, check the content by the “print()“ function. The “print()” function displays the content on your monitor.

x = 1
print(x)

>>  1

Similarly, strings can also be stored in variables. Strings are represented by ” “.

y = "This is a test."
print(y)

>>  This is a test.

Variables have information of data type, e.g. “int”, “float”, and “str”. “int” is for integer and “float” are for decimal. And, “str” is for a string. Let’s check it by the “type()“ function.

type(x)  # x = 1
type(y)  # y = "This is a test."

>>  int
>>  str

z = 1.5
type(z)

>>  float

Python has the functions to convert variables into “int”, “float”, or “str” types. These functions are “int()”, “float()”, and “str()”.

a = 10

type( int(a) )
>>  int

type( float(a) )
>>  float

type( str(a) )
>>  str

Here, some basic rules for variable name are introduced. It’s okay if you know the following, but see the official Python documentation for more details.

Rules of variable name

• It is case sensitive.
• Only the alphabet, numbers, and “_” can be used.
• The initial letter starts with a number.

For example, “abc” and “ABC” are distinguished. “number_6” is OK, but “number-6” is prohibited. The frequent mistake is including “-“, where “-“ is prohibited. Besides, “5_abc” is also prohibited. The initial letter must be numbers or “_”. However, some variables starting from “_” are for Python itself so that you can’t use them. Therefore, the author highly recommends that variables start from an alphabet, especially for beginners.

Comment “#”

Note that “#” is for comments which are NOT executed by python. Python ignores the contents after “#”. For example as the following one, python regards “print(“Hello world!”)” as a code. The other contents, such as “This is for a tutorial of Python.” and “print the first message!”, are regarded as a comment.

# This is for a tutorial of Python.
print("Hello world!")  # print the first message!

There is another method for comments. Python recognizes as a comment when the statement is between “”” and “””. In the following example, the sentence of “This is a comment.” is skipped by Python.

"""
   This is a comment.
"""
print("test")

>>  test

A comment is so important because we forget what the code we wrote in the past was for. Therefore, programmers leave the explanations and their thought as the comments. Concise and clear. That is very important.

Arithmetic operations

Here, let’s see simple examples of arithmetic operations. Complex calculations are just constructed by simple arithmetic operations. So, most programmers do NOT treat complex mathematics but utilize combinations of simple operations.

2 + 3  # addition
>>  5

9 - 3  # subtraction
>>  6

5 * 20  # multiplication
>>  100

100 / 20  # division
>>  5

2**3  # power
>>  8

Boolean

Boolean is just True or False. It is, however, important because we construct an algorithm by controlling True or False. For example, we assume the following situation. When an examination score is over 70, it is passed. On the other hand, when the score is less than 70, it is not passed. To control the code by syntax, a boolean is needed.

The concept of a boolean may be unfamiliar to beginners, however, python tells us intuitively. The example below is that the result of judging x > 0(or x < 0) is assigned to the variable “boolean”.

x = 1   # Assign 1 to the variable "x"

boolean = x > 0
print(boolean)

>>  True

boolean = x < 0
print(boolean)

>>  False

There is no problem with the above code. However, the author recommends the following style due to readability. In the following style, we can clearly understand what is assigned to the variable “boolean“.

boolean = (x > 0)

Comparison operator

Here, let me Introduce comparison operators related to Boolean. We have already seen examples such as “<” and “>”. The typical ones we use frequently are listed below.

Examples against numerical values.

1 > 2
>>  False

5 < 9
>>  True

5 >= 5
>>  True

10 <= 8
>>  False

10 == 10
>>  True

10 != 10
>>  False

Examples against strings. Recall that Python is case sensitive.

"apple" == "APPLE"
>>  False

"apple" != "APPLE"
>>  True

List

List is for a data structure, which has a sequence of data. For example, 1, 2, 3,.., we can treat as a group by a list. The list is represented by “[]”, and let’s see the example.

A = [1,  2,  3]
print(A)

>>  [1, 2, 3]

“A” is the list, which stores the array of 1, 2, and 3. Each element can be appointed by the index, e.g. A[0]. Note that, in Python, an index starts from 0.

A[0]

>>  1

It is easy to replace, add or delete any elements. This property is called “mutable”.

Let’s replace the 2 in A[1] with 9. You will see that A[1] will be rewritten from 2 to 9.

A[1] = 5  # A = [1,  2,  3]
print(A)

>>  [1, 5, 3]

We can easily add a new element to the end of the list by the “.append()” method. Let’s add 99 to the list “A”.

A.append(99)  # A = [1,  5,  3]
print(A)

>>  [1,  5,  3,  99]

Of course, it is easy to delete an element. We do it by the “del” keyword. Let’s delete element 3 in A[2]. We will see that the list “A” will change from “[1, 5, 3, 99]” to “[1, 5, 99]”.

del A[2]  # A = [1,  5,  3,  99]
print(A)

>>  [1, 5, 99]

One more point, the list can handle numerical variables and strings together. Let’s add the string “apple” to the list “A”.

A.append("apple")
print(A)

>>  [1, 5, 99, 'apple']

Actually, this is the thing, where python is different from other famous programming languages such as C(C++) and Fortran. If you know such languages, please imagine that you first declare a variable name and its type. Namely, an array(“list” in Python) can treat only a sequence of data whose variable type is the same.

This is one of the reasons that Python is said to be a flexible language. Of course, there are some disadvantages. One is that the processing speed is slower. Therefore, when performing only numerical operations, you should use a so-called NumPy array, which handles only arrays composed of the same type of variables. NumPy is introduced in another post.

We can easily get the length of a list by the “len()” function.

len(A)  # A = [1, 5, 99, 'apple']

>>  4

Finally, the author would like to introduce one frequent mistake. Let me generate a new list, which is the same for the list A.

B = A  # A = [1, 5, 99, 'apple']
print(B)

>>  [1, 5, 99, 'apple']

B has the same elements as A. Next, let’s replace the one element of B.

B[3] = "orange"  # B[3] = "apple"
print(B)

>>  [1, 5, 99, 'orange']

We have confirmed that the element of B[3] is replaced from “apple” to “orange”. Then, let’s check A too.

A

>>  [1, 5, 99, 'orange']

Surprisingly, A is also changed! Actually, B was generated by “B = A” so that B refers to the same domain of memory as A. We can avoid the above mistake with the code below.

A = [1, 5, 99, 'apple']
##-- Generate the list "B"
B = A.copy()
B[3] = "orange"  # B[3] = "apple"

print(A) # Check the original list "A"

>>  [1, 5, 99, 'apple']

We have confirmed that the list A was NOT changed. When we copy the list, we have to use the “.copy()” method.

Summary

We have seen the basics of Python with the sample codes. The author would be glad if the reader feels Python is so flexible and intuitive. Python must be a good choice for beginners.

In the next post, we will see the other contents for learning the basics of Python.