Web Scraping with Python

The main modules which are needed for web related operations are urllib, urlopen, requests, selenium. Other than this four there are other main and supporting modules like beautifulsoup, webbrowser are there. so, let's start with urllib  -

Follow the below code closely -

import urllib
ts=urllib.urlopen("http://pythonbeginner.in")
for textscript in ts:
    print textscript

This code first fetches the whole content of a website and display them line wise. As simple like that. In web scraping, we have to find a given information in a web page. The projects I have got are mostly something like  -  go through the website on a regular basis and update the price of a product accordingly. Or, it can be useful in share market auto trading etc. 

Suppose, we have to find how many times the word 'python' is used in the code. Then what will we do? Web scrapers often face problems like this. Let's look at the second code - 

import urllib,re

ts=urllib.urlopen("http://pythonbeginner.in")

count=0

for line in ts:

    if re.search('python',line):

        print line

        count+=1

print count

Remember the codes from the entry Regular Expression. Here I used the same code to find out the number of times the word 'python' occurs. It found out that I have used it 38 times on the website main page😮.

You can open any page in your default browser with the module webbrowser. Look at the example -

import webbrowser
webbrowser.open("www.yahoo.com")

But for dynamic web pages where the pages are created by ajax, javascript or any other run-time programs then the web scraping becomes difficult. As in the above example, we are getting a static web page, not a dynamic one. To get dynamic values we need to use selenium. I will discuss it in the next blog entry. Wait for it.

About Python

Python is a multi function programming language

- this means it can be used to virtually every program. So, in this blog we will try to know Python and how to apply the language to modern day life. But first of all we need to understand the language.

There are many tutorials available in the web. You must visit those tutorials and here is my version of the tutorial. You can learn from here too, if you bear with me.

You can also learn python from sites like -

• docs.python.org
• tutorialspoint
• Codecademy
Or you can use my own site -
• Pythonbeginner

Install Python:

Install Python by going to the site - www.python.org and download the python 2.7  or 3. Here I am concentrating on Python 2.7. So it is advisable to download the later one.
For windows you have to install Python actually, Python is already installed in Linux or Mac. Just extract the downloaded file and double click setup. exe.

How to check python is  installed properly:
For Linux just open the terminal and type Python. Of out shows a reply message with version on it and a '>>' sign, then you are OK.
For windows, open command prompt and type Python.. That's it. If there is an error like command not found, then set the part variable properly or see if you are in proper directory.

After installation of the python from the instruction above you are ready to rock in the python world. You need to download some modules for required project when needed. You can get all the module in github so no worries in this respect.
Why python is Multi-function Programming Language?
Python can be used in many aspects of our every programming needs- from core programming options where we program different hardware to sophisticated web development , we can use python. Python is in its core is made to do tasks without thinking much about syntax of the programming, and concentrate on the semantics of the program. We will start from our very next blog - let's start by saying 'Hello World'.

Python Tkinter Programming

Desktop GUI application creation in an easy mode is one of the specialty of the python programming. Python 2.7 and Python 3 both have an inbuilt class tkinter which is needed to program graphical interface in python. There are other modules as well but here we will focus only on tkinter.

A basic tkinter program to show 'Hello World' in GUI mode -

from Tkinter import *
# create a root window
root = Tk()
root.title("Lazy Buttons")
root.geometry("400x120")
lbl=Label(root,text="Hello World").pack()
root.mainloop()

If this shows error, change the Tkinter to tkinter in the import statement. After importing the module we create the root window by defining is as the Tk() object. In the next statement we named the window as "Lazy Buttons" by calling the root.title() method.
root.geometry() is used to define the width and height of the window. It has a formal notation like root.geometry("w x h+ x+ y") where w and h are for width and height respectively and the x and y are the coordinates of the screen.

In the next statement we created a Label, which will show the "Hello World" texts to be displayed  in the root window. The Label creator method takes the window name as input i.e. root and the text as input.The pack() method is needed to pack the label to the root window. The mainloop is needed to make the window to exist forever until you click the cross button.

Buy One Plus Two without Registration

Hello World with Python

Creating programs to print 'hello world' is a general concept, so I will start with that. Unlike any language where you have to call library to print the statement, Python has nothing Luke that.
So, for 'hello world' program -
If you are in Linux -
1. Open terminal and type Python.
         $python
2. Then just write print "hello world"
        >>print "hello world "
3. It will show the output "hello world"

For windows it is even more simpler,
Open the Python idle and type the above said statement. The out put will be the same.
But we need something more than "hello world" !

Open the terminal and write Python again. Write:  >>5+4
>>9
You can multiply,divide and do all the basic operations on the Python console.
You can also have variables.
>>a=5
>>b=4
>>c=a+b
>>9
You can see the output of c. The program is simply assign values to a and b variables and their addition in variable c. Python is smart enough to guess the  types of variables by itself.