Get links to all articles from site with python
Hi, everybody! In this post will tell you about one small python script. Recently I have met task to get all links to articles from my blog. I have made some Google search and find some points to this task. I used python and its package BeautifulSoup for html parsing. Also used re module and urllib3 for http request. Here is it, my script. I will be happy if you find it usefull. from bs4 import BeautifulSoup import urllib3 import re links = [] site = "https://alextech18.blogspot.com" pattern = "://alextech18.blogspot.com" def getLinks(url, pattern, html_only=True): http = urllib3.PoolManager() html_page = http.request('GET', url) soup = BeautifulSoup(html_page.data, features="html.parser") # Use separate pattern, because, blogspot for example, can use links leading with http and https for link in soup.findAll('a', attrs={'href': re.compile(pattern)}): if link.get('href') not in links...