Python on Wanzi's Blog – Notes on life, tech, and DevOps.

Generate a Sitemap with a Python Script

wzdushu@gmail.com (wanzi) — Sun, 10 Mar 2024 10:22:42 +0800

Recently, I was helping a friend with a CMS cluster and found that the CMS did not have a sitemap feature. Since Python 3 was already installed on the server, I decided to temporarily generate a sitemap using a script. Here is a record of it.

Script Content:

#!/usr/bin/env python

import datetime
import mysql.connector
import xml.etree.cElementTree as ET
from lxml import etree

# Database connection parameters
config = {
    'host': '10.80.0.3', # For example: '192.168.1.100'
    'user': 'wnote_r',
    'password': 'Wnote#Pss2024',
    'database': 'wnote',
    'raise_on_warnings': True
}

# Sitemap path
sdir = '/opt/wwwroot/'

def mselect(site, n1, n2):
    # Connect to the database
    cnx = mysql.connector.connect(**config)
    cursor = cnx.cursor()

    # Get article IDs
    article_ids = []
    tag_ids = []

    # Query all records in the article table
    query1 = "select id from article order by newstime desc  limit {};".format(n1)
    cursor.execute(query1)
    for row in cursor:
        article_ids.append(row[0])

    # Query all tags
    query2 = "select id from phome_ecms_book order by newstime desc limit {};".format(n2)
    cursor.execute(query2)
    for row in cursor:
        tag_ids.append(row[0])

    # Generate URL lists
    article_urls = [f"https://{site}/article/{id}.html" for id in article_ids]
    tag_urls = [f"https://{site}/tags/{id}.html" for id in tag_ids]

    # Merge URL lists
    urls = article_urls + tag_urls

    # Create the XML tree structure for the sitemap
    root = ET.Element('urlset',
                  {'xmlns': 'http://www.sitemaps.org/schemas/sitemap/0.9',
                   'xmlns:xsi': 'http://www.w3.org/2001/XMLSchema-instance',
                   'xmlns:mobile': 'http://www.baidu.com/schemas/sitemap-mobile/1/',
                   'xsi:schemaLocation': 'http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd'})

    # Generate the sitemap file and build URL elements
    for url in urls:
        s_url = ET.SubElement(root, 'url')
        s_loc = ET.SubElement(s_url, 'loc')
        s_loc.text = url
        s_lastmod = ET.SubElement(s_url, 'lastmod')
        s_lastmod.text = datetime.datetime.now().strftime('%Y-%m-%dT%H:%M:%S+08:00')
        s_changefreq = ET.SubElement(s_url, 'changefreq')
        s_changefreq.text = 'always'
        s_priority = ET.SubElement(s_url, 'priority')
        s_priority.text = '0.95'

    # Generate the XML string
    sitemap_str = ET.tostring(root, encoding='unicode')

    # Save to file
    with open(spfile, "w", encoding="utf-8") as f:
        f.write(sitemap_str)

    # Close the database connection
    cursor.close()
    cnx.close()

with open('sites.list', 'r') as files:
    for tmp in files:
        site = tmp.strip().split()[0]
        spfile = sdir + site + '/' + tmp.strip().split()[1]
        n1 = tmp.strip().split()[2]
        n2 = tmp.strip().split()[3]
        print(site, spfile)
        mselect(site, n1, n2)

The content of sites.list is:

1
2

www.xxxxx.com 100 120
www.xxxxx.com 200 300

Summary

Writing a Python script is quite convenient. However, if performance is a concern, Golang might be a better choice.

A Powerful Tool for Operating Baidu Cloud on Linux

wzdushu@gmail.com (wanzi) — Mon, 04 Dec 2023 16:22:42 +0800

Recently, I was performing a private deployment of our company’s SAAS product at the customer site. Since the customer’s network cannot access the internet, data can only be transferred through a Linux jump server. In addition to the k8s offline deployment program and images, there are also several hundred GB of pre-cut video data, which can only be transferred via Baidu Netdisk. I wondered if it was possible to synchronize Baidu Netdisk data via the command line. A quick Google search revealed that it is indeed possible. Below, I will briefly introduce the use of bypy.

Introduction to bypy

bypy is a Python client for Baidu Cloud/Baidu Netdisk, mainly used for operating Baidu Netdisk under Linux. It provides functionalities such as file listing, downloading, uploading, comparison, upsync, and downsync. Key features include support for Unicode/Chinese, retry on failure, recursive upload/download, directory comparison, and hash caching.

Due to Baidu PCS API permission restrictions, the program can only access files and directories under the /apps/bypy directory in Baidu Cloud.

Installing bypy

Install pip

curl -O https://bootstrap.pypa.io/pip/2.7/get-pip.py
curl -O https://bootstrap.pypa.io/pip/3.6/get-pip.py
python get-pip.py

Install bypy

pip install bypy

python -m bypy info # Generate an authorization code, then open a browser to authorize and enter the authorization code

Copy and paste the authorization code into the terminal, press Enter, and you’re done.

Detailed bypy Commands

$ bypy help command # View command help
$ bypy info  # View Baidu Netdisk space
Quota: 13.294TB
Used: 1.917TB
$ bypy list # View "My Application Data" in Baidu Netdisk
/apps/bypy ($t $f $s $m $d):
D course 0 2023-12-04, 15:21:41
D k8s 0 2023-12-04, 15:47:47

$ bypy downdir k8s # Directly download the k8s directory under bypy
$ bypy syncup # Sync the current directory to the cloud
$ bypy upload # Sync the current directory to the cloud
$ bypy syncdown # Sync the cloud content to the current directory
$ bypy downdir / # Sync the cloud content to the current directory

$ bypy compare # Compare the local current directory with the root directory of the cloud (program)
$ bypy -v # Adding the -v parameter during runtime will display progress details.
$ bypy -d # Adding -d during runtime will display some debug information
$ bypy -ddd # Display more HTTP communication information

Summary

This concludes my exploration of using bypy. During my research, I found many tools based on baidupcs, but most of these projects are no longer maintained. I hope bypy can continue to be supported.

Reference: https://github.com/houtianze/bypy

Python Coding Standards

wzdushu@gmail.com (wanzi) — Sat, 20 Jan 2018 10:22:42 +0800

Encoding Declaration

Default encoding is not ASCII. You can specify it using:

`1`	`# coding=<encoding name>`

1
2

#!/usr/bin/python
# -*- coding: <encoding name> -*-

1
2

#!/usr/bin/python
# coding: utf-8

Code Layout

Use four spaces for indentation; avoid using tabs, and never mix tabs with spaces.
Limit line length to 79 characters. Use backslashes for line continuation, but prefer parentheses instead.
Leave two blank lines between classes and top-level functions; leave one blank line between methods within a class.
Module structure order: module description and docstring, import statements, global variables, constants, others. Within import, arrange in order: standard library, third-party, local modules. Avoid writing multiple imports on one line.
Add one space on both sides of operators. Omit spaces around assignment operators in default function arguments. Do not add spaces before closing brackets (], }, )).
Comments must be in English, preferably complete sentences. Start with a capital letter, end with a period (or other punctuation), followed by two spaces before the next sentence. For phrases, punctuation may be omitted.
Always write docstrings for public modules, functions, classes, and methods. Non-public ones don’t require docstrings, but comments are acceptable (placed on the line immediately after def).

Naming Conventions

Avoid using single lowercase letter l or uppercase O as they can easily be confused.
Module names should be short and use all lowercase letters, optionally separated by underscores.
Package names should be short and use all lowercase letters without underscores.
Class names use CapWords. Internal classes use _CapWords.
Exception names use CapWords with an Error suffix.
Global variables should ideally be limited to module scope (like static in C). Two ways to achieve this: use __all__ mechanism or prefix with an underscore.
Function names use all lowercase letters, optionally separated by underscores.
Constant names use all uppercase letters, optionally separated by underscores.
Class attributes (methods and variables) use all lowercase letters, optionally separated by underscores.
Class attributes have three scopes: public, non-public, and subclass API—similar to public, private, protected in C++. Prefix non-public attributes with a single underscore.
If a class attribute conflicts with a keyword, append an underscore. Avoid abbreviations or other workarounds.
To prevent naming conflicts with subclasses, prefix certain class attributes with double underscores. For example, in class Foo, define __a; access via Foo._Foo__a to avoid ambiguity. If a subclass also uses Foo, this protection fails.
The first parameter of instance methods must be self. The first parameter of static methods must be cls.

Code Checking Tools

`1`	`pip install flake8`

After installing flake8 successfully, open VSCode, go to File → Preferences → Settings, and add "python.linting.flake8Enabled": true in settings.json.

Auto Formatting Tool

`1`	`pip install yapf`

After installing yapf successfully, open VSCode, go to File → Preferences → Settings, and add "python.formatting.provider": "yapf" in settings.json.

Project Development Notes:

When starting a new project, keep the following in mind:

README.md: Write your project overview, quick start guide, etc. Although distutils requires no extension, GitHub renders .md files directly as HTML.
ChangeLog.txt: Store version change logs with a consistent format. Refer to web.py’s ChangeLog.txt for examples.
LICENSE.txt: Include the license used by your project. Do not create custom licenses.
requirements.txt: List dependencies on separate lines. This allows pip install to automatically install them.
setup.py: Installation script — detailed later.
docs: Store project documentation such as high-level design, detailed design, maintenance documents, and auto-generated pydoc content. Strongly recommend using Markdown format.
src: Place main module code here. Avoid placing module directories directly in the root directory. Module paths can be specified in setup.py.
tests: Store all unit tests and performance test scripts. Test files must start with test_ so that distutils will automatically include them in packaging. Run tests using:
python -m unittest discover -s ./ -p 'test_*.py' -v

References:

https://www.python.org/dev/peps/pep-0008/ https://wiki.woodpecker.org.cn/moin/PythonCodingRule