Automating Clean Code Checks

Using Git Hooks to Simplify Coding Checks

Ensuring that code is clean and well-organized is essential for the success of any software development project. This is because clear and maintainable code is easier to understand, modify, and work with. One way to promote clean code practices is by using automation tools like pre-commit to check for issues before code is committed to the repository. Pre-commit runs checks on code changes before they are added to the repository, helping to ensure that code adheres to established standards and practices.

This article will continue and expand upon the previous article that has been provided for your reference.

Installing Clean Code Best Practices for Data Engineering Repositories

Using

Daniel Beach

’s repository, we will automate standard checks using the pre-commit package and build our own custom checks to improve the quality of your code.

Installing pre-commit

Pre-commit hooks can be used to automatically check for various issues in the code, such as syntax errors, linting issues, and code formatting issues. This helps ensure that the code being committed is of high quality and follows the project's coding standards.

We can install pre-commit package using pip or brew.

pip install pre-commit

or

brew install pre-commit

Once the framework is installed, you will need to create a configuration file, .pre-commit-config.yaml, that defines the checks that should be run before each commit.

Here is our example file

repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.2.0
    hooks:
    -   id: check-yaml
    -   id: end-of-file-fixer
    -   id: trailing-whitespace
-   repo: https://github.com/pre-commit/mirrors-mypy
    rev: 'v0.942'  # Use the sha you want to point at
    hooks:
    -   id: mypy
-   repo: https://github.com/pre-commit/mirrors-pylint
    rev: ''  # Use the sha / tag you want to point at
    hooks:
    -   id: pylint

It is worth noting that this configuration will automatically remove trailing whitespaces to your code. Whether or not you want to use this feature is a matter of personal or team preference, which is why it is important to have a conversation with your team members about this issue.

Here are some other preference hooks:

• double-quote-string-fixer will replace double quotes with single quotes.

• end-of-file-fixer adds an extra line if there is code in the file or verifies that the file is empty.

• pydocstyle checks for compliance with Python docstring conventions.

• trailing-whitespace will remove trailing whitespace.

Now we can run it.

pre-commit            
[WARNING] The 'rev' field of repo 'https://github.com/pre-commit/mirrors-pylint' appears to be a mutable reference (moving tag / branch).  Mutable references are never updated after first install and are not supported.  See https://pre-commit.com/#using-the-latest-version-for-a-repository for more details.  Hint: `pre-commit autoupdate` often fixes this.
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pre-commit/mirrors-pylint.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
check yaml...........................................(no files to check)Skipped
fix end of files.....................................(no files to check)Skipped
trim trailing whitespace.............................(no files to check)Skipped
mypy.................................................(no files to check)Skipped
pylint...............................................(no files to check)Skipped

I chose only python checks, and none of these folders have python files. The check are skipped.

Custom Hooks

You can add your own custom hooks by adding scripts to a folder in the repo and copying them over to pre-commit folder in .git.

In the pre-commit hook (under git-hooks/hooks/), only two checks will be ran and are located in the same file.

1. Using main or master branch to commit.

   current_branch=`git rev-parse --abbrev-ref HEAD`
   if [[ $current_branch =~ master|main ]]; then
       message="Please don't commit directly to $current_branch."
       echo -e "\033[1;31mERROR: $message\033[0m";
       exit 1
   fi

2. We also will run black, a commonly used Python code formatter.

   black -t py39 --check $repo_dir

Adding this line to the pre-commit hook file will add the already set up the other python checks from the pre-commit package.

pre-commit $repo_dir

We will first need to copy over the newly created hooks to the hooks folder under .git/hooks

sh git-hooks/copy_hooks.sh

We can now test these hooks.

First, we will need to make a change in any of the files and add them to local repo on the main branch.

git add .

Commit the file.

git commit -m "build: add custom hook"   
ERROR: Please don't commit directly to main.

And there is your error!

You can add the change to a new branch:

git switch -c hooks

-c stands for create and hooks is the branch’s name.

Try again.

git commit -m "build: add instructions for hooks" 
Info: [Policy] Checking code in /Users/silvergenova/Documents/GitHub/deequ-examples/dataEngineeringTemplate with black...
[WARNING] Unstaged files detected.
[INFO] Stashing unstaged files to /Users/silvergenova/.cache/pre-commit/patch1672552212-60525.
[WARNING] The 'rev' field of repo 'https://github.com/pre-commit/mirrors-pylint' appears to be a mutable reference (moving tag / branch).  Mutable references are never updated after first install and are not supported.  See https://pre-commit.com/#using-the-latest-version-for-a-repository for more details.  Hint: `pre-commit autoupdate` often fixes this.
check yaml...........................................(no files to check)Skipped
fix end of files.....................................(no files to check)Skipped
trim trailing whitespace.............................(no files to check)Skipped
mypy.................................................(no files to check)Skipped
pylint...............................................(no files to check)Skipped
[INFO] Restored changes from /Users/silvergenova/.cache/pre-commit/patch1672552212-60525.
.git/hooks/pre-commit: line 18: black: command not found
OK: [Policy] Passed black checking.
On branch hooks
Your branch is up to date with 'origin/hooks'.

You are all set now.

Final Thoughts

Using a pre-commit hook can help ensure that your code is clean and follows best practices, making it easier for you and your team to work on and maintain the codebase. It is a simple but effective way to improve the quality of your code and make your development workflow more efficient.