Home

Arduino Command Line Interface (CLI)

Description

Build and run arduino sketches from the command line.

Installation

Homebrew:

brew install arduino-cli

Setup

Website: Getting Started

arduino-cli help core # Help page
# Create config file to store commands so i dont have to type them in the terminal
arduino-cli config init # Create yaml file 
arduino-cli sketch new SKETCH_NAME # Create new sketch in current dir
arduino-cli core update-index # Update cli with latest libraries and boards
arduino-cli board list # List connected devices (core shows the device i need to install)
arduino-cli core install arduino:samd # Install boards (change arduino:samd for core value)
arduino-cli core list # List all installed boards
arduino-cli lib search SEARCH_TERM # search for a library
arduino-cli lib install LIBRARY # Install a lib from the searched items

If I want to install 3^rd party boards see the above link.

Run

arduino-cli core update-index
arduino-cli sketch new FILENAME # Folder must be same name as file
arduino-cli board list # Get name of board and FQBN (Fully qualified board name)
arduino-cli compile --fqbn arduino:avr:nano FILENAME # replace arduino bit with FQBN
arduino-cli upload -p YourBoardPort --fqbn YourBoardFQBN YourSketchName # Upload

Probably best to create a bash executable for each project / board so you can run all these processes in the background.

LoRaWAN Libraries

Low power wide area network protocol (LoRaWAN) is used to send long range messages between nodes and clients. These messages are quite small ~51 bytes, but with proper packaging can provide sufficient information for capturing remote sensor data.

Setup

MCCI-lmic - arduino-lmic

Fairly low-level library that requires some configuration to ensure the right region is selected before integrating into your project.

Documentation: arduino-lmic

Installation

To install navigate to the Arduino Library Manager and search for "MCCI-limc".

Alternatively, download the repository as a zip and unpack it in your Arduino/Libraries folder.

Before using, navigate to lmic_project_config.h (in Arduino/Libraries/MCCI-lmic folder) and comment out regions except for your specific region.

//#define CFG_eu868 1
//#define CFG_us915 1
#define CFG_au915 1
//#define CFG_as923 1
//#define LMIC_COUNTRY_CODE LMIC_COUNTRY_CODE_JP	/* for as923-JP */
//#define CFG_kr920 1
//#define CFG_in866 1

Alternative (arduino-lorawan)

Higher level more "arduinoesque" type library.

Documentation: arduino-lorawan

Other good resources

Node-Red with LoRaWAN Node Adafruit LoRaWAN tutorial

PlatformIO

Description

Platform IO allows for easy uploading and viewing of microcontroller input and outputs. This information covers the command line interface for PlatformIO.

Website: PlatformIO

Install

Curl (requires python):

curl -fsSL https://raw.githubusercontent.com/platformio/platformio-core-installer/master/get-platformio.py -o get-platformio.py
python3 get-platformio.py

Add given path to PATH variables. If confused see this link.

Homebrew (MacOS & Linux):

brew install platformio

Setup

Website: User Guide

Pretty sure platformio-cli requires the main file name to be the same as the folder name.

pio project init # Init project in current dir with folders and resources

Edit the .yml file with project specific configuration.

[env:teensy41]
platform = teensy
board = teensy41
framework = arduino

board_build.mcu = imxrt1062

lib_deps =
milesburton/DallasTemperature @^3.9.1

upload_protocol = teensy-cli

platform_packages = 
toolchain-gccarmnoneeabi@~1.90301.200702
tool-teensy@https://github.com/maxgerhardt/pio-tool-teensy-arm/archive/master.zip

Execute

pio --help # Display help
pio run # Compile code
pio run -t upload # Run then upload
pio run -e board_name -t upload # Run and upload to specific board

Miscellaneous

Description

Various bash commands that are useful but don't have a specific order that is worthy of their own page. Also contains useful terminal based programs.

Date and Time

cal # Display calendar
cal -3 # Display calendar with next and prev month
date # Display date

System

Display certain system information.

top # Display current processes
system_profiler SPDisplayDataType | grep Resolution # Get sys resolution (MacOS)

Tree

Displays the current folders tree structure in the terminal.

brew install tree # Install
tree # Run to display tree structure

Newsboat

RSS terminal reader for RSS feeds.

brew install newboat # Install
newsboat # Run
vi ~/.newsboat/urls # Add RSS feed URL's
vi ~/.newsboat/config # Add specific configuration

For config resources see this link.

File transfer (for android devices)

Add and remove files from android device on a mac. Run from GUI application after installation.

brew install --cash android-file-transfer # Install

zshrc



# If you come from bash you might have to change your $PATH.
# export PATH=$HOME/bin:/usr/local/bin:$PATH
# Path to your oh-my-zsh installation.
export ZSH="/Users/harveybates/.oh-my-zsh"

# Set name of the theme to load --- if set to "random", it will
# load a random theme each time oh-my-zsh is loaded, in which case,
# to know which specific one was loaded, run: echo $RANDOM_THEME
# See https://github.com/ohmyzsh/ohmyzsh/wiki/Themes
ZSH_THEME="robbyrussell"

# Set list of themes to pick from when loading at random
# Setting this variable when ZSH_THEME=random will cause zsh to load
# a theme from this variable instead of looking in $ZSH/themes/
# If set to an empty array, this variable will have no effect.
# ZSH_THEME_RANDOM_CANDIDATES=( "robbyrussell" "agnoster" )

# Uncomment the following line to use case-sensitive completion.
# CASE_SENSITIVE="true"

# Uncomment the following line to use hyphen-insensitive completion.
# Case-sensitive completion must be off. _ and - will be interchangeable.
# HYPHEN_INSENSITIVE="true"

# Uncomment the following line to disable bi-weekly auto-update checks.
# DISABLE_AUTO_UPDATE="true"

# Uncomment the following line to automatically update without prompting.
# DISABLE_UPDATE_PROMPT="true"

# Uncomment the following line to change how often to auto-update (in days).
# export UPDATE_ZSH_DAYS=13

# Uncomment the following line if pasting URLs and other text is messed up.
# DISABLE_MAGIC_FUNCTIONS="true"

# Uncomment the following line to disable colors in ls.
# DISABLE_LS_COLORS="true"

# Uncomment the following line to disable auto-setting terminal title.
# DISABLE_AUTO_TITLE="true"

# Uncomment the following line to enable command auto-correction.
# ENABLE_CORRECTION="true"

# Uncomment the following line to display red dots whilst waiting for completion.
# Caution: this setting can cause issues with multiline prompts (zsh 5.7.1 and newer seem to work)
# See https://github.com/ohmyzsh/ohmyzsh/issues/5765
# COMPLETION_WAITING_DOTS="true"

# Uncomment the following line if you want to disable marking untracked files
# under VCS as dirty. This makes repository status check for large repositories
# much, much faster.
# DISABLE_UNTRACKED_FILES_DIRTY="true"

# Uncomment the following line if you want to change the command execution time
# stamp shown in the history command output.
# You can set one of the optional three formats:
# "mm/dd/yyyy"|"dd.mm.yyyy"|"yyyy-mm-dd"
# or set a custom format using the strftime function format specifications,
# see 'man strftime' for details.
# HIST_STAMPS="mm/dd/yyyy"

# Would you like to use another custom folder than $ZSH/custom?
# ZSH_CUSTOM=/path/to/new-custom-folder

# Which plugins would you like to load?
# Standard plugins can be found in $ZSH/plugins/
# Custom plugins may be added to $ZSH_CUSTOM/plugins/
# Example format: plugins=(rails git textmate ruby lighthouse)
# Add wisely, as too many plugins slow down shell startup.
plugins=(git)

source $ZSH/oh-my-zsh.sh

# User configuration

# export MANPATH="/usr/local/man:$MANPATH"

# You may need to manually set your language environment
# export LANG=en_US.UTF-8

# Preferred editor for local and remote sessions
# if [[ -n $SSH_CONNECTION ]]; then
#   export EDITOR='vim'
# else
#   export EDITOR='mvim'
# fi

# Compilation flags
# export ARCHFLAGS="-arch x86_64"

# Set personal aliases, overriding those provided by oh-my-zsh libs,
# plugins, and themes. Aliases can be placed here, though oh-my-zsh
# users are encouraged to define aliases within the ZSH_CUSTOM folder.
# For a full list of active aliases, run `alias`.

# Custom configuration
# Use vim keystrokes in terminal
bindkey -v
plugins=(
	  vi-mode
  )

# Farm Decsion TECH
# Run farm decision tech docker
alias fdt="docker run -p 80:80 -v $PWD/var/www:/var/www --security-opt seccomp=unconfined --name zz fdtwebsite"
# Run a shell in farm decision tech docker
alias fdtshell="docker exec -it zz bash"
# Remove verbose and with confirmation
alias rm="rm -i -v"
# Youtube download
alias yt="youtube-dl --add-metadata -ic" # Video
alias yta="youtube-dl --add-metadata -xic" # Audio only

export PATH="/usr/local/opt/openjdk/bin:$PATH"
export ANDROID_HOME=/Users/$USER/Library/Android/sdk
export ANDROID_SDK_ROOT=$ANDROID_HOME
PATH="$PATH:$ANDROID_HOME/emulator:$ANDROID_HOME/tools:$ANDROID_HOME/platform-tools"

export GNUTERM="qt font \"Arial,12\""

Searching in the Terminal

Description

Commands for searching for/within files and folders in the terminal.

Find

Useful for finding files and folders that match your search term. Must include quotations when searching. Full-stop represents the current working directory.

find . -name "search_term" # Search for files matching *search_term*
find . -name "*.cpp" # Search for files matching widcard - extension

Search within files with grep

Use grep to search for occurrences of words or characters within files and folders.

grep "search_term" "filename" # Search for occurances of search_term in a given file
grep "search_term" * # Search all files in dir for occurances
grep -E "word1|word2" * # Search for multiple search terms
grep -rl "search_term" /path # Search subriectories and print out filenames with matches

Grep flags

-w # Only display whole words
-i # Non-case sensitive
-r # Search sub-directories as well
-c # Count number of matches in each file
-E # Search for multiple search terms
-x # Only show exact matches

Regex

\ # Escape following character	
. # Any one character
* # Any number of previous character (including none)
+ # Any number of previous character (not including zero)
$ # End of line
^ # Beginning of line
\S # Any non-whitespace character
\s # Any whitespace character
? # Optional
[a-z] or [:lower:] # Match any lowercase character from a to z
[A-Z] or [:upper:] # Match any uppercase character from A to Z
[0-9] or [:digit:] # Match any digit between 0 and 9
[A-Za-z] # Any letter
\n # New line
\r # Carriage return 
\t # Tab
? # Optional

Web Functions in the Terminal

Description

Commands for downloading and interacing with websites and web hosted files.

Curl

Website: Curl

Used in termial to tranfer data over http/s.

curl https://www.example.com # Show html in terminal
curl https://www.example.com -O example.html # Save to html
curl https://www.example.com/filename.cpp -O "filename.cpp" # Save a file
# or
curl -O http://www.example.com/filename.cpp "filename.cpp"

WGet

wget http://www.example.com/filename.cpp # Download file

Ping

Test website, get IP address and echo response.

ping sitename.com # Test site and get public IP address
ping -6 sitename.com # Request IPv6
ping -4 sitename.com # Request IPv4
ping -a sitename.com # Generate sound on successful request

Lync (Web Browser in Terminal)

Lynx is a command line web browser, it works well in most cases.

Website: Lynx

Install (Homebrew):

brew install lynx

Execute

lynx # Run
lynx sitename.com # Run on specific site

CMake Introduction

Description

CMake allows for C++ applications to be compiled without entering library, linkers and files manually each time you want to compile your project.

Installation

Main Site: CMake

Homebrew (MacOS & Linux)

brew install --cask cmake

Setup

Create a file called CMakeLists.txt in your working directory

Define Operating System

if(UNIX AND NOT APPLE)
	... # Linux
elseif(APPLE)
	... # MacOS
else()
	... # Windows
endif()

Define Minimum Version

Documentation: cmake_minimum_required

cmake_minimum_required(VERSION "3.19.2")
# Just use current version you are using

Define Project Name

project("project_name")

Locate Installed Package

find_package(*package_name* REQUIRED)

Set Source Directory

set(NAME_SRC
	src/main.cpp
	src/example.cpp)

Set Include Directory

This is where all your C++ header (.h or .hpp) files will be located

set(NAME_H
	include/main.h
	include/example.h)

Set Executable Path

This is where an executable (.exe) file will be generated.

set(EXE_PATH ${CMAKE_BINARY_DIR}/bin)
# CMAKE_BINARY_DIR represents the directroy where your CMakeLists.txt file is located

Include Directories

include_directories(${CMAKE_SOURCE_DIR}/include ...)

Add Executable

Add components to create an executable.

add_executable(${PROJECT_NAME} ${NAME_SRC} ${NAME_H})

Add Compile Features

Add custom compile features such as the C++ version to be used.

target_compile_features(${PROJECT_NAME} PUBLIC cxx_std_17)

Link Libraries

Link your project to external libraries.

target_link_libraries(${PROJECT_NAME} PRIVATE *library_name* ...)

Target Include Directories

Target - links to specific project

Normal - links globally

target_include_directories(${PROJECT_NAME} PRIVATE *library_name* ...)

Overall Structure

cmake_minimum_required(VERSION "3.19.2")

project("project_name")

find_package(*package_name* REQUIRED)

set(NAME_SRC
	src/main.cpp
	src/example.cpp)

set(NAME_H
	include/main.h
	include/example.h)

set(EXE_PATH ${CMAKE_BINARY_DIR}/bin)

include_directories(${CMAKE_SOURCE_DIR}/include ...)

add_executable(${PROJECT_NAME} ${NAME_SRC} ${NAME_H})

target_compile_features(${PROJECT_NAME} PUBLIC cxx_std_17)

target_link_libraries(${PROJECT_NAME} PRIVATE *library_name* ...)

target_include_directories(${PROJECT_NAME} PRIVATE *library_name* ...)

CPM-Make

Description

Adds dependencies to current machine upon building CMake for the first time. For example, if a specific dependency is needed for the current project, it will be downloaded before compiling your C++ application. This module also allows for linking of said dependency if it is downloaded successfully.

Installation

Main Site: CPM-Make GitHub

Manual Install

To install navigate to this link and download the latest release.

Add CPM.camke to the location where CMakeLists.txt file is located.

Automatic download (with CMake)

CPM-Make can be downloaded directly by CMake. To do this you need to add the following code snippet to the start of your CMakeLists.txt file.

set(CPM_DOWNLOAD_VERSION 0.28.4)
# Set download version to latest release before executing 

if(CPM_SOURCE_CACHE)
	set(CPM_DOWNLOAD_LOCATION "${CPM_SOURCE_CACHE}/cpm/CPM_${CPM_DOWNLOAD_VERSION}.cmake")
elseif(DEFINED ENV{CPM_SOURCE_CACHE})
	set(CPM_DOWNLOAD_LOCATION "$ENV{CPM_SOURCE_CACHE}/cpm/CPM_${CPM_DOWNLOAD_VERSION}.cmake")
else()
	set(CPM_DOWNLOAD_LOCATION "${CMAKE_BINARY_DIR}/cmake/CPM_${CPM_DOWNLOAD_VERSION}.cmake")
endif()

if(NOT (EXISTS ${CPM_DOWNLOAD_LOCATION}))
	message(STATUS "Downloading CPM.cmake to ${CPM_DOWNLOAD_LOCATION}")
	file(DOWNLOAD
			https://github.com/TheLartians/CPM.cmake/releases/download/v${CPM_DOWNLOAD_VERSION}/CPM.cmake
			${CPM_DOWNLOAD_LOCATION}
		)
endif()

include(${CPM_DOWNLOAD_LOCATION})

Add a Package

Add a package to your C++ project. I am not sure if the list of packages is limited to those listed in on this page. Perhaps other packages can be installed this way with the right configuration.

CPMAddPackage(
		NAME # The unique name of the dependency (should be the exported target's name)
		VERSION	# The minimum version of the dependency (optional, defaults to 0)
		OPTIONS	# Configuration options passed to the dependency (optional)
		DOWNLOAD_ONLY # If set, the project is downloaded, but not configured (optional)
		[...]         # Origin parameters forwarded to FetchContent_Declare, see below
		)

Link a Library

Once downloaded and added it needs to be linked to your C++ project.

if(*library_name*_ADDED)
	add_library(*library_name* INTERFACE)
	include_directories(${*library_source_dir}/include}
	target_include_directories(${PROJECT_NAME} INTERFACE ${*library_source_dir}/include)
	target_link_libraries{${PROJECT_NAME} *library_name*}
endif()

Using JSON with C++

Description

JSON formatted data is a great asset for sending and receiving formatted information. In C++ the nlohmann library provides an easy way to create and display JSON formatted data.

Documentation: https://github.com/nlohmann/json

Installation

Homebrew (MacOS & Linux)

brew install nlohmann-json

Linking

Link with CMakeLists.txt (CMake):

find_package(nlohmann_json 3.2.0 REQUIRED)
target_link_libraries(foo PRIVATE nlohmann_json::nlohmann_json)

CPM-Make (Package manager):

Check in releases for the latest version.

You will need the URL specific to that release version and the SHA256 key.

Only get the include folder as we don't want to download all the test examples.

CPMAddPackage(
	NAME nlohmann_json
	VERSION 3.9.0
	# Just get the latest (3.9.0) version, and only the include folder as the rep is big
	URL https://github.com/nlohmann/json/releases/download/v3.9.0/include.zip
	URL_HASH SHA256=5b9b819aed31626aefe2eace23498cafafc1691890556cd36d2a8002f6905009
)

Usage

Initialisation

#include "nlohmann/json.hpp"
using json = nlohmann::json;

Construct JSON Class

json j;

Add values

j["value_name"] = "value"; // Single value
j["value_name1"]["value_name2"] = "value"; // Nested value

Convert string to JSON object

std::string input = "{"happy":true,"pi":3.141}";
j = input;
std::cout << j.dump(4) << std::endl;

Print formatted JSON object

int n = 4;
// n represents an int for the number of spaces to indent
std::cout << j.dump(n) << std::endl;

Example output

{
	"value_name": "value", 
	"value_name1" {
		"value_name2": "value"
	}
}

LLDB Cheatsheet

LLDB is a debugger for C/C++ that runs in the terminal.

(lldb) denotes a function being run in lldb.

Summary

# Opening
c++ -g -std=c++17 -o <output_exe> <file_name> # With debug flags (-g)
lldb main # Open executable main

# Breakpoints
(lldb) b main.cpp : <line_number> # Set breakpoint online number
(lldb) b set -n <function_name> # e.g. Foo(int)
(lldb) br l # List breakpoints
(lldb) br del <line_number> # Delete breakpoint

# Run
(lldb) r

# Watch variable
(lldb) watch set var <variable_name>
(lldb) watch modify -c '(<variable_name> == 5)' # Break on condition
(lldb) watch list

# Navigation
(lldb) n # Next line
(lldb) s # Step into function on current line
(lldb) c # Continue until next breakpoint (or end)

# Variables
(lldb) p <variable_name> # Print variable value
(lldb) fr v # Print all variables in current frame

# Frames
(lldb) fr s # Current point of execution
(lldb) fr v # Print all variables in current frame
(lldb) fr s <frame_id> # Go to specific frame
(lldb) bt # List frames up until point

# Exiting
(lldb) kill # Kill the process (but stay in lldb)
(lldb) quit # Exit lldb

Open File

lldb main # Where main is your executable
# Note main must be compiled with -g flag
# E.g. c++ -g -std=c++17 -o main main.cpp

# or
lldb
(lldb) file "path/to/file"

# or if arguments are supplied 
lldb -- main arg1 arg2 # note the -- before the executable

Breakpoints

Executable must be compiled with the -g flag.

Set

By line number:

(lldb) breakpoint set -f <filename> -l <line_number>
# or
(lldb) b <filename> : <line_number>
# or
(lldb) break <filename> : <line_number>

By function:

(lldb) breakpoint set --name <function_name>
# or
(lldb) b set -n <function_name> # e.g. Foo(int)

On a class method:

(lldb) b Foo::foo()

Inside a namespace:

(lldb) b NameSpace::foo(int)

List

(lldb) breakpoint list
# or
(lldb) br l

Delete

Deletion is based on the breakpoint ID (obtained using br l).

(lldb) br del # All
(lldb) br del <line_number> # Specific

Watch

Watch a specific variable to see it changes (also based on conditionals). Must be running before setting.

(lldb) watch set var <variable_name>
# Stop based on condition
(lldb) watch modify -c '(<variable_name> == 5)'
# List watch
(lldb) watch list

Run

process launch
# or
(lldb) run
# or
(lldb) r

Step Over

Go to next line.

(lldb) next
# or 
(lldb) n

Step Into

Go into function on current line. If multiple (functions) exist it will go into them in order.

(lldb) step
# or
(lldb) s

Continue

Run until next breakpoint or end of program.

(lldb) continue
# or
(lldb) c

Variables

Inspecting

View variable contents.

(lldb) print <variable_name>
# or
(lldb) p <variable_name>

Frames

Inspect all variables around a specific call site.

(lldb) frame <variable_name>
# or
(lldb) fr v

See what frame the debugger is up to. Or select a specific frame.

(lldb) frame select
# or
(lldb) fr s # Current point of execution

# Or to see a specific frame
(lldb) frame select <frame_id>
# or
(lldb) fr s <frame_id>

Back-tracing

Lists the current call stack. Most recent function at the top of the stack (descending order).

(lldb) bt

Useful for debugging crashes by seeing the programs state before the crash occurred, for example:

lldb main
(lldb) r
# Error occurs
(lldb) bt # List frames of execution
(llsb) fr s <frame_id> # Select frame before crash
(llsb) fr v # List varaibles prior to crash

Terminating

# Kill the process (but stay in lldb)
(lldb) kill
# Exit lldb
(lldb) quit

Docker commands

Commands

Containers

Build a container using a Dockerfile in the current directory.

docker build -t <desired_container_name> .

Run a container in the background using port mapping.

docker run -dp 3000:3000 <container_name>

List containers

docker ps # Running only
docker ps -a # All containers
docker ps -s # Disk usage by container

Stop a container

docker stop <container_id>

Remove a container

docker rm <container_id> # Container id is obtained from "docker ps"

Volumes

Persistant storage (db) that wont be removed if the container is stopped. Create a volume

docker volume create <desired_volume_name>

Run a container with a volume

docker run -dp 3000:3000 -v <volume_name>:/etc/<folder_name> <container_name>

Bind mounts

Locations where updates to files can be made without having to rebuild the container. Useful for coding changes locally and having the updates occur in the docker container. Example from: https://docs.docker.com/get-started/06_bind_mounts/

 docker run -dp 3000:3000 \
     -w /app -v "$(pwd):/app" \
     node:12-alpine \
     sh -c "yarn install && yarn run dev"

The -v "$(pwd):/app" controls the current bind directory.

Logs

See log processes:

docker log -f <container_id>

Simple processes

MySQL db

# Create mysql instance
docker run -d \
     --network todo-app --network-alias mysql \
     -v todo-mysql-data:/var/lib/mysql \
     -e MYSQL_ROOT_PASSWORD=<db_password> \
     -e MYSQL_DATABASE=<db_name> \
     mysql:5.7

# Check if working and connect to db
docker exec -it <mysql-container-id> mysql -u root -p

Networking functions

docker run -it --network todo-app nicolaka/netshoot

Version control using Git

Description

Useful commands for using git control system.

Setup

Direct: Install Git

Homebrew (MacOS & Linux):

brew install git

Clone repository

git clone "https://repositoryname.com"

Initialise remote repository

git remote add origin "https://repositoryname.com"
git remote -v # Check if remote origin has been added

Normal workflow

git add .
git commit -m "commit message goes here"
git push origin master

Status

git status # Get information

Branches

git branch "branchname" # Create a branch
git branch or git branch --list # List branches
git branch -d "branchname" # Delete branch
git branch -m "newbranchname" # Rename branch
git branch -d "branchname" # Delete branch
git checkout "branchname" # Switch to branch (after commiting chages to current branch)

Pushing changes

git push origin master # Push to master branch
git push -u origin "branchname" # Push to new branch (create it on push)

Stash

Save current branch without committing.

git stash # Save current changes
git stash # Get back changes
git stash list # List stashed changes

Cached

git rm --cached . # Remove files from commit (usesful for when adding items to .gitignore)

Reset

After a bad commit use this to get back to a previous commit. Note you will loose changes made so back up your files somehow...

git reset --hard "gitcommitkey" # Git commit key found on github

Useful Pandoc commands

Description

Useful for converting files from one markup to another (e.g. .tex -> html).

Setup

Direct: Install Pandoc

Homebrew (MacOS & Linux):

brew install pandoc

Examples

Metadata

Insert at top of .md file to give some metadata in the form of a .yml file (without actually creating a .yml file).

---
title: TITLE
author:
- Author
date: \today{}
header: Header
footer: Footer
geometry: margin=3cm
abstract: Enter abstract 
header-includes:
- \usepackage{setspace}
- \doublespacing
- \usepackage{lineno}
- \linenumbers
...

Links

Add some links to the header.

<p style="text-align: center;"><a href="https://harveybates.xyz/">Home</a> - <a 
href="https://harveybates.xyz/knowledge/index">Back</a></p>

Markdown to html

pandoc -s input.md -c style.css -o output.html --metadata="Title"

LaTeX to markdown

pandoc -s input.tex -o output.text

Markdown to pdf

pandoc -s input.md -o output.pdf

Citations

Define in .bib file.

@article{name2020,
	author={Name, LastName},
	title={Title},
	journal={Journal},
	year={2020},
}

Define in text.

[@name2020]

Compile in terminal.

pandoc -s input.md --bibliography=ref.bib --citeproc -o output.pdf

Figure numbering

Using pandoc-fignos.

Install

python3 -m pip install pandoc-fignos --user

Use

Define figure with caption:

![Caption.](image.png){#fig:Name}

Reference figure with "Fig" extension:

\*@Fig:Name

When compiling use the following tag.

--filter pandoc-fignos

Slide show (beamer)

pandoc -t beamer input.md -o output.pdf
pandoc -t beamer input.tex -o output.pdf

Other

Useful commands can be found on the pandoc site. https://pandoc.org/demos.html

Working with dates in Python

Description

Useful commands when working with dates in python.

Imports

Data sorting, visualisation, basic statistics and line fitting.

import pandas as pd
import numpy as np
from datetime import datetime

Converting

Convert pandas dataframe column values to datetime.

dataFrame["timestamps"] = pd.to_datetime(dataFrame["timestamps"])

Alternative for array data.

dfTimeStamps = [datetime.strptime(time, "%Y-%m-%d %H:%M:%S") for time in dataFrame["timestamps"]]
# Replace formatted string with your time stamp structure

Indexing

Get dataframe values between particular dates.

range = dataFrame[dataFrame["timestamps"].between("DD-MM-YYYY HH:MM:SS", "DD-MM-YYYY HH:MM:SS")]

Aligning

Align date times from two separate dataframes. Useful for analysing data from separate sources.

# May have to remove minute and second values if they don't match
df1["matching_column_name"] = df1["matching_column_name"].apply(lambda t: t.replace(minute=0, second=0))
df2["matching_column_name"] = df2["matching_column_name"].apply(lambda t: t.replace(minute=0, second=0))

mergedDataFrame = pd.merge_asof(df1, df2, on="matching_column_name") # This will drop any values that don't match

Numpy

Description

Useful commands for numpy.

Import

import numpy as np

Random

np.random.randint(10) # Generates random int between 0 and 10
np.random.random(4) # Generates random array of length 4 between 0 and 1
np.random.rand(2, 3) # Generates 2 x 3 array of floats between 0 and 1

Create numpy array

Basic array:

array = np.array([1, 2, 3, 4])
# or 
array = [1, 2, 3, 4]
array = np.array(array) # >[1, 2, 3, 4]

Zero filled array:

array = np.zeros(4) # >[0, 0, 0, 0]

1D array with n numbers:

array = np.arange(4) # >[0, 1, 2, 3]

1D random array:

array = np.random.random(4) # >[0.521, 0.219, 0.972, 0.814]

1D evenly spaced array between two numbers:

array = np.linspace(0, 1, 4) # >[0, 0.333, 0.666, 1]
# np.linspace(start, stop, n)

Operations

np.copy(arr) # Copy array
np.append(arr, vals) # Appends to end of array
np.insert(arr, 3, vals) # Appends before index 3
np.concatenate((arr, arr1), axis=0) # Put arr1 at end of arr
np.split(arr, 2) # Split into two arrays
np.sort(arr)
np.reshape(3, 5) # Reshape to 3 by 5

Information

np.shape(arr)
arr.dtype # Get datatype within array
arr.size # Get size of array
np.array_equal(arr, arr1) # Compare arrays

Arithmetic Operations

np.max(arr)
np.min(arr)
np.sum(arr)
np.mean(arr) # Mean of arr
np.std(arr) # Stdev of arr
np.sqrt(arr)
np.exp(arr) # Exponent of arr
np.add(arr, arr1) 
np.subtract(arr, arr1)
np.divide(arr, arr1)
np.multiply(arr, arr1)

Pandas useful commands

Import

Data sorting, visualisation, basic statistics and line fitting.

import pandas as pd

Create dataframe (pandas)

Using .csv files.

dataframe = pd.read_csv("filename.csv") # Basic loading
dataframe = pd.read_csv("filename.csv").dropna() # Remove null values
dataframe = pd.read_csv("filename.csv", delimiter=';') # Define seperator

Using .xlsx files.

dataframe = pd.read_excel("filename.csv") # Basic loading
dataframe = pd.read_csv("filename.csv", sheet_name="name") # Define sheet name

Plotting data in Python with Matplotlib

Description

Useful commands for programming within Jupyter Notebook.

Data Preparation

Useful imports

Data sorting, visualisation, basic statistics and line fitting.

import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import matplotlib.ticker as mticker
import seaborn as sns

Plotting Data

Subplots

fig, ax = plt.subplots(1, 1, figsize=(8,8)) # Single column
fig, (ax, ax1) = plt.subplots(1, 2, figsize=(8, 8)) # Single column multiple plots
fig, [[ax, ax1], [ax2, ax3]] = plt.suplots(2, 2) # Multi-row multi-column

Basic Settings

ax.set_ylabel("LABEL")
ax.set_xlabel("LABEL")
ax.grid(color="lightgrey")
ax.set_ylim(lower, upper)
ax.set_xlim(lower, upper)
ax.legend(loc="upper right")

Dates

ax.set_xlim([datetime.date(YYYY, MM, DD), datetime.date(YYYY, MM, DD)])
ax.xaxis.set_major_locator(mdates.DayLocator(interval=n))
ax.xaxis.set_major_locator(mdates.MonthLocator(interval=n))
plt.setp(ax.get_xticklabels(), rotation=30, ha="right, rotation_mode="anchor")

Text and Arrows

ax.text(0.05, 0.95, "TEXT", transform=ax.transAxes, fontsize=16, va='top')
ax.annotate('TEXT', xy=(0, 0), xytext=(0, 0),
            arrowprops=dict(facecolor='black', shrink=0.05))

Exporting and Displaying

plt.tight_layout() # Use this or bbox_inches
plt.savefig("name.png", dpi=300, bbox_inches='tight')
plt.show()

Raspberry Pi Web-Server (http and https)

Description

Information for setting up a secure (https) webserver using a raspberry pi, nginx, certbot and a DNS provider. Based on a tutoiral from Luke Smith; however, in this version, a Raspberry Pi was used to host the site instead of a cloud service provider such as Amazon. Other good resources include this video from Fireship, which talks about the deployment of a website on a Raspberry Pi.

Proxy Server (nginx)

Reverse proxy server (TCP/UDP)
Documentation: nginx
Tutorial: PiMyLifeUp

Installation

sudo apt-get remove apache2 # Ensure apache2 in not installed
sudo apt-get install nginx # Install

Setup

Copy the default nginx configuration to a new configuration:

cp /etc/nginx/sites-available/default /etc/nginx/sites-available/your_site_name # Replace your_site_name

Edit the newly created file (/etc/nginx/sites-available/your_site_name) to display the following

server {
	listen 80;
	listen [::]:80 ;

	root /var/www/*website_file_dir*;

	index index.html index.htm index.nginx-debian.html;
	
	server_name *your_website_address*;

	location / {
		if ($request_uri ~ ^/(.*)\.html) {
			return 302 /$1;
		}
		try_files $uri $uri.html $uri/ =404;
	}

The code in the location scope allow html files to be indexed without a .html extension in the websites URL.

Put newley created configuration into enabled sites directory:

ln -s /etc/nginx/sites-available/your_site_name /etc/nginx/sites-enabled/

Make directory that holds website files (index.html etc.):

mkdir /var/www/your_site_name
cd /var/www/your_site_name
touch index.html # Create and edit mainpage as needed

nginx Commands

sudo systemctl start nginx # Start server
sudo systemctl status nginx # Server status
sudo systemctl stop nginx # Stop server
sudo systemctl restart nginx # Reboot server
sudo killall nginx # Stop all processes

HTTPS Certificate (certbot)

Gives your website a certificate of security that browsers can recognise and thus wont show that the site is insecure.

Installation

sudo apt-get install certbo python-certbot-nginx

Setup

sudo certbot --nginx

Follow the following commands:

Enter email
Agree to terms of service
Don't share email
Select https addresses
Redirect http to https

Note that certbot certificates expire every six months. To renew:

sudo certbot renew # Renew certificate

Port Forwarding

You need to setup port forwarding on your router in order for users outside your local network to access your site.

ip route # Find router address

Type the router address into a browser, navigate to port forwarding and add:

Protocol: TCP
WAN port: 80
LAN port: 80
Destination IP address: local_ip_address

You may also need to set (port 80 for http and 443 for https):

Protocol: TCP
WAN port: 443
LAN port: 443
Destination IP address: local_ip_address

Other Useful Commands

hostname -I # Get local IP address
curl icanhazip.com # Get public IPv6 address
curl ipv4.icanhazip.com # Get public IPv4 address

Tmux Bindings

Description

Useful tmux keybindings.

Tmux mode

Ctrl-a # Enter tmux mode

c # New window
tab # Swap between windows
n # Next window
0...9 # Window by number

; # Swap between panes
" # Split horizontally
% # Split vertically
} # Enter scroll mode
! # Convert pane to window
resize-pane -U (-D -L -R) 10 # Resize pane by number

To exit tmux, type exit in the command line.

Tmux Configuration

Description

Settings used for Tmux.

Sourcing Tmux Configuration

In the terminal.

tmux source-file ~/.tmux.conf

In Tmux (Requires being in tmux mode (ctrl-a)).

:source-file ~/.tmux.conf

Powerline

Powerline is a status line plugin for vim and tmux (plus some others).

Installation

Powerline requires Python (so if you don't have python you need that). Also you need pip to install.

pip install powerline status
# or
pip3 install powerline-status
# or
python3 -m pip install powerline-status

You need to add this line to your .tmux.conf file and then source your configuration.

source "/usr/local/lib/python3.9/site-packages/powerline/bindings/tmux/powerline.conf"

If you get problems with the installed fonts (i.e. you get question marks in your powerline) you need to install a powerline specific font.

To do this, navigate to this link and clone the library.

git clone https://github.com/powerline/fonts
cd fonts
./install.sh # Install powerline fonts

Now navigate to your terminal preferences and change your font to a powerline specific font. (I am using "Meslo LG S Regular for Powerline 11pt").

Configuration

# -- general -------------------------------------------------------------------

set -g default-terminal "screen-256color" # colors!
source "/usr/local/lib/python3.9/site-packages/powerline/bindings/tmux/powerline.conf"
setw -g xterm-keys on
set -s escape-time 10                     # faster command sequences
set -sg repeat-time 600                   # increase repeat timeout
set -s focus-events on

unbind C-b
set-option -g prefix C-a
bind-key C-a send-prefix

bind C-a send-prefix -2

set -q -g status-utf8 on                  # expect UTF-8 (tmux < 2.2)
setw -q -g utf8 on

set -g history-limit 5000                 # boost history

# edit configuration
bind e new-window -n "~/.tmux.conf.local" "EDITOR=\${EDITOR//mvim/vim} && EDITOR=\${EDITOR//gvim/vim} && \${EDITOR:-vim} ~/.tmux.conf.local && tmux source ~/.tmux.conf && tmux display \"~/.tmux.conf sourced\""

# reload configuration
bind r source-file ~/.tmux.conf \; display '~/.tmux.conf sourced'


# -- display -------------------------------------------------------------------

set -g base-index 1           # start windows numbering at 1
setw -g pane-base-index 1     # make pane numbering consistent with windows

setw -g automatic-rename on   # rename window to reflect current program
set -g renumber-windows on    # renumber windows when a window is closed

set -g set-titles on          # set terminal title

set -g display-panes-time 800 # slightly longer pane indicators display time
set -g display-time 1000      # slightly longer status messages display time

set -g status-interval 10     # redraw status line every 10 seconds

# clear both screen and history
bind -n C-l send-keys C-l \; run 'sleep 0.1' \; clear-history

# activity
set -g monitor-activity on
set -g visual-activity off


# -- navigation ----------------------------------------------------------------

# find session
bind C-f command-prompt -p find-session 'switch-client -t %%'

# split current window horizontally
bind - split-window -v
# split current window vertically
bind _ split-window -h

# pane navigation
bind -r h select-pane -L  # move left
bind -r j select-pane -D  # move down
bind -r k select-pane -U  # move up
bind -r l select-pane -R  # move right
bind > swap-pane -D       # swap current pane with the next one
bind < swap-pane -U       # swap current pane with the previous one

# maximize current pane
bind + run 'cut -c3- ~/.tmux.conf | sh -s _maximize_pane "#{session_name}" #D'

# window navigation
bind Tab last-window        # move to last active window

# toggle mouse
bind m run "cut -c3- ~/.tmux.conf | sh -s _toggle_mouse"


# -- urlview -------------------------------------------------------------------

bind U run "cut -c3- ~/.tmux.conf | sh -s _urlview #{pane_id}"


# -- facebook pathpicker -------------------------------------------------------

bind F run "cut -c3- ~/.tmux.conf | sh -s _fpp #{pane_id}"


# -- list choice (tmux < 2.4) --------------------------------------------------

# vi-choice is gone in tmux >= 2.4
run -b 'tmux bind -t vi-choice h tree-collapse 2> /dev/null || true'
run -b 'tmux bind -t vi-choice l tree-expand 2> /dev/null || true'
run -b 'tmux bind -t vi-choice K start-of-list 2> /dev/null || true'
run -b 'tmux bind -t vi-choice J end-of-list 2> /dev/null || true'
run -b 'tmux bind -t vi-choice H tree-collapse-all 2> /dev/null || true'
run -b 'tmux bind -t vi-choice L tree-expand-all 2> /dev/null || true'
run -b 'tmux bind -t vi-choice Escape cancel 2> /dev/null || true'


# -- edit mode (tmux < 2.4) ----------------------------------------------------

# vi-edit is gone in tmux >= 2.4
run -b 'tmux bind -ct vi-edit H start-of-line 2> /dev/null || true'
run -b 'tmux bind -ct vi-edit L end-of-line 2> /dev/null || true'
run -b 'tmux bind -ct vi-edit q cancel 2> /dev/null || true'
run -b 'tmux bind -ct vi-edit Escape cancel 2> /dev/null || true'


# -- copy mode -----------------------------------------------------------------

bind Enter copy-mode # enter copy mode

run -b 'tmux bind -t vi-copy v begin-selection 2> /dev/null || true'
run -b 'tmux bind -T copy-mode-vi v send -X begin-selection 2> /dev/null || true'
run -b 'tmux bind -t vi-copy C-v rectangle-toggle 2> /dev/null || true'
run -b 'tmux bind -T copy-mode-vi C-v send -X rectangle-toggle 2> /dev/null || true'
run -b 'tmux bind -t vi-copy y copy-selection 2> /dev/null || true'
run -b 'tmux bind -T copy-mode-vi y send -X copy-selection-and-cancel 2> /dev/null || true'
run -b 'tmux bind -t vi-copy Escape cancel 2> /dev/null || true'
run -b 'tmux bind -T copy-mode-vi Escape send -X cancel 2> /dev/null || true'
run -b 'tmux bind -t vi-copy H start-of-line 2> /dev/null || true'
run -b 'tmux bind -T copy-mode-vi H send -X start-of-line 2> /dev/null || true'
run -b 'tmux bind -t vi-copy L end-of-line 2> /dev/null || true'
run -b 'tmux bind -T copy-mode-vi L send -X end-of-line 2> /dev/null || true'

# copy to macOS clipboard
if -b 'command -v pbcopy > /dev/null 2>&1' 'bind y run -b "tmux save-buffer - | pbcopy"'
if -b 'command -v reattach-to-user-namespace > /dev/null 2>&1' 'bind y run -b "tmux save-buffer - | reattach-to-user-namespace pbcopy"'
# copy to X11 clipboard
if -b 'command -v xsel > /dev/null 2>&1' 'bind y run -b "tmux save-buffer - | xsel -i -b"'
if -b '! command -v xsel > /dev/null 2>&1 && command -v xclip > /dev/null 2>&1' 'bind y run -b "tmux save-buffer - | xclip -i -selection clipboard >/dev/null 2>&1"'
# copy to Windows clipboard
if -b 'command -v clip.exe > /dev/null 2>&1' 'bind y run -b "tmux save-buffer - | clip.exe"'
if -b '[ -c /dev/clipboard ]' 'bind y run -b "tmux save-buffer - > /dev/clipboard"'


# -- buffers -------------------------------------------------------------------

bind b list-buffers  # list paste buffers
bind p paste-buffer  # paste from the top paste buffer
bind P choose-buffer # choose which buffer to paste from


# -- user defined overrides ----------------------------------------------------

if '[ -f ~/.tmux.conf.local ]' 'source ~/.tmux.conf.local'


# -- 8< ------------------------------------------------------------------------

#run 'cut -c3- ~/.tmux.conf | sh -s _apply_configuration'
run -b '[ -z "#{window_active}" ] && [ -z "#{version}" ] && tmux set display-time 3000 \; display "This configuration will soon require tmux >= 2.4" \; set -u display-time || true'

Vim Commands

Description

Useful vim commands for navigating files, creating, editing and deleting code.

Modes

i # Insert mode (before cursor)
I # Insert mode (start of line)
a # Insert mode (after cursor) 
A # Insert mode (end of line)
v # Visual mode
V # Visual line mode
Ctrl-v # Visual block
: # Command mode

e # Move to end of word
w # Jump forward a word
w # Jump word forward
b # Jump word backward
% # Jump to corresponding bracket
} # Jump up to next space
{ # Jump down to next space
gg # Top of file
G # Bottom of file
M # Middle of screen
nG # Move to line number n
'' # Return to last position

View

zt # Cursor at top of screen
zz # Put cursor at center of screen
zb # Put cursor at bottom of screen

Modifying

dd # Delete line
dw # Delete word
dit # Delete inside html tags
dct # Delete inside html tages and go to insert mode
da* # Delete around defined char (*)
di* # Delete inside defined char (*)
dG # Delete from cursor to end of file
d} # Delete from cursor to next line break
d{ # Delete from cursor to previous line break

P # Paste before cursor
p # Paste after cursor
"+P # Paste before cursor (or above line) from system clipboard
"+p # Paste after cursor (or next line) from system clipboard

yw # Copy word
yy # Copy line

gQQ # Single line to multi-line
J # Join lines

:%s/\s+$//e # Remove whitespace from file

Searching

?"term" # Search for term (case-sensitive)
:noh # Stop highlighting

Regex

Find some useful vim regex patterns here: vimregex.com

Multi-line Editing

Ctrl-v # Enter visual block mode
j or k # Move down or up rows
i # Insert mode
esc # Escape (whereafter rows should load)

Alternative using visual-multi (if installed).

Ctrl-n # Open visual multi
n # Select next item
i # Edit and insert new text
esc esc # Escape visual multi-mode (must press twice)

Indent or De-Indent Multiple Lines

Entire file.

gg=G

Selection

v # Enter visual mode
j or k # To select multiple lines
0...9< # To de-indent multiple times
0...9> # To indent multiple times