Taogen's Blog

1. Use a new git branch to develop new features. After finishing a new feature, switch to the main branch.

It’s much clearer when you’re doing multiple functions at the same time.

2. The committed code needs to push to the remote repository as soon as possible.

Code in a local environment can easily get lost. One time I deleted the entire project directory and my code was lost.

3. The test code is for testing only. Don’t use test code as your functional source code.

4. Use environment variables for configurations. Don’t hardcode configuration.

It’s safe and convenient.

5. Don’t keep uncommitted changes of code all the time. Either commit and push it to a remote branch or restore it.

First, uncommitted and unpushed changes are easily lost. Second, keep uncommitted changes makes new updates to the same file difficult. Third, It also messes up code commits.

If the changes are important and you won’t need them for a while, you can copy and save them to another place.

6. First, solve the problem. Then, write the code. Before writing code, you need to understand and organize the requirements and complete the database design, API design, and detailed design.

It makes development easier.

7. Anything that can be automated should be automated.

The problem “cannot resolve symbol” means IDEA hasn’t successfully indexed your project code and dependent libraries.

There are two common problems that cause the “cannot resolve symbol” problems:

1. Maven dependency configuration problems.
2. IDEA problems.

Maven Configuration Problems

Ensure Maven dependency resolve correctly

Your project’s Maven dependencies must be resolved correctly. Otherwise, IDEA can’t index project files successfully.

To check that Maven dependencies are resolved correctly, you can execute the following command:

mvn -Dmaven.test.skip=true clean package

If you can’t pass the above command, then there is something wrong with your Maven configuration. You can update your pom.xml and run the above command again.

After you passed the above command by updating your pom.xml, you need to reload the maven project to download missing dependencies. Right click the pom.xml in IDEA -> Maven -> Reload project. If the “cannot resolve symbol” errors are gone, the problem is solved.

If you have passed the above command and reloaded the project but there are still the “cannot resolve symbol” problems in IDEA, then it’s IDEA problems not Maven. You can go to the “Problems with IDEA” section.

Common problems and solutions with Maven

Problem 1: The dependency cannot be found

This means that the dependency cannot be found from the remote Maven repository.

Error Information:

Could not resolve dependencies for project {your_project}: Failed to collect dependencies at {groupId}:{artifactId}.jar:{version}

Solutions:

Check if the groupId, artifactId or version of the dependency is correct.

Check if there is an unofficial repository defined in the project’s pom.xml or ~/.m2/settings.xml, try using Maven central repository.

<repository>
    <id>central</id>
    <name>Maven Central Repository</name>
    <url>https://repo1.maven.org/maven2</url>
</repository>

Problem 2: dependency conflict

This means that multiple versions of a dependency are added or indirectly added to the Maven project. A dependency can only exist in one version. The actual version used causes the referenced method or class to be unable to be found.

Error Information:

NoClassDefFoundError, ClassNotFoundException, or NoSuchMethodError

Solutions

Use the “Maven Helper” plugin to check if there is a dependency conflict. You can try excluding unwanted versions by right-clicking on the version in the Maven Helper plugin window.

If your Maven project has a parent project, you also need to check whether the version defined in the parent project is compatible with your current project.

More tips

If you pom.xml is correct, but you still can’t pass mvn clean package. You can try to force update dependencies:

# force update the dependencies
mvn clean package -U

Problems with IDEA

Try to re-index your project files

Close the project or exit IDEA, then delete the .idea folder, then reopen project or restart IDEA. When there is no .idea folder in your project root path, IDEA will re-index project files automatically.

In addition to the .idea folder, if your project is a git repository, it’s recommended to delete all file ignored by git.

# delete git ignored files
git clean -dfx

After you re-indexed the project (delete .idea), if the “cannot resolve symbol” errors are gone, the problem is solved.

If the “cannot resolve symbol” errors still exist, you can try rebuild project. In top menu bar, Build -> Rebuild project.

If you have re-indexed the project (delete .idea) and rebuilt the project but the “cannot resolve symbol” errors still exist, you can go to the next step “Invalidate IDEA cache”.

Invalidate IDEA cache

IDEA cache also affects IDEA index project files. If your Maven configurations are right, but there are still “cannot resolve symbol” problems. You can try to Invalidate Caches:

File -> Invalidate Caches -> checked “clear file system cache and Local History”, and then click “Invalidate and Restart”.

Re-enter IDEA and waiting for “Update indexes” to complete. After “Update indexes” is done. Then we need rebulid the project:

Build -> click “Rebuild Project”.

After invalidated IDEA cache, if the “cannot resolve symbol” errors are gone, the problem is solved. Otherwise, You can try to re-index your project files again.

In this post, we will introduce several ways to find HTML DOM elements in JavaScript.

querySelector methods

Document or Element methods

querySelectorAll(selectors)

• Parameters: the selectors parameter is a valid CSS selector string.
• Return Value: returns a static (not live) NodeList representing a list of the document’s elements that match the specified group of selectors, or an empty NodeList in case of no matches.

querySelector(selectors)

• Parameters: the selectors parameter is a valid CSS selector string.
• Return Value: return an Element object representing the first element in the document that matches the specified set of CSS selectors, or null is returned if there are no matches.

Find by id

document.querySelector("#test")

Find by HTML tag

document.querySelectorAll("p")

Find by class names

document.querySelectorAll("div.className1")
// or
document.querySelectorAll(".className1")

Find by attributes

// match elements contain a attribute
document.querySelectorAll("elementName[attrName]")
// match attribute value
container.querySelectorAll("elementName[attrName='attrValue']");

and

document.querySelectorAll("div.className1.className2");
// or
document.querySelectorAll(".className1.className2");

document.querySelectorAll("div[attr1='value1'][attr2='value2']");
// or
document.querySelectorAll("[attr1='value1'][attr2='value2']");

or

document.querySelectorAll("div.className1, div.className2");
// or
document.querySelectorAll(".className1, .className2");

Find descendants, children, siblings

// descendants
document.querySelectorAll("div span");
// children
document.querySelectorAll("div>span");
// siblings
document.querySelectorAll("div~img");
// adjacent sibling
document.querySelectorAll("div+img");

For more information about CSS selectors, you can see the CSS selectors documentation.

Traversing match elements

const matchedElements = document.querySelectorAll("div");
matchedElements.forEach((item) => {
  console.log(item.innerText)
});

Filter elements

const matchedElements = document.querySelectorAll("div");
// querySelectorAll returns a NodeList not an Array. You can convert it to an Array
Array.from(matchedElements).filter((item) => item.classList.contains('note'));

const matchedElements = document.querySelectorAll("div");
Array.prototype.filter.call(
  matchedElements,
  (item) => item.classList.contains('note'),
);

getElementsBy methods

Document or Element methods

getElementById(id)

• Return value: An Element object describing the DOM element object matching the specified ID, or null if no matching element was found in the document.

getElementsByClassName(className)

• Parameters: className a string representing the class name(s) to match; multiple class names are separated by whitespace.
• Return value: A live HTMLCollection of found elements.

getElementsByName(nameAttrValue)

• Parameters: nameAttrValue the value of the name attribute of the element(s) we are looking for.
• Return value: A live NodeList collection, meaning it automatically updates as new elements with the same name are added to, or removed from, the document.

getElementsByTagName(elementName)

• Parameters: elementName a string representing the name of the elements. The special string * represents all elements.
• Return value: A live HTMLCollection of found elements.

Traverse elements

const testElements = document.getElementsByClassName('test');
for (let i = 0; i < testElements.length; i++) {
    console.log(testElements[i].innerText)
}

Filter elements

const matchedElements = document.getElementsByTagName("div");
Array.from(matchedElements).filter((item) => item.classList.contains('yourClassName'));

const matchedElements = document.getElementsByTagName("div");
Array.prototype.filter.call(
  matchedElements,
  (item) => item.classList.contains('yourClassName'),
);

The closest method

closest(selector) looks for the nearest ancestor that matches the CSS-selector.

Summary

References

[1] Searching: getElement*, querySelector*

[2] Document.querySelector() - mdn

[3] Document.querySelectorAll() - mdn

[4] Document.getElementsByClassName() - mdn

[5] CSS selectors - mdn

In this post, we will introduce the basic concepts of Docker. You’ll learn what docker is, why and when to use it.

What is Docker

Docker is an open source project for building, shipping, and running programs. It is a command line program, a background process, and a set of remote services that take a logistical approach to solving common software problems and simplifying your experience installing, running, publishing, and removing software. It accomplishes this by using an operating system technology called containers.

Running the hello-world in a container

Before running the hello-world program in a container, you need install Docker on your computer. You can download and install a Docker Desktop from https://docs.docker.com/install/. If you want to use docker on a Linux cloud server, you can download and install a Docker Engine from https://docs.docker.com/engine/install/.

After Docker is up and running on your computer, you can enter the following command to run the hello-world program provided by Docker in a container:

docker run hello-world
# or
docker run library/hello-world

After execute the above command you can see the output of the hello-world program

Hello from Docker!
This message shows that your installation appears to be working correctly.
...

After print the above text, the program exits, and container is marked as stopped. The running state of a container is directly tied to the state of a single running program inside the container. If a program is running, the container is running. If the program is stopped, the container is stopped. Restarting a container will run the program again.

In the second time to run a container, you can use docker start <container> to run an existing container directly instead of create a new similar container from its image again.

The process of the docker run command execution is:

The hello-world is called the image or repository name. You can think of the image name as the name of the program you want to install or run. The image is a collection of files and metadata. The metadata includes the specific program to execute and other relevant configuration details.

Docker Hub is a public registry provide by Docker Inc. It is a repository service and it is a cloud-based service where people push their Docker Container Images and also pull the Docker Container Images from the Docker Hub.

Container

Historically, UNIX-style operating systems have used the term jail to describe a modified runtime environment that limits the scope of resources that a jailed program can access. Jail features go back to 1979 and have been in evolution ever since. In 2005, with the release of Sun’s Solaris 10 and Solaris Containers, container has become the preferred term for such a runtime environment. The goal has expanded from limiting filesystem scope to isolating a process from all resources except where explicitly allowed.

Using containers has been a best practice for a long time. But manually building containers can be challenging and easy to do incorrectly. Docker uses existing container engines to provide consistent containers built according to best practices. This puts stronger security within reach for everyone.

Containers vs Virtual machines

Virtual machines

• Every virtual machine has a whole operating system
• Take a long time (often minutes) to create.
• Require significant resource overhead.

Container

• All Docker containers share an operating system.
• Docker containers don’t use any hardware virtualization. Programs running inside Docker containers interface directly with the host’s Linux kernel.
• Many programs can run in isolation without running redundant operating systems or suffering the delay of full boot sequences.

Running software in containers for isolation

Each container is running as a child process of the Docker engine, wrapped with a container, and the delegate process is running in its own memory subspace of the user space. Programs running inside a container can access only their own memory and resources as scoped by the container.

Shipping containers

Docker use images to shipping containers. A Docker image is a bundled snapshot of all the files that should be available to a program running inside a container. You can create as many containers from an image as you want. Images are the shippable units in the Docker ecosystem.

Docker provides a set of infrastructure components that simplify distributing Docker images. These components are registries and indexes. You can use publicly available infrastructure provided by Docker Inc., other hosting companies, or your own registries and indexes. You can store and search images from a registry.

Why Use Docker

Docker makes it easy and simple to use the container and isolation features provided by operating systems.

Why use the container and isolation features

• Dependency conflict.
• Portability between operating systems. Docker runs natively on Linux and comes with a single virtual machine for macOS and Windows environments. You can run the same software on any system.
• Protecting your computer. Docker prevents malicious program attacks through operating system resource access control.
• Application removal. All program execution files and program-produced files are in a container. You can remove all of these files easily.

When to use Docker

Docker can run almost anywhere for any application. But currently Docker can run only applications that can run on a Linux operating system, or Windows applications on Windows Server. If you want to run a macOS or Windows native application on your desktop, you can’t yet do so with Docker.

References

[1] Jeffrey, Nickoloff and Stephen, Kuenzli. Docker in Action. 2nd ed., Manning Publications, 2019.

Pass data between a parent and a child component

Pass data from parent component to its child

Use props

Parent.js

<Child name={childName}></Child>

Child.js

<p>Receive from parent by props: {this.props.name}</p>

Pass data from child component to its parent

Pass function from parent component to its child.

The Child component call the function with parameters.

A complete example

import { PureComponent } from 'react';
import { Input } from 'antd';

class Parent extends PureComponent {

    constructor(props) {
        super(props);
        this.state = ({
            childName: "default child name",
            name: "default parent name"
        });

    }

    handleInputChange = (e) => {
        const name = e.target.value;
        this.setState({
            childName: name
        })
    }

    handleParentValueChange = (value) => {
        this.setState({
            name: value
        })
    }

    render() {
        return (
            <div>
                <h2>I'm the parent page</h2>
                <p>Receive from child: {this.state.name}</p>
                Enter data to pass to child: <Input onChange={this.handleInputChange}></Input>
                <Child name={this.state.childName} onParentNameChange={this.handleParentValueChange}></Child>
            </div>
        )
    }
}

export default Parent;

import { PureComponent } from 'react';
import { Input } from 'antd';

class Child extends PureComponent {

    constructor(props) {
        super(props);
        this.state = ({});
    }

    handleInputChange = (e) => {
        const name = e.target.value;
        this.props.onParentNameChange(name);
    }

    render() {
        return (
            <div style={{backgroundColor: "lightgray", padding: "10px 10px 10px 10px"}}>
                <h2>I'm the child page</h2>
                <p>Receive from parent by props: {this.props.name}</p>
                Enter data to pass to parent: <Input onChange={this.handleInputChange}></Input>
            </div>
        )
    }
}

export default Child;

Pass functions between a parent and a child component

Pass functions to child components

If you need to have access to the parent component in the handler, you also need to bind the function to the component instance (see below).

There are several ways to make sure functions have access to component attributes like this.props and this.state.

Bind in Constructor (ES2015)

class Foo extends Component {
  constructor(props) {
    super(props);
    this.handleClick = this.handleClick.bind(this);
  }
  handleClick() {
    console.log('Click happened');
  }
  render() {
    return <button onClick={this.handleClick}>Click Me</button>;
  }
}

Note: Make sure you aren’t calling the function when you pass it to the component

render() {
    // Wrong: handleClick is called instead of passed as a reference!
    // The function being called every time the component renders.
    return <button onClick={this.handleClick()}>Click Me</button>
}

Class Properties (ES2022)

class Foo extends Component {
  handleClick = () => {
    console.log('Click happened');
  };
  render() {
    return <button onClick={this.handleClick}>Click Me</button>;
  }
}

Note: Make sure you aren’t calling the function when you pass it to the component

render() {
    // Wrong: handleClick is called instead of passed as a reference!
    // The function being called every time the component renders.
    return <button onClick={this.handleClick()}>Click Me</button>
}

Bind in Render

class Foo extends Component {
  handleClick() {
    console.log('Click happened');
  }
  render() {
    return <button onClick={this.handleClick.bind(this)}>Click Me</button>;
  }
}

Pass a parameter to an event handler in parent component’s render method. Values passed by child components will be ignored.

<button onClick={this.handleClick.bind(this, id)} />

Note: Using Function.prototype.bind in render creates a new function each time the component renders, which may have performance implications.

Arrow Function in Render

class Foo extends Component {
  handleClick() {
    console.log('Click happened');
  }
  render() {
    return <button onClick={() => this.handleClick()}>Click Me</button>;
  }
}

Pass a parameter to an event handler

<button onClick={() => this.handleClick(id)} />

Note: Using an arrow function in render creates a new function each time the component renders, which may break optimizations based on strict identity comparison.

Call child functions in a parent component

refs

Previously, refs were only supported for Class-based components. With the advent of React Hooks, that’s no longer the case.

Modern React with Hooks (v16.8+)

Hook parent and hook child (Functional Component Solution)

const { forwardRef, useRef, useImperativeHandle } = React;

const Parent = () => {
  // In order to gain access to the child component instance,
  // you need to assign it to a `ref`, so we call `useRef()` to get one
  const childRef = useRef();

  return (
    <div>
      <Child ref={childRef} />
      <button onClick={() => childRef.current.getAlert()}>Click</button>
    </div>
  );
};

// We need to wrap component in `forwardRef` in order to gain
// access to the ref object that is assigned using the `ref` prop.
// This ref is passed as the second parameter to the function component.
const Child = forwardRef((props, ref) => {

  // The component instance will be extended
  // with whatever you return from the callback passed
  // as the second argument
  useImperativeHandle(ref, () => ({
    getAlert() {
      alert("getAlert from Child");
    }
  }));
  return <h1>Hi</h1>;
});

Legacy API using Class Components (>= react@16.4)

Class parent and class child (Class Component Solution)

class Parent extends React.Component {
  constructor(props) {
    super(props)
    this.myRef = React.createRef()
  }

  render() {
    return (<View>
      <Child ref={this.myRef}/>
      <Button title={'call me'}
              onPress={() => this.myRef.current.childMethod()}/>
    </View>)
  }
}

class Child extends React.Component {

  childMethod() {
    console.log('call me')
  }

  render() {
    return (<View><Text> I am a child</Text></View>)
  }
}

Class component and Hook

Class parent and hook child

class Parent extends React.Component {
  constructor(props) {
    super(props)
    this.myRef = React.createRef()
  }

  render() {
    return (<View>
      <Child ref={this.myRef}/>
      <Button title={'call me'}
              onPress={() => this.myRef.current.childMethod()}/>
    </View>)
  }
}

const Child = React.forwardRef((props, ref) => {

  useImperativeHandle(ref, () => ({
    childMethod() {
      childMethod()
    }
  }))

  function childMethod() {
    console.log('call me')
  }

  return (<View><Text> I am a child</Text></View>)
})

Hook parent and class child

function Parent(props) {

  const myRef = useRef()

  return (<View>
    <Child ref={myRef}/>
    <Button title={'call me'}
            onPress={() => myRef.current.childMethod()}/>
  </View>)
}

class Child extends React.Component {

  childMethod() {
    console.log('call me')
  }

  render() {
    return (<View><Text> I am a child</Text></View>)
  }
}

useEffect

Parent

const [refresh, doRefresh] = useState(0);
<Button onClick={() => doRefresh(prev => prev + 1)} />
<Children refresh={refresh} />

Children

useEffect(() => {
    performRefresh(); //children function of interest
  }, [props.refresh]);

Others

class Parent extends Component {
 render() {
  return (
    <div>
      <Child setClick={click => this.clickChild = click}/>
      <button onClick={() => this.clickChild()}>Click</button>
    </div>
  );
 }
}

class Child extends Component {
 constructor(props) {
    super(props);
    this.getAlert = this.getAlert.bind(this);
 }
 componentDidMount() {
    this.props.setClick(this.getAlert);
 }
 getAlert() {
    alert('clicked');
 }
 render() {
  return (
    <h1 ref="hello">Hello</h1>
  );
 }
}

Share data between components with Redux

A basic example:

import { createStore } from 'redux'

/**
 * This is a reducer - a function that takes a current state value and an
 * action object describing "what happened", and returns a new state value.
 * A reducer's function signature is: (state, action) => newState
 *
 * The Redux state should contain only plain JS objects, arrays, and primitives.
 * The root state value is usually an object. It's important that you should
 * not mutate the state object, but return a new object if the state changes.
 *
 * You can use any conditional logic you want in a reducer. In this example,
 * we use a switch statement, but it's not required.
 */
function counterReducer(state = { value: 0 }, action) {
  switch (action.type) {
    case 'counter/incremented':
      return { value: state.value + 1 }
    case 'counter/decremented':
      return { value: state.value - 1 }
    default:
      return state
  }
}

// Create a Redux store holding the state of your app.
// Its API is { subscribe, dispatch, getState }.
let store = createStore(counterReducer)

// You can use subscribe() to update the UI in response to state changes.
// Normally you'd use a view binding library (e.g. React Redux) rather than subscribe() directly.
// There may be additional use cases where it's helpful to subscribe as well.

store.subscribe(() => console.log(store.getState()))

// The only way to mutate the internal state is to dispatch an action.
// The actions can be serialized, logged or stored and later replayed.
store.dispatch({ type: 'counter/incremented' })
// {value: 1}
store.dispatch({ type: 'counter/incremented' })
// {value: 2}
store.dispatch({ type: 'counter/decremented' })
// {value: 1}

Pass data to redirect component

React Router

Pass data with <Redirect>:

<Redirect to={{
            pathname: '/nav',
            state: { id: '123' }
        }}
/>

Pass data with history.push():

history.push({
  pathname: '/about',
  search: '?the=search',
  state: { some: 'state' }
})

Access data:

this.props.location.state.id

UmiJS

Pass data with query string

import { history, router } from 'umi';

history.push('/path?field=value&field2=value2')
// or
history.push({
    pathname: '/path',
    query: {
        field: value
    }
})
// or
router.push('/path?field=value&field2=value2')
// or
router.push({
    pathname: '/path',
    query: {
        field: value
    }
})

Access query string

import { useLocation } from 'umi';

const location = useLocation();
console.log(location.query)

URL query string

Pass data to components use URL query string: url?field=value&field2=value2

Get query string parameters

const params = new Proxy(new URLSearchParams(window.location.search), {
  get: (searchParams, prop) => searchParams.get(prop),
});
// Get the value of "some_key" in eg "https://example.com/?some_key=some_value"
let value = params.some_key; // "some_value"

References

• Passing Functions to Components - React Docs

Design

Creating good architecture design.

Creating good database design.

Creating good API design.

Unit test

Write unit tests or test-drive development.

Keep tests clean.

Name

Creating good names

Don’t use literals.

Functions

Keep functions small and do one thing

Write functions as writing a story.

Code

Avoid repeated code. Extract repeated code into a common function or utility function.

Consider using better data structures and algorithms.

Classes

Keep small classes

Consider creating a better class hierarchy

Consider using a better design pattern

Error Handling

Consider possible exceptions for each line of code.

match and match_phrase do not work for partial searches. We can use wildcard, query_string or regexp to match a partial string.

For example, search gt3p in the content column.

wildcard

{
    "query": {
        "wildcard": { 
            "content" : "*gt3p*" 
        }
    }
}

query_string

{
    "query": {
        "query_string": {
            "default_field": "content",
            "query": "*gt3p*"
        }
    }
}

regexp

{
    "query": {
        "regexp": {
            "content": ".*gt3p.*"
        } 
    }
}

What does the shit code look like

Bad names for variables and functions. And there are a lot of literals.

Bad data structures and database schema design.

Very long functions. Do a lot of things in one function. The business processing logic is chaotic.

Implementing algorithms are very ugly and low performance.

There are some potential bugs and problems in the code.

What it’s like to read shit code

It’s hard to understand it. You need to read carefully line by line. This is very painful and time consuming.

When I read the code behind, I have forgotten the previous code. It’s hard to figure out the entire processing logic.

Hard to read and understand, ugly code implementation and existing potential bugs drive me crazy.

How should we read the shit code

It’s known reading shit code is very painful. But there are some tips that may relieve your headaches.

1. Write a description of the logic of the code in your own words. It helps you understand the code more easily.
2. Do some work to modify the code slightly, such as renaming some variables and updating code order. It makes the code easier to read.

Getting Started

Hello World

> print("Hello World")
Hello World

Get input data from console

input_string_var = input("Enter some data: ")

Comment

# Single line comments start with a number symbol.
""" Multiline strings can be written
    using three "s, and are often used
    as documentation.
"""

Variables and Data Types

Variables

There are no declarations, only assignments. Convention is to use lower_case_with_underscores.

some_var = 5

Data Types

> x = 5
> print(type(x))
<class 'int'>

String and Array

String

Strings are created with “ or ‘

str1 = "This is a string."
str2 = 'This is also a string.'

A string can be treated like a list of characters

"Hello world!"[0]  # => 'H'

Properties of Strings

len("This is a string")

String concatenation

"Hello " + "world!"

String formatting

name = "Reiko"
format_str = f"She said her name is {name}." 
format_str2 = f"{name} is {len(name)} characters long."

format_str = "She said her name is {}.".format("Reiko")
format_str = "She said her name is {name}.".format(name="Reiko")

Array

li = []
other_li = [4, 5, 6]

Access

# Access a list like you would any array
li[0]
# Look at the last element
li[-1]

# Examine the length with "len()"
len(li)

Lookup

# Check for existence in a list with "in"
1 in li  # => True

# Get the index of the first item found matching the argument
li.index("a")

Insert

# Add stuff to the end of a list with append
li.append(1) 
# Insert an element at a specific index
li.insert(1, 2)

Update

li[1] = 11

Remove

# Remove from the end with pop
li.pop()
# Remove by index
del li[2]  # delete the 2th element
# Remove by value
li.remove(2) # Remove first occurrence of a value

Slice

li[start:end:step]

li[1:3]   # Return list from index 1 to 3
li[2:]    # Return list starting from index 2
li[:3]    # Return list from beginning until index 3
li[::2]   # Return list selecting every second entry
li[::-1]  # Return list in reverse order

one layer deep copy

li2 = li[:]

Concatenate

li + other_li
li.extend(other_li)

Tuple

Tuples are like lists but are immutable. You can’t insert, update, remove elements.

tup = (1, 2, 3)
# Tuples are created by default if you leave out the parentheses
tup2 = 11, 22, 33
tup[0]      # => 1
tup[0] = 3  # Raises a TypeError

Access

tup[0]

len(tup)

Lookup

1 in tup  # => True

li.index("a")

Slice

tup[:2]

Concatenate

tup + (4, 5, 6) 

Unpack tuples (or lists) into variables

a, b, c = (1, 2, 3)
d, e, f = 4, 5, 6
# swap two values
e, d = d, e

Dict

empty_dict = {}
filled_dict = {"one": 1, "two": 2, "three": 3}

Note keys for dictionaries have to be immutable types. This is to ensure that the key can be converted to a constant hash value for quick look-ups. Immutable types include ints, floats, strings, tuples.

invalid_dict = {[1,2,3]: "123"}  # => Yield a TypeError: unhashable type: 'list'
valid_dict = {(1,2,3):[1,2,3]}   # Values can be of any type, however.

Access

filled_dict["one"]
# Looking up a non-existing key is a KeyError
filled_dict["four"]  # KeyError
# Use "get()" method to avoid the KeyError
filled_dict.get("one")
# The get method supports a default argument when the value is missing
filled_dict.get("one", 4)

Put

# Adding to a dictionary
filled_dict.update({"four":4})  # => {"one": 1, "two": 2, "three": 3, "four": 4}
filled_dict["four"] = 4         # another way to add to dict

# "setdefault()" inserts into a dictionary only if the given key isn't present
filled_dict.update({"four":4})  # => {"one": 1, "two": 2, "three": 3, "four": 4}
filled_dict["four"] = 4         # another way to add to dict

Delete

# Remove keys from a dictionary with del
del filled_dict["one"]  # Removes the key "one" from filled dict

Lookup

"one" in filled_dict

list(filled_dict.keys())
list(filled_dict.values())

Get all keys as an iterable with “keys()”. We need to wrap the call in list() to turn it into a list. Note - for Python versions <3.7, dictionary key ordering is not guaranteed. Your results might not match the example below exactly. However, as of Python 3.7, dictionary items maintain the order at which they are inserted into the dictionary.

Set

empty_set = set()
# Initialize a set with a bunch of values.
some_set = {1, 1, 2, 2, 3, 4}  # some_set is now {1, 2, 3, 4}

# Similar to keys of a dictionary, elements of a set have to be immutable.
invalid_set = {[1], 1}  # => Raises a TypeError: unhashable type: 'list'
valid_set = {(1,), 1}

Insert

my_set.add(5)

Delete

my_set.remove(1)

Lookup

2 in filled_set

Intersection/union/difference/subset

filled_set = {1, 2, 3, 4, 5}
other_set = {3, 4, 5, 6}
# Do set intersection with &
filled_set & other_set # => {3, 4, 5}
# Do set union with |
filled_set | other_set  # => {1, 2, 3, 4, 5, 6}
# Do set difference with -
{1, 2, 3, 4} - {2, 3, 5}  # => {1, 4}
# Do set symmetric difference with ^
{1, 2, 3, 4} ^ {2, 3, 5}  # => {1, 4, 5}
# Check if set on the left is a superset of set on the right
{1, 2} >= {1, 2, 3} # => False
# Check if set on the left is a subset of set on the right
{1, 2} <= {1, 2, 3} # => True

Copy

# Make a one layer deep copy
filled_set = some_set.copy()  # filled_set is {1, 2, 3, 4, 5}
filled_set is some_set        # => False

Expressions

Arithmetic Operators

• +: add
• -: subtract
• *: multiply
• /: divide
• //: integer division rounds down
• %: modulo
• **: exponentiation

Logical Operators

• and
• or
• not

Note “and” and “or” are case-sensitive

Comparison operators

==, !=, >, <, >=, <=

Statements

Simple statements

Assignment

Call

return

Control Flow Statements

If Conditions

if…else

if some_var > 10:
    print("some_var is totally bigger than 10.")
elif some_var < 10:    # This elif clause is optional.
    print("some_var is smaller than 10.")
else:                  # This is optional too.
    print("some_var is indeed 10.")

case/switch

For loop

for

for animal in ["dog", "cat", "mouse"]:
    print("{} is a mammal".format(animal))
for i, value in enumerate(["dog", "cat", "mouse"]):
    print(i, value)

# "range(number)" returns an iterable of numbers from zero up to (but excluding) the given number
for i in range(4):
    print(i)
# "range(lower, upper)" returns an iterable of numbers
from the lower number to the upper number
for i in range(4, 8):
    print(i)
# "range(lower, upper, step)"
for i in range(4, 8, 2):
    print(i)

while

x = 0
while x < 4:
    print(x)
    x += 1

do…while

Exception handling

# Handle exceptions with a try/except block
try:
    # Use "raise" to raise an error
    raise IndexError("This is an index error")
except IndexError as e:
    pass                 # Refrain from this, provide a recovery (next example).
except (TypeError, NameError):
    pass                 # Multiple exceptions can be processed jointly.
else:                    # Optional clause to the try/except block. Must follow
                         # all except blocks.
    print("All good!")   # Runs only if the code in try raises no exceptions
finally:                 # Execute under all circumstances
    print("We can clean up resources here")

Functions

def add(x, y):
    print("x is {} and y is {}".format(x, y))
    return x + y

add(5, 6)

# Another way to call functions is with keyword arguments
add(y=6, x=5)  # Keyword arguments can arrive in any order.

# You can define functions that take a variable number of positional arguments
def varargs(*args):
    return args

varargs(1, 2, 3)

# You can define functions that take a variable number of keyword arguments, as well
def keyword_args(**kwargs):
    return kwargs

keyword_args(big="foot", loch="ness")

Expand arguments

all_the_args(*args)            # equivalent: all_the_args(1, 2, 3, 4)
all_the_args(**kwargs)         # equivalent: all_the_args(a=3, b=4)
all_the_args(*args, **kwargs)  # equivalent: all_the_args(1, 2, 3, 4, a=3, b=4)

# global scope
x = 5

def set_global_x(num):
    # global indicates that particular var lives in the global scope
    global x
    print(x)   # => 5
    x = num    # global var x is now set to 6
    print(x)

Nested function

def create_adder(x):
    def adder(y):
        return x + y
    return adder

add_10 = create_adder(10)
add_10(3)   # => 13

Anonymous functions

# There are also anonymous functions
(lambda x: x > 2)(3)                  # => True
(lambda x, y: x ** 2 + y ** 2)(2, 1)  # => 5

Modules

Python modules are just ordinary Python files. You can write your own, and import them. The name of the module is the same as the name of the file.

If you have a Python script named math.py in the same folder as your current script, the file math.py will be loaded instead of the built-in Python module. This happens because the local folder has priority over Python’s built-in libraries.

# You can import modules
import math
print(math.sqrt(16))  # => 4.0

# You can get specific functions from a module
from math import ceil, floor
print(ceil(3.7))   # => 4.0
print(floor(3.7))  # => 3.0

# You can import all functions from a module.
# Warning: this is not recommended
from math import *

# You can shorten module names
import math as m
math.sqrt(16) == m.sqrt(16)

Classes

Classes

Class members

• attribute
  • class attribute (set by class_name.class_attribute = value)
  • instance attribute (initialized by initializer)
  • instance properties (Properties are special kind of attributes which have getter, setter and delete methods like get, set and delete methods.)
• Methods
  • initializer
  • instance method (called by instances)
  • class method (called by instances)
  • static method (called by class_name.static_method())
  • getter
  • setter

Note that the double leading and trailing underscores denote objects or attributes that are used by Python but that live in user-controlled namespaces. Methods(or objects or attributes) like: __init__, __str__, __repr__ etc. are called special methods (or sometimes called dunder methods). You should not invent such names on your own.

# We use the "class" statement to create a class
class Human:

    # A class attribute. It is shared by all instances of this class
    species = "H. sapiens"
    
    # Basic initializer
    def __init__(self, name):
        # Assign the argument to the instance's name attribute
        self.name = name

        # Initialize property
        self._age = 0
        
	# An instance method. All methods take "self" as the first argument
    def say(self, msg):
        print("{name}: {message}".format(name=self.name, message=msg))

    # Another instance method
    def sing(self):
        return 'yo... yo... microphone check... one two... one two...'
    
    # A class method is shared among all instances
    # They are called with the calling class as the first argument
    @classmethod
    def get_species(cls):
        return cls.species

    # A static method is called without a class or instance reference
    @staticmethod
    def grunt():
        return "*grunt*"
    
    # A property is just like a getter.
    @property
    def age(self):
        return self._age
    
    # This allows the property to be set
    @age.setter
    def age(self, age):
        self._age = age
        
	# This allows the property to be deleted
    @age.deleter
    def age(self):
        del self._age

# Instantiate a class
i = Human(name="Ian")
# Call instance method
i.say("hi")                     # "Ian: hi"

j = Human("Joel")
j.say("hello") 

# Call our class method
i.say(i.get_species())          # "Ian: H. sapiens"
# Change the class attribute (shared attribute)
Human.species = "H. neanderthalensis"
i.say(i.get_species())          # => "Ian: H. neanderthalensis"
j.say(j.get_species())          # => "Joel: H. neanderthalensis"

# Call the static method
print(Human.grunt())            # => "*grunt*"

# Static methods can be called by instances too
print(i.grunt())  

# Update the property for this instance
i.age = 42
# Get the property
i.say(i.age)                    # => "Ian: 42"
j.say(j.age)                    # => "Joel: 0"
# Delete the property
del i.age
# i.age   

Inheritance

# Define Batman as a child that inherits from both Superhero and Bat
class Batman(Superhero, Bat):

Standard Library

I/O Streams and Files

Read

# Instead of try/finally to cleanup resources you can use a with statement
with open("myfile.txt") as f:
    for line in f:
        print(line)

# Reading from a file
with open('myfile1.txt', "r+") as file:
    contents = file.read()           # reads a string from a file
print(contents)

with open('myfile2.txt', "r+") as file:
    contents = json.load(file)       # reads a json object from a file
print(contents)

Write

# Writing to a file
contents = {"aa": 12, "bb": 21}
with open("myfile1.txt", "w+") as file:
    file.write(str(contents))

Advanced Topics

References

[1] Learn Python in Y minutes

[2] Python Tutorial

IO Streams

Input Streams

Get a Input Stream From a Path

Get Input Stream from filepath

String filepath = "D:\\test.txt"
// Java IO
InputStream is = new FileInputStream(filepath);

// Java NIO
Path path = Paths.get(filepath); 
System.out.println(path.normalize().toUri().toString()); // "file:///D:/test.txt"
InputStream is = new URL(path.toUri().toString()).openStream();

Get Input Stream from classpath

// Spring framework ClassPathResource
InputStream resourceAsStream = new ClassPathResource("application.yml").getInputStream();
// or
InputStream resourceAsStream = new ClassPathResource("/application.yml").getInputStream();

// Java ClassLoader
InputStream resourceAsStream = <CurrentClass>.class.getResourceAsStream("/application.yml");
// or
InputStream resourceAsStream = <CurrentClass>.class.getClassLoader().getResourceAsStream("application.yml");

Get Input Stream from file HTTP URL

// Java 8
InputStream input = new URL("http://xxx.xxx/fileUri").openStream();
// or
URLConnection connection = new URL(url + "?" + query).openConnection();
connection.setRequestProperty("Accept-Charset", charset);
InputStream response = connection.getInputStream();

// Java 9
HttpResponse response = HttpRequest
    .create(new URI("http://xxx.xxx/fileUri"))
    .headers("Foo", "foovalue", "Bar", "barvalue")
    .GET()
    .response();

// Spring Resource
Resource resource = new UrlResource("http://xxx.xxx/fileUri");
InputStream is = resource.getInputStream();

Read/Convert a Input Stream to a String

Using Stream API (Java 8)

new BufferedReader(new InputStreamReader(in)).lines().collect(Collectors.joining("\n"))

Using IOUtils.toString (Apache Commons IO API)

String result = IOUtils.toString(inputStream, StandardCharsets.UTF_8);

Using ByteArrayOutputStream and inputStream.read (JDK)

ByteArrayOutputStream result = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
for (int length; (length = inputStream.read(buffer)) != -1; ) {
    result.write(buffer, 0, length);
}
// StandardCharsets.UTF_8.name() > JDK 7
return result.toString("UTF-8");

Performance: ByteArrayOutputStream > IOUtils.toString > Stream API

Output Streams

Write data to file

Write string to file

String s = "hello world";
String outputFilePath = new StringBuilder()
    .append(System.getProperty("java.io.tmpdir"))
    .append(UUID.randomUUID())
    .append(".txt")
    .toString();
try (BufferedOutputStream out = new BufferedOutputStream(new FileOutputStream(outputFilePath))) {
    out.write(s.getBytes(StandardCharsets.UTF_8));
}
System.out.println("output file path: " + outputFilePath);

Read and write

Read From and Write to Files

Java IO

String inputFilePath = new StringBuilder()
    .append(System.getProperty("java.io.tmpdir"))
    .append("7d43f2b6-2145-4448-9c8f-c43f97ba4d9e.txt")
    .toString();
String outputFilePath = new StringBuilder()
    .append(System.getProperty("java.io.tmpdir"))
    .append(UUID.randomUUID())
    .append(".txt")
    .toString();
try (BufferedInputStream in = new BufferedInputStream(new FileInputStream(inputFilePath));
     BufferedOutputStream out = new BufferedOutputStream(new FileOutputStream(outputFilePath))) {
    int b;
    while ((b = in.read()) != -1) {
        out.write(b);
    }
}
System.out.println("output file path: " + outputFilePath);

For read and write, you can use the following two ways:

int b;
while ((b = in.read()) != -1) {
    out.write(b);
}

or

byte[] buffer = new byte[1024];
int lengthRead;
while ((lengthRead = in.read(buffer)) > 0) {
    out.write(buffer, 0, lengthRead);
    out.flush();
}

Java NIO.2 API

String inputFilePath = new StringBuilder()
    .append(System.getProperty("java.io.tmpdir"))
    .append("7d43f2b6-2145-4448-9c8f-c43f97ba4d9e.txt")
    .toString();
String outputFilePath = new StringBuilder()
    .append(System.getProperty("java.io.tmpdir"))
    .append(UUID.randomUUID())
    .append(".txt")
    .toString();
Path originalPath = new File(inputFilePath).toPath();
Path copied = Paths.get(outputFilePath);
Files.copy(originalPath, copied, StandardCopyOption.REPLACE_EXISTING);
System.out.println("output file path: " + outputFilePath);

Get a Path object by Paths.get(filePath) or new File(filePath).toPath()

By default, copying files and directories won’t overwrite existing ones, nor will it copy file attributes.

This behavior can be changed using the following copy options:

• REPLACE_EXISTING – replace a file if it exists
• COPY_ATTRIBUTES – copy metadata to the new file
• NOFOLLOW_LINKS – shouldn’t follow symbolic links

Apache Commons IO API

FileUtils.copyFile(original, copied);

Files

Get File Path

get file path by class path

// Spring framework ClassPathResource
String filePath = new ClassPathResource(fileClassPath).getFile().getAbsolutePath();

// Java ClassLoader
URL url = FileUtils.class.getClassLoader()
    .getResource(fileClassPath);
String filePath = Paths.get(url.toURI()).toFile().getAbsolutePath();

Creation

Create directory

File dir = new File(dirPath);
if (!dir.exists() || !dir.isDirectory()) {
    dir.mkdirs();
}

Files.createDirectories(new File(outputDir).toPath());

Delete

Delete a file

File file = new File(filePath);
file.delete();
// or
file.deleteOnExit();

Delete a directory

Java API

// function to delete subdirectories and files
public static void deleteDirectory(File file)
{
    // store all the paths of files and folders present inside directory
    for (File subfile : file.listFiles()) {

        // if it is a subfolder,e.g Rohan and Ritik,
        // recursiley call function to empty subfolder
        if (subfile.isDirectory()) {
            deleteDirectory(subfile);
        }

        // delete files and empty subfolders
        subfile.delete();
    }
}

Apache Common IO API

FileUtils.deleteDirectory(new File(dir));

or

FileUtils.forceDelete(new File(dir));

Update

Traversal

Information

Java File Mime Type

// 1
String mimeType = Files.probeContentType(file.toPath());
// 2
String mimeType = URLConnection.guessContentTypeFromName(fileName);
// 3
FileNameMap fileNameMap = URLConnection.getFileNameMap();
String mimeType = fileNameMap.getContentTypeFor(file.getName());
// 4
MimetypesFileTypeMap fileTypeMap = new MimetypesFileTypeMap();
String mimeType = fileTypeMap.getContentType(file.getName());

Temporary Files and Directories

Temporary Directory

// java.io.tmpdir
System.getProperty("java.io.tmpdir")

Windows 10: C:\Users\{user}\AppData\Local\Temp\

Debian: /tmp

Temporary file

// If you don't specify the file suffix, the default file suffix is ".tmp".
File file = File.createTempFile("temp", null);
System.out.println(file.getAbsolutePath());
file.deleteOnExit();

Path path = Files.createTempFile(fileName, ".txt");
System.out.println(path.toString());

Problems

Character Encoding Problems

The one-arguments constructors of FileReader always use the platform default encoding which is generally a bad idea.

Since Java 11 FileReader has also gained constructors that accept an encoding: new FileReader(file, charset) and new FileReader(fileName, charset).

In earlier versions of java, you need to use new InputStreamReader(new FileInputStream(pathToFile), ).

References

• Java FileReader encoding issue