Internals of a Thread Pool

In order to perform an asynchronous task, a Thread is required. If you have to perform a large number of tasks then you might need to initialize multiple threads. As there is a limit to the number of threads which can be executed in parallel, you will need to create and destroy these threads again and again. This is suboptimal because of following reasons:

1. There will be a lot of overhead of creating and destroying multiple thread objects.
2. Also, management of the number of threads which can be initialized or the number of tasks which can be executed in parallel becomes very difficult.

A Thread Pool overcomes both of these issues. It has a pre-defined pool of pre-initialized long running threads. Whenever a task needs to be executed, it is assigned to one of these threads. Once the task is done, that thread again becomes available for the next task.

Thus, it provides improved performance by reducing the overhead of thread creation and destruction. Also, it provides a means of bounding and managing resources. For obvious reasons, this performance gain is huge when a large number of short tasks need to be executed.

Initialization

When a thread pool is initialized, its core pool size, maximum pool size, keep alive time and work queue are defined. After initialization, based on the requirement, either

1. one thread can be initialized which can idly wait for a task to be submitted or,
2. all the threads equal to core pool size can be initialized which can all idly wait for tasks to be submitted or,
3. threads can be initialized on demand whenever a new task is submitted

After initialization, a thread pool is ready to accept requests for task execution.

Source code for basic and advanced implementations for a Thread Pool based on below mentioned concepts is available here.

Important Components

A Thread Pool Executor has three major components:

1. Work Queue (e.g. BlockingQueue)
2. Workers (e.g. Set)
3. Thread Factory

Work Queue

Maintains a list of all the tasks which need to be performed. A work queue can be bounded (e.g. ArrayBlockingQueue) or unbounded (e.g. LinkedBlockingQueue).

Workers

As the name implies, a Worker (implements Runnable) is the one who picks the task from Work Queue and executes it. Once done, it waits for more tasks to be available in the Work Queue. The moment a task is available in the queue, it fetches it and starts executing it. This cycle/loop keeps on going until an exceptional termination condition has been met or no more tasks are available based on one of the two conditions (in the given order):

1. till keep alive time if either idle thread timeout has been defined or worker count is more than core pool size
2. until a new task is available in the queue

The maximum number of workers allowed can be Integer.MAX_VALUE.

Note: ThreadPoolExecutor in Java has limited this to 29 bits, (2^29)–1 as it uses the last 3 bits of the same integer field to keep track of its own state.

Thread Factory

A Thread Factory is used to create new threads which are required to run the workers. Each Worker runs in a different thread. Basically, when a thread’s start method is called, it calls run method overridden by the Worker which takes the Worker in a task execution loop as mentioned above.

Important Attributes

A Thread Pool Executor has some important properties which affect the Worker management:

1. core pool size, the minimum number of workers that need to be maintained. If the worker count ever goes below this, a new worker is created when a new task is submitted to the executor instead of storing it in the queue. If workers count is more this number then idle workers are killed.
2. maximum pool size, the maximum number of workers which can be running at a particular moment. If worker count reaches this limit, no new workers are created in the executor and newly submitted tasks are stored in the queue. So, whenever any worker is free, it picks up an available task from the queue.
3. keep alive time, the time after which an idle worker would be killed.

4. allow core thread timeout, a boolean which tells the system if idle core workers should be killed.

Shutdown

When asked to shut down, the thread pool executor will change its state to SHUTDOWN and will interrupt all the workers. If asked for immediate shutdown then it will interrupt all the workers which have started and drain the task queue else it will interrupt only idle workers and wait for running workers to complete tasks from the task queue. Once the shutdown is triggered, no new tasks are accepted.

Source code for basic and advanced implementations for a Thread Pool based on above mentioned concepts is available here.

Internals of Java Future<V>

Future<V> is one of the most interesting Interfaces of Java. This is a concurrency abstraction also know as Promise which promises to return the result of an asynchronous computation in future without blocking the code execution. This is helpful when the job to be executed is going to take a while and you have some more computation to be done before you need the results of this job. Once the executor executes the job, it stores the results of the job in outcome member variable of the Future object which is returned to you when the job is submitted to the executor.

Following is the usage of Future:

interface ImageDownloader {
  Object download(String imageUrl);
}

class App {
  ExecutorService executor = ...
  ImageDownloader downloader = ...

  Object getImage(final String imageUrl) throws InterruptedException {
    Future<Object> future
      = executor.submit(new Callable<Object>() {
        public String call() {
          return downloader.download(imageUrl);
        }});
    Object metadata = getImageMetadata(); // do other things while downloading
    try {
      Object image = prepareImage(future.get()); // use future
      return prepareImageResult();
    } catch (ExecutionException ex) {
      cleanup();
      return;
    }
  }
}

Here, the system has submitted a Job (Callable<T>) to the scheduler whose work is the download the image for the given URL. Upon submission, executor returned an instance of Future. By the time the image is downloaded, the system has to fetch other metadata related to the image (let’s say from the internal database).

Once, the system is ready to use the image object, it calls get method on the Future instance and gets the image object. If the image is not downloaded till the time get was called, Future will block the call and wait till the time image is not downloaded. In order to prevent infinite wait, you can pass a timeout as input, after which get will throw a TimeoutExecption.

How does Future do this?

Now, the interesting question is, how does the system know which Future instance is linked to which (Callable or Runnable) Job? Does the Executor maintain some kind of internal Map to keep track of it or is there a better way of handling it?

It’s actually very simple but a clever design. Following class-diagram shows the type hierarchy of Future interface. This is where things get interesting.

Future<V> type hierarchy

Here, we can see that the instance of Future is actually an instance of its subclass FutureTask which stores Callable reference as its member variable. This is how a Future instance is linked to the submitted job.

Following are the steps which are performed when a Job is submitted to an executor:

1. The client creates a Callable instance callable by providing a definition of call method.
2. The callable is submitted to an Executor.
3. The Executor creates a FutureTask instance future by passing this callable to its constructor. This means future has a reference to the Callable which needs to be executed.
4. This future is returned to the client by the Executor.
5. When it’s time, Executor calls run method on the future which in turn calls a call method on callable.
6. When callable is executed successfully, its result is stored in the future and is returned to the client when get method is called on the future.

Future<V> Usage Sequence diagram

Important Features

1. Future wraps Callable or Runnable object and submits it to the task executor for asynchronous execution.
2. Future provides a get method which is a blocking method and blocks the flow until the result is available.
3. Future provides isDone as a non-blocking method to check if the job has completed and the result is available. Also, isCancelled method is provided which tells if the job has been canceled before completion.
4. Future also provides a cancel method to cancel the execution of a job.
5. There is a sub-interface of Future which is ScheduledFuture. This is useful to get results of a scheduled task.

This is how the Future<V> is handled internally by Java. Knowing about Future interface is very important for building an asynchronous system.