Content

Java Thread Relationship

Thread Pool

Scenario

Trending Products

Electronics

Games

Beauty

Fashion

1

Gaming Mouse

2

Headset

3

Card Reader

4

Mouse

Involved Endpoints

Retrieve top 5 user preference product categories
• Example: Electronics, beauty, toys, sports, fashion etc.
Retrieve top 10 trending products for specified product category

Java 8

Spring Boot 2.7.0

Endpoint Calling Pattern

Endpoint Implementation

More Endpoints

Thread Pool Exhaustion Event

ForkJoinPool Common Pool

User Perspective

Experience slow response time due to high latency.
Loading...
HTTP request timeout error:

• 408 Request Timeout

• 503 Service Unavailable

• 504 Gateway Timeout

Developer Perspective

Observed unpredictable delays between the execution of subsequent asynchronous tasks, as thread pool exhaustion can cause irregular time gaps in task scheduling.
Errors:
• java.net.SocketException: Broken pipe (Write failed)
• java.io.IOException: Broken pipe

Potentail Causes And Fixes

Thread Pool Executor Mechanism

Task Num: 6 | Queue Capacity: 8

Defining Thread Pool Executor

Using Thread Pool Executor

Benefits of Thread Pool Executor

Control number of threads to be created in the pool.
Configurable to handle spikes in the number of requests during high traffic.
Avoid OutOfMemoryError by limiting the number of threads in the pool and tasks in task queue.

Little's Law

\(L = \lambda \times W\)

\(L:\) Required pool size (approximately)

\(\lambda:\) Estimated average incoming task rate

\(W:\) Estimated average task processing time

Core Pool Size

\(L = \lambda \times W\)

\(L:\) Required core pool size

\(\lambda:\) Thread spawn rate (per second)

\(W:\) Request completion time (second)

\(\lambda = 100 \times 5 = 500 s^{-1} \)

\(W = 0.2 s \)

\(L = 500 s^{-1} \times 0.2 s = 100\)

Max Pool Size

\(L = \lambda \times W\)

\(\lambda= 500 \times 5 = 2500 s^{-1} \)

\(W= 0.2 s \)

\(L= 2500 s^{-1} \times 0.2 s = 500\)

Queue Capacity Guideline

Core Pool Size \( \le \) Queue Capacity \( < \) Max Pool Size
Too small:
• Pool size expands to max pool size frequently
• Low task execution delay
Too large:
• Pool size will not expand immediately
• High task execution delay
• High memory usage

Configuration Result

Java Management Extensions (JMX)

A Java technology that provides tools for managing and monitoring applications and services using a standardized interface.

Datadog

Grafana

Prometheus

Elastic Stack

Important Values

Request Volume

Scenario 1

Scenario 2

Scenario 3

Scenario 4

Guidelines

Monitor CPU and Memory Usage
• Increase hardware resources if usage consistently exceeds 90%.
Caution in Decreasing Pool Size
• Traffic is extremely overestimated and remains consistently low.
• Reducing hardware resources for cost savings.
Setting up Alerts
• Telegram, Slack, or PagerDuty

Java Thread Relationship

Platform Threads vs Virtual Threads

Using Virtual Threads

Using Virtual Threads

Notes on Virtual Threads

Not optimized for CPU-intensive operations.
Not all version of Database driver libraries support.
Please verify compatibility.
Can improve overall throughput but do not necessarily guarantee speed improvements for individual tasks.
Best practice to conduct benchmarks before and after implementing virtual threads.

Key Takeaways

• Understand how concurrency / asynchronous is achieved.
  • Java - Platform Thread & Virtual Thread
• Familiarize method to configure thread pool.
  • Java - Thread Pool Executor
• Find the right way for monitoring.
  • Java - JMX