Continuous Code Profiling<\/h2>\n\n- Continuous code profiling allows you to measure code performance at all times in any environment.<\/li>\n
- In production, an application process is mostly a closed box. You can observe external behaviors but not internal ones. Profiling is a way to look inside the box, observe these internal behaviors, and measure the application\u2019s code performance. You can detect and optimize the most time-consuming and resource-intensive lines of code that affect costs and end-user experience.<\/li>\n
- Profiling becomes much more effective when done in production because it\u2019s usually difficult and time-consuming to simulate production behavior or reproduce specific bottlenecks and outages in non-production environments.<\/li>\n<\/ol>\n
Continuous profiling in Datadog<\/h2>\n![\"Example](\"https:\/\/www.tothenew.com\/blog\/wp-ttn-blog\/uploads\/2025\/03\/Screenshot-from-2025-03-06-19-04-57.png\")
<\/a>Example of a CPU Time profiler flame graph in Continuous Profiler.<\/p><\/div>\n
Datadog Continuous Profiler is an always-on, production code profiler that enables you to analyze code-level performance across your entire environment, with minimal overhead. Profiles reveal which methods\/functions consume the most resources, such as CPU, memory allocation, wall time, and I\/O time spent. With this information, you can optimize your code to reduce end-user latency and cloud provider costs.<\/p>\n
Continuous profiling in Datadog allows you to do the following:<\/p>\n
\n- Perform code-level tracing with zero instrumentation<\/li>\n
- Visualize all your stack traces in one place<\/li>\n
- Discover bottlenecks in your code at a glance<\/li>\n
- Filter profile data using tags<\/li>\n
- Get actionable insights for performance improvements<\/li>\n<\/ul>\n
Continuous Profiling Goes Beyond Distributed Tracing<\/h2>\n
Consider you have a service called movies-api-java that has multiple endpoints that return movie metadata stored in a database. You learn that the service isn\u2019t performing well and users are experiencing increased latency issues. You want to optimize the service\u2019s performance to ensure that users have the best experience.<\/p>\n
You start using Datadog APM and distributed tracing to investigate the performance issues. For example, if you see additional spans in traces for unnecessary, repeated calls to a database when requests are made to an endpoint, you know you can use this information to determine a fix. However, for some issues, you see that you need to go beyond distributed tracing to be able to diagnose issues and optimize performance. In the example below, the trace shows that it takes over 2 seconds to respond to a request, but there are no child spans or other indicators that can tell you more about what may be causing the slow performance.<\/p>\n
<\/p>\n
![\"Example](\"https:\/\/www.tothenew.com\/blog\/wp-ttn-blog\/uploads\/2025\/03\/Screenshot-from-2025-03-13-08-55-14.png\")
<\/a>Example of a high-latency trace with no spans.<\/p><\/div>\n
For issues that you can\u2019t solve using APM, you can use Continuous Profiler to investigate at a more granular level. The difference between distributed tracing and continuous profiling is that traces tell you which requests were slow at the service level, whereas profiles tell you why they were slow at the code level.<\/p>\n
Profiles<\/h2>\n
In Datadog, trace data and profiling data are automatically linked for application processes that have both APM and Continuous Profiler enabled. In the trace details panel in APM, you can move directly from the information for a trace or a selected span in a trace to the associated profiling data using the Profiles tab.<\/p>\n
![\"The](\"https:\/\/www.tothenew.com\/blog\/wp-ttn-blog\/uploads\/2025\/03\/Screenshot-from-2025-03-13-09-00-02.png\")
<\/a>The Profiles UI found in the details panel for a trace.<\/p><\/div>\n
Interpreting the Profiler Flame Graph<\/h2>\n
Similar to traces, profiles are represented in flame graphs. However, profiler flame graphs are interpreted differently because the data they represent is different. The profiler flame graph pictured below represents CPU Time by method.<\/p>\n
![\"Profiler](\"https:\/\/www.tothenew.com\/blog\/wp-ttn-blog\/uploads\/2025\/03\/Screenshot-from-2025-03-13-09-04-19.png\")
<\/a>Profiler flame graph with important details explained.<\/p><\/div>\n
\n- The x-axis represents the total CPU consumption.<\/li>\n
- Each horizontal bar is a frame.<\/li>\n
- Each frame represents a method. The frames are arranged from top to bottom, in the order that each method was called during a program\u2019s execution.<\/li>\n
- Each color represents a different package.<\/li>\n
- The top frame defined here is usually called the \u201croot frame\u201d and its value is the sum of the child frames. Comparing it to a pie chart, the root frame is the total pie and each stack trace represents a different piece of the pie.<\/li>\n
- The width of each frame corresponds to its resource consumption. The longer the frame, the more CPU Time was used.<\/li>\n
- Two methods appearing side by side could have been called in parallel or in any order. Frames are ordered alphabetically from left to right.<\/li>\n
- The bottom frame is called the leaf frame and represents the last method called in the stack. The leaf frame only represents its self time because it has no child frames.<\/li>\n<\/ul>\n
Breaking Down Resource Consumption Using Profile Types<\/h2>\n
There are a variety of profile types that you can view in Continuous Profiler to help you understand and investigate different aspects of your code\u2019s performance.<\/p>\n
![\"Example](\"https:\/\/www.tothenew.com\/blog\/wp-ttn-blog\/uploads\/2025\/03\/Screenshot-from-2025-03-13-09-29-25.png\")
<\/a>Example of available profile types based on the filtered profile data.<\/p><\/div>\n
Understanding Profile Types and Resource Consumption<\/h2>\n
Each profile type represents a type of resource consumption, such as CPU, wall time, or memory. The profile types available for you to select may differ based on the language being profiled. The following are the most common profile types:<\/p>\n
\n- CPU<\/strong> profiling measures which methods consume the most CPU on an application.<\/li>\n
- Allocation<\/strong> profiling measures the amount of memory allocated by a given method.<\/li>\n
- Heap<\/strong> profiling measures the amount of heap memory allocated by each function that hasn\u2019t been garbage collected. This is useful for investigating the overall memory usage of your service and identifying potential memory leaks.<\/li>\n
- Lock<\/strong> profiling measures the amount of time a thread is waiting to acquire a lock and is hence doing nothing.<\/li>\n
- Wall Time<\/strong> profiling measures the effective time spent by methods. It can be useful to debug latency at first glance and then dig into the other profiling types to find out what was causing the latency. The wall time profile can be considered to be the most similar to the associated APM flame graph.<\/li>\n
- File I\/O and Socket I\/O<\/strong> measure the number of time spent by methods on disk.<\/li>\n
- Exceptions<\/strong> measure the amount of exceptions thrown. The profiler doesn\u2019t catch\/handle exceptions, but it tracks their creation.<\/li>\n<\/ul>\n
Investigating Slow Endpoints with Endpoint Profiling<\/h2>\n
Endpoint profiling allows you to scope profiler flame graphs by any API endpoint of a service, so you can find endpoints that are slow and causing poor end-user experience. Debugging and understanding why an endpoint has high latency can be tricky. For example, high latency could be caused by a method that is CPU-heavy and unknowingly on the critical path of a request process where latency is important.<\/p>\n
You can do the following with endpoint profiling:<\/p>\n
\n- Identify the bottleneck methods that are slowing down the endpoint\u2019s overall response time.<\/li>\n
- Isolate the top endpoints that are responsible for consuming resources like CPU and memory. This is particularly helpful when you\u2019re trying to optimize your service for performance gains.<\/li>\n
- Understand if third-party code or runtime libraries are the reason for endpoints being slow or heavy on resource consumption.<\/li>\n<\/ul>\n
Continuous profiling in Datadog<\/h2>\n<\/a>Example of a CPU Time profiler flame graph in Continuous Profiler.<\/p><\/div>\n
Datadog Continuous Profiler is an always-on, production code profiler that enables you to analyze code-level performance across your entire environment, with minimal overhead. Profiles reveal which methods\/functions consume the most resources, such as CPU, memory allocation, wall time, and I\/O time spent. With this information, you can optimize your code to reduce end-user latency and cloud provider costs.<\/p>\n
Continuous profiling in Datadog allows you to do the following:<\/p>\n
\n- Perform code-level tracing with zero instrumentation<\/li>\n
- Visualize all your stack traces in one place<\/li>\n
- Discover bottlenecks in your code at a glance<\/li>\n
- Filter profile data using tags<\/li>\n
- Get actionable insights for performance improvements<\/li>\n<\/ul>\n
Continuous Profiling Goes Beyond Distributed Tracing<\/h2>\n
Consider you have a service called movies-api-java that has multiple endpoints that return movie metadata stored in a database. You learn that the service isn\u2019t performing well and users are experiencing increased latency issues. You want to optimize the service\u2019s performance to ensure that users have the best experience.<\/p>\n
You start using Datadog APM and distributed tracing to investigate the performance issues. For example, if you see additional spans in traces for unnecessary, repeated calls to a database when requests are made to an endpoint, you know you can use this information to determine a fix. However, for some issues, you see that you need to go beyond distributed tracing to be able to diagnose issues and optimize performance. In the example below, the trace shows that it takes over 2 seconds to respond to a request, but there are no child spans or other indicators that can tell you more about what may be causing the slow performance.<\/p>\n
<\/p>\n
<\/a>Example of a high-latency trace with no spans.<\/p><\/div>\n
For issues that you can\u2019t solve using APM, you can use Continuous Profiler to investigate at a more granular level. The difference between distributed tracing and continuous profiling is that traces tell you which requests were slow at the service level, whereas profiles tell you why they were slow at the code level.<\/p>\n
Profiles<\/h2>\n
In Datadog, trace data and profiling data are automatically linked for application processes that have both APM and Continuous Profiler enabled. In the trace details panel in APM, you can move directly from the information for a trace or a selected span in a trace to the associated profiling data using the Profiles tab.<\/p>\n
<\/a>The Profiles UI found in the details panel for a trace.<\/p><\/div>\n
Interpreting the Profiler Flame Graph<\/h2>\n
Similar to traces, profiles are represented in flame graphs. However, profiler flame graphs are interpreted differently because the data they represent is different. The profiler flame graph pictured below represents CPU Time by method.<\/p>\n
<\/a>Profiler flame graph with important details explained.<\/p><\/div>\n
\n- The x-axis represents the total CPU consumption.<\/li>\n
- Each horizontal bar is a frame.<\/li>\n
- Each frame represents a method. The frames are arranged from top to bottom, in the order that each method was called during a program\u2019s execution.<\/li>\n
- Each color represents a different package.<\/li>\n
- The top frame defined here is usually called the \u201croot frame\u201d and its value is the sum of the child frames. Comparing it to a pie chart, the root frame is the total pie and each stack trace represents a different piece of the pie.<\/li>\n
- The width of each frame corresponds to its resource consumption. The longer the frame, the more CPU Time was used.<\/li>\n
- Two methods appearing side by side could have been called in parallel or in any order. Frames are ordered alphabetically from left to right.<\/li>\n
- The bottom frame is called the leaf frame and represents the last method called in the stack. The leaf frame only represents its self time because it has no child frames.<\/li>\n<\/ul>\n
Breaking Down Resource Consumption Using Profile Types<\/h2>\n
There are a variety of profile types that you can view in Continuous Profiler to help you understand and investigate different aspects of your code\u2019s performance.<\/p>\n
<\/a>Example of available profile types based on the filtered profile data.<\/p><\/div>\n
Understanding Profile Types and Resource Consumption<\/h2>\n
Each profile type represents a type of resource consumption, such as CPU, wall time, or memory. The profile types available for you to select may differ based on the language being profiled. The following are the most common profile types:<\/p>\n
\n- CPU<\/strong> profiling measures which methods consume the most CPU on an application.<\/li>\n
- Allocation<\/strong> profiling measures the amount of memory allocated by a given method.<\/li>\n
- Heap<\/strong> profiling measures the amount of heap memory allocated by each function that hasn\u2019t been garbage collected. This is useful for investigating the overall memory usage of your service and identifying potential memory leaks.<\/li>\n
- Lock<\/strong> profiling measures the amount of time a thread is waiting to acquire a lock and is hence doing nothing.<\/li>\n
- Wall Time<\/strong> profiling measures the effective time spent by methods. It can be useful to debug latency at first glance and then dig into the other profiling types to find out what was causing the latency. The wall time profile can be considered to be the most similar to the associated APM flame graph.<\/li>\n
- File I\/O and Socket I\/O<\/strong> measure the number of time spent by methods on disk.<\/li>\n
- Exceptions<\/strong> measure the amount of exceptions thrown. The profiler doesn\u2019t catch\/handle exceptions, but it tracks their creation.<\/li>\n<\/ul>\n
Investigating Slow Endpoints with Endpoint Profiling<\/h2>\n
Endpoint profiling allows you to scope profiler flame graphs by any API endpoint of a service, so you can find endpoints that are slow and causing poor end-user experience. Debugging and understanding why an endpoint has high latency can be tricky. For example, high latency could be caused by a method that is CPU-heavy and unknowingly on the critical path of a request process where latency is important.<\/p>\n
You can do the following with endpoint profiling:<\/p>\n
\n- Identify the bottleneck methods that are slowing down the endpoint\u2019s overall response time.<\/li>\n
- Isolate the top endpoints that are responsible for consuming resources like CPU and memory. This is particularly helpful when you\u2019re trying to optimize your service for performance gains.<\/li>\n
- Understand if third-party code or runtime libraries are the reason for endpoints being slow or heavy on resource consumption.<\/li>\n<\/ul>\n
<\/a>Example of a CPU Time profiler flame graph in Continuous Profiler.<\/p><\/div>\n
Datadog Continuous Profiler is an always-on, production code profiler that enables you to analyze code-level performance across your entire environment, with minimal overhead. Profiles reveal which methods\/functions consume the most resources, such as CPU, memory allocation, wall time, and I\/O time spent. With this information, you can optimize your code to reduce end-user latency and cloud provider costs.<\/p>\n
Continuous profiling in Datadog allows you to do the following:<\/p>\n
- \n
- Perform code-level tracing with zero instrumentation<\/li>\n
- Visualize all your stack traces in one place<\/li>\n
- Discover bottlenecks in your code at a glance<\/li>\n
- Filter profile data using tags<\/li>\n
- Get actionable insights for performance improvements<\/li>\n<\/ul>\n
Continuous Profiling Goes Beyond Distributed Tracing<\/h2>\n
Consider you have a service called movies-api-java that has multiple endpoints that return movie metadata stored in a database. You learn that the service isn\u2019t performing well and users are experiencing increased latency issues. You want to optimize the service\u2019s performance to ensure that users have the best experience.<\/p>\n
You start using Datadog APM and distributed tracing to investigate the performance issues. For example, if you see additional spans in traces for unnecessary, repeated calls to a database when requests are made to an endpoint, you know you can use this information to determine a fix. However, for some issues, you see that you need to go beyond distributed tracing to be able to diagnose issues and optimize performance. In the example below, the trace shows that it takes over 2 seconds to respond to a request, but there are no child spans or other indicators that can tell you more about what may be causing the slow performance.<\/p>\n
<\/p>\n
<\/a>Example of a high-latency trace with no spans.<\/p><\/div>\n
For issues that you can\u2019t solve using APM, you can use Continuous Profiler to investigate at a more granular level. The difference between distributed tracing and continuous profiling is that traces tell you which requests were slow at the service level, whereas profiles tell you why they were slow at the code level.<\/p>\n
Profiles<\/h2>\n
In Datadog, trace data and profiling data are automatically linked for application processes that have both APM and Continuous Profiler enabled. In the trace details panel in APM, you can move directly from the information for a trace or a selected span in a trace to the associated profiling data using the Profiles tab.<\/p>\n
<\/a>The Profiles UI found in the details panel for a trace.<\/p><\/div>\n
Interpreting the Profiler Flame Graph<\/h2>\n
Similar to traces, profiles are represented in flame graphs. However, profiler flame graphs are interpreted differently because the data they represent is different. The profiler flame graph pictured below represents CPU Time by method.<\/p>\n
<\/a>Profiler flame graph with important details explained.<\/p><\/div>\n
- \n
- The x-axis represents the total CPU consumption.<\/li>\n
- Each horizontal bar is a frame.<\/li>\n
- Each frame represents a method. The frames are arranged from top to bottom, in the order that each method was called during a program\u2019s execution.<\/li>\n
- Each color represents a different package.<\/li>\n
- The top frame defined here is usually called the \u201croot frame\u201d and its value is the sum of the child frames. Comparing it to a pie chart, the root frame is the total pie and each stack trace represents a different piece of the pie.<\/li>\n
- The width of each frame corresponds to its resource consumption. The longer the frame, the more CPU Time was used.<\/li>\n
- Two methods appearing side by side could have been called in parallel or in any order. Frames are ordered alphabetically from left to right.<\/li>\n
- The bottom frame is called the leaf frame and represents the last method called in the stack. The leaf frame only represents its self time because it has no child frames.<\/li>\n<\/ul>\n
Breaking Down Resource Consumption Using Profile Types<\/h2>\n
There are a variety of profile types that you can view in Continuous Profiler to help you understand and investigate different aspects of your code\u2019s performance.<\/p>\n
<\/a>Example of available profile types based on the filtered profile data.<\/p><\/div>\n
Understanding Profile Types and Resource Consumption<\/h2>\n
Each profile type represents a type of resource consumption, such as CPU, wall time, or memory. The profile types available for you to select may differ based on the language being profiled. The following are the most common profile types:<\/p>\n
- \n
- CPU<\/strong> profiling measures which methods consume the most CPU on an application.<\/li>\n
- Allocation<\/strong> profiling measures the amount of memory allocated by a given method.<\/li>\n
- Heap<\/strong> profiling measures the amount of heap memory allocated by each function that hasn\u2019t been garbage collected. This is useful for investigating the overall memory usage of your service and identifying potential memory leaks.<\/li>\n
- Lock<\/strong> profiling measures the amount of time a thread is waiting to acquire a lock and is hence doing nothing.<\/li>\n
- Wall Time<\/strong> profiling measures the effective time spent by methods. It can be useful to debug latency at first glance and then dig into the other profiling types to find out what was causing the latency. The wall time profile can be considered to be the most similar to the associated APM flame graph.<\/li>\n
- File I\/O and Socket I\/O<\/strong> measure the number of time spent by methods on disk.<\/li>\n
- Exceptions<\/strong> measure the amount of exceptions thrown. The profiler doesn\u2019t catch\/handle exceptions, but it tracks their creation.<\/li>\n<\/ul>\n
Investigating Slow Endpoints with Endpoint Profiling<\/h2>\n
Endpoint profiling allows you to scope profiler flame graphs by any API endpoint of a service, so you can find endpoints that are slow and causing poor end-user experience. Debugging and understanding why an endpoint has high latency can be tricky. For example, high latency could be caused by a method that is CPU-heavy and unknowingly on the critical path of a request process where latency is important.<\/p>\n
You can do the following with endpoint profiling:<\/p>\n
- \n
- Identify the bottleneck methods that are slowing down the endpoint\u2019s overall response time.<\/li>\n
- Isolate the top endpoints that are responsible for consuming resources like CPU and memory. This is particularly helpful when you\u2019re trying to optimize your service for performance gains.<\/li>\n
- Understand if third-party code or runtime libraries are the reason for endpoints being slow or heavy on resource consumption.<\/li>\n<\/ul>\n
- Allocation<\/strong> profiling measures the amount of memory allocated by a given method.<\/li>\n
- CPU<\/strong> profiling measures which methods consume the most CPU on an application.<\/li>\n