Search

Gathering Empirical Data for Troubleshooting

Amanda Jennewein
Amanda Jennewein
  • Updated

Author: Amanda Jennewein

Updated:  September 2024

Audience: Everyone

Environment: Self-Hosted 

Summary

This document is intended for customers who host Jama Connect® on their servers. If you are a Cloud/SaaS customer, you will not have access to Jama's servers. For Cloud/SaaS customers, only the Browser Developer Tools section will be helpful. Once you have the file, please attach it to a support ticket on Jama's support portal.

Jama Software® does not offer support for installing, administrating, or using the tools listed in this document. However, we encourage their use to help troubleshoot technical issues.

Customer-Side Tools

How To Collect HAR files for Support Troubleshooting

If you encounter a situation where Jama Connect®'s performance is affected by external factors, you can generate a HAR file to obtain detailed data on the issue. While recording a HAR file, we recommend using the features of Jama Connect® that may be causing the relevant activity. For instance, if you receive an error message in the Review Center, perform the actions that led to the error message. 

A HAR file contains all the information recorded during a session, including private or personally identifiable information such as cookies, passwords, credit card numbers, etc. Anyone accessing the HAR file can impersonate your account and access your data. All network traffic that occurred during the session is in the HAR file. If we ask for a HAR file from you, we recommend you open it with a text editor and delete or obfuscate any sensitive information you submitted through a form. It's important to note that customers are responsible for any data they provide to Jama Software®, including personal details, passwords, credit card numbers, etc. To protect sensitive information, you can navigate to the " cookies " section in the file and delete any lines containing sensitive information, such as session IDs.

Google Chrome
Depending on your browser version, the path names may vary to open the console:

  1. You can navigate to the area in Jama Connect® where you will perform the slow or faulty action.
  2. From the Chrome menu (top right), select Tools > Developer tools or More tools > Developer tools as applicable.
  3. Alternatively, you can right-click anywhere and choose Inspect Element.
    1. Select the Network tab.
    2. Select the red record icon.
    3. Select the Preserve log check box to save network transaction data if applicable.
  4. Before trying to reproduce the issue, select the Clear icon to remove unnecessary header information.
  5. Perform the action that causes or suffers from the issue.
  6. Select the button, run through the reuse process, open a review, etc.
  7. Once you experience the issue, you can click on any items within the Network tab, select Save all as HAR with Content (preferred), or take a screenshot of all the requests.
  8. Please upload your HAR file to your ticket or attach it to your email so our Support team can review it.

Firefox
To capture data with the network tab in Firefox:

  1. From the application menu, select Tools > Web Developer > Network, or press Ctrl+Shift+I (Windows/Linux) or Cmd+Option+I (OS X).
  2. Select the Clear option (lower-right corner of the Developer window) to remove unnecessary header information.
  3. The recording auto-starts when you start performing actions in the browser.
  4. Navigate to the place in Jama Connect where you will perform the slow or faulty action.
  5. Perform the action that causes or suffers from the issue.
  6. Capture the information by right-clicking the transaction list in the console and selecting either Copy All As HAR to place the data in your buffer or Save All As HAR to copy it to a file.
  7. Please upload your HAR file to your ticket or attach it to your email so Support can analyze it.

Internet Explorer

  1. Please open Internet Explorer and navigate to the Jama Connect page where the issue occurs.
  2. Press F12 on your keyboard (or click the gear icon > F12 Developer Tools).
  3. Click the Network tab.
  4. While recording network requests, could you reproduce the problem you were having before?
  5. Once done, click the Save button.
  6. Give the trace a file name, then click the Save button to save it as a .har or .xml file.
  7. Upload your HAR file to your ticket or attach it to your email so that we may analyze it.

Microsoft Edge

    1. Open Edge and go to the Jama Connect page where the issue occurs.
    2. Open the menu in the top-right > More tools > Developer Tools > Network.
    3. Navigate to the place where you will perform the slow or faulty action.
    4. Refresh the page to start capturing the traffic between the browser and the server, or click on a link for the affected page.
    5. Perform the action that causes or suffers from the issue.
    6. Click on the button, run through the reuse process, open a review, etc.
    7. Press Ctrl + S or select the Export as HAR option in the top-right corner (designed to look like a floppy disk) and Save As HAR file.
    8. Upload your HAR file to your ticket or attach it to your email so we can analyze it.

Apple Safari

  1. Enable the Develop menu.
    1. Safari > Preferences > Advanced > Select Show Develop menu in the menu bar.
  2. Select Develop > Show Web Inspector > Select Network tab.
  3. Within the Network tab, check the Preserve log option.
  4. Navigate to the place where you will perform the slow or faulty action.
  5. Start capturing traffic by refreshing the page or clicking a link for the affected site.
  6. Perform the action that causes or suffers from the issue.
    1. Click on the button, run through the reuse process, open a review, 
  7. Once the page is loaded, select Export at the top-right in the network tab window.
  8. Upload your HAR file to your ticket or attach it to your email so that we may analyze it.

When This Is Useful?

  • Network output is valuable whenever a web page or its elements load slowly. It can also help identify the backend code being called during a specific action taken on the front end. This information will allow our Support Engineers to investigate the issue more thoroughly. I'd like to point out that you should include this message as part of the ticket submission process.
  • The Javascript console is helpful anytime data or the UI displays abnormally, items or data appear missing, or clicking on buttons does not appear to do anything. Generally speaking, you should also include this output in any ticket.

Server-Side Tools

Support BundlesA support bundle contains all logs we would typically request in Support and records for Docker and Replicated, comprehensive system information, and all configurations and environment variables used in a customer's instance. There are two ways to generate a support bundle: the Replicated web portal user interface (UI) or the command line interface (CLI).

Using the UI
This is the easiest way to generate a support bundle and is the method you will use most often.

  1. You can log into the Replicated portal for your Jama instance.
    1. This will be located at https://<IP or hostname>:8800
  2. Select the Support tab.
  3. Select the Download Support Bundle button to download the bundle to your local machine.
    1. You can find information on generating the Support bundle from the UI here.

Using the CLI
If you cannot access the Replicated UI, you can still download the support bundle from the Replicated CLI by running the following commands on the server:

replicated apps
replicated support-bundle <app_id>

Where <app_id> is the ID of your application taken from the output of the first command. This is available in versions of Replicated 1.2.73 and higher.

Note: If your self-hosted console has a password, you must authenticate on the CLI.

Thread Dumps
For details on taking thread dumps, please review our knowledge article on Performing a Thread Dump in Linux or Windows.

In the case of application-wide performance degradation affecting all users in a self-hosted installation, thread dumps may be critical to understanding what is occurring within the application at the time of the issue. A thread (short for "thread of execution") is generally a subset of instructions (code) that is designed to run concurrently (i.e., in parallel) to other code.

Threads are essential to a computer's ability to multitask or execute multiple operations simultaneously. For instance, you can stream music and work on a document simultaneously because it breaks down sequences of functions into separate threads of execution. It allows the computer to switch between them quickly, making it appear like they are happening simultaneously. A thread dump is a snapshot of the application's state when the dump is initiated.

To effectively identify and resolve issues with an application, it is crucial to take thread dumps while the problem occurs. By taking multiple dumps over a short interval, we can determine the application's actions during that time frame and pinpoint specific threads that may be taking too long or getting stuck. Particular processes may be expected to run for extended periods.

To capture thread dumps in Jama, you need direct access to the server where they're being hosted. The method of capturing will depend on the server's OS. Typically, valuable data requires 3-5 thread dumps with a time interval of 3-5 seconds (totaling 9-15 seconds). If you want to collect samples for a longer period, taking more dumps is recommended instead of increasing the time interval.

When is This Useful?
The information provided by thread dumps can be helpful in many cases. However, it may only sometimes be relevant to some situations. Thread dumps are beneficial when investigating performance-related issues, specific memory usage problems, and issues that don't generate error messages or exceptions in the logs. In general, if Jama Support deems thread dumps necessary to troubleshoot your specific problem, they will direct you to gather them.
However, there are a couple of instances where gathering them before filing a ticket can be helpful:

  • If the application has stopped responding but the Tomcat process is still running, you must obtain thread dumps before restarting the machine.
    • Gathering thread dumps before restarting the application is crucial for effective troubleshooting or RCA in case of a crash.
  • When a specific functionality is slow for all users, regardless of other factors, capturing thread dumps while a user is performing that action can help identify the cause of the slowness.
  • For site-wide performance issues, thread dumps captured at any time can help provide a picture of what is happening within the application. Support may request more specific thread dumps as needed.

Heap Dumps
A heap dump is a snapshot of all objects currently stored in memory. It requires more effort to generate than gathering a log file or thread dumps and can be quite cumbersome to deal with. However, it can help troubleshoot memory-related issues. It's important to note that generating a heap dump will stop the Java Virtual Machine for the time it takes to create the file. This effectively pauses the Jama application during that time, so Jama Support will only request heap dumps when necessary.

We recommend using the jmap command built into the Oracle JDK to generate a heap dump. To run this, execute the following command in a command prompt (Windows) or shell (Linux):

  • Linux
    jmap -dump:format=b,file=heap.bin <pid>

  • Windows 
    %JAVA_HOME%\bin\jmap -dump:format=b,file=heap.bin <pid>

In both cases, <pid> would be the process ID for the Java process running Tomcat. This command generates a file in the current directory that is at least as large as the heap memory allocated. Ensuring that the directory has enough space to accommodate the file is crucial. Jama Support will send you a link to store your file securely in the cloud.


When is This Useful?
Heap dumps can help identify memory issues that have already been identified by analyzing the Garbage Collection logs or OutOfMemory errors in the contour.log. Normally, heap dumps should only be gathered when Jama Support is told directly through a ticket. This is because heap dumps can be helpful only in limited scenarios, and gathering them can be costly in terms of overhead and impact. Therefore, we recommend against preemptively gathering heap dumps before filing a ticket.

Wireshark (Cross-Platform) or TCPDump (Linux)

These "packet sniffer" tools intercept and read network packets during transmission. Although Jama Software doesn't support third-party network infrastructure or commit to troubleshooting it, these tools can help us gather additional data regarding the state of a network connection between the client and the server. This information can help determine whether the issue is related to Jama Connect or the network itself.


When is This Useful?
Suppose Jama cannot communicate with external services such as an email server or an integration provider like Planview (formerly Tasktop), Jira, Rally, TFS, etc. In that case, it may be due to a network issue. In such cases, packet information can help identify the location of the network problem. However, preemptively gathering packet dumps is only sometimes necessary for some issues. Jama Support will request them as needed.

Monitoring Tools

A wide range of application monitoring tools are available, both free and paid. Jama Software uses multiple monitoring solutions to ensure the stability and uptime of its Cloud solution. While Jama Software does not support third-party tools, including monitoring solutions, self-hosted customers are encouraged to investigate deploying real-time monitoring solutions for their Jama application. For customers who implement a monitoring solution, providing data to Jama Support from your monitoring tools when reporting intermittent or performance-related issues can be an invaluable auxiliary data source.

Database Tools

SQL Server
The Microsoft Technet and Microsoft Developer Network websites have a collection of articles detailing tools and methods to assist in troubleshooting database issues. Jama Support may recommend configurations or changes from this list, including (but not limited to) enabling or disabling trace flags or additional logging. You can view the full lists at https://technet.microsoft.com/en-us/library/bb510669(v=sql.105).aspx and at https://msdn.microsoft.com/en-us/library/ms189081(v=sql.110).aspx.

MySQL
The MySQL performance schema and the slow query log can help track database performance issues. Jama Support may request information from these sources if we determine that the problem may lie on the database server. Depending on the situation, you can also request additional logs or data.

Support Bundles 

To learn about Support Bundles and what they contain, visit this companion article: Jama Support Bundle: How to Obtain it, Contents, and Relevant Logs

Additional Resources:

 

Related to

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request

Comments

0 comments

Article is closed for comments.