Menu
503 Service Unavailable Jump to solution. 2020-05-08T12:58:34.733Z tomcat-http-39 vsphere.local 97b81203-5e98-4499-b11f-1f5b6ade0789 INFO com.vmware.identity.sts. For tomcat the user and group are root/root and for httpd, the user and group are apache/apache Am I missing something or am I doing it wrong? Httpd version is 2.4.6 and Tomcat's version is 8.0.
order
This paper focuses on the 502, 504, 503 errors of nginx and Tomcat and their common causes.
- You might want to check the Tomcat catalina.out and localhost log files to make sure that port 8009 didn't fail to open. Although your proxy log message seems to be indicating that Apache might have been trying to connect using IPV6 and your allowable address (127.0.0.1) is an IPv4 address. If that's the case, I'd try and en.
- Tomcat will not start and complains that port 8080 is already in use. When accessing the site via a web browser you get a '503 - Service Unavailable' error.
- If yes, I am recommending to check Tomcat logs, there some exceptions are thrown and can help with investigation. 503 Service Unavailable error.
502
Definition
502 bad gateway: a server working as a gateway or proxy receives an invalid response from the upstream server when it attempts to execute a request.
Common causes
- If the backend service is suspended, 502
- Back end services are restarting
Example
Turn off the back-end service and send the request back-end interface to nginx. The log is as follows:
- access.log
- error.log
504
Definition
504 gateway timeout: when a server working as a gateway or proxy attempts to execute a request, it fails to receive a response in time from the upstream server (the server identified by the URI, such as HTTP, FTP, LDAP) or the secondary server (such as DNS). Note: some proxy servers will return 400 or 500 errors when DNS query is over time
Common causes
- The interface is too time-consuming. The back-end service receives the request and starts execution. It fails to return data to nginx at the set time
- The overall load of the back-end server is too high. After receiving the request, due to the busy thread, it failed to arrange the interface for the request, resulting in the failure to return data to nginx at the set time
Example
- Front end return
- access.log
- error.log
- nginx.conf
- java code
The server accepts the request and does not return it. Nginx reports connection timed out after waiting for 100 seconds and returns 504. However, the backend continues to execute and does not complete until the 120th second.
503(Relatively rare
)
Definition
503 service unavailable: indicates that the server is currently in a temporary unavailable state. Whether intentionally or unintentionally, the status code will be returned when the server is in a state that cannot be answered. Among them, it is intentional for the server to stop service due to maintenance needs. When the server itself is overloaded and in a state of unresponsiveness, it is unintentional. In addition, servers in load balancer or front end of web server may return 503
Common causes
- Nginx to limit current, over the speed limit to return to 503
- Back end services for routine maintenance, such as pause Tomcat
Nginx current limiting returns 503 instances
- config
- error.log
- access.log
- client
Tomcat returns 503 instances
- Http11Processor
tomcat-embed-core-8.5.23-sources.jar!/org/apache/coyote/http11/Http11Processor.java
503 as long as the endpoint status is paused
- AbstractEndpoint
tomcat-embed-core-8.5.23-sources.jar!/org/apache/tomcat/util/net/AbstractEndpoint.java
Here are the pause and resume methods of endpoint
- request
When the request enters the service method of http11processor to execute the endpoint. Ispaused() method, Tomcat is paused, and 503 will be returned as follows:
Summary
- 502
Usually the back-end service hangs or restarts
- 504
It is usually the execution time of the requested interface, or the execution time of the back-end service is high
- 503
It is usually maintained by nginx current limiting or back-end service pause
doc
- Summary of nginx status codes
- Analysis of nginx error 502503504
- Summary of trigger conditions and solutions for nginx 502 503 errors
0
reset
Hi krish,
I think this topic can help you to understand :
regards,
mathieu
0
reset
Does the Error trace indicate the error is coming from BW? If you have the complete error message, then you may be able to see the reference to bw or tomcat code lines. As per your new response, there is a load balancer involved. Its possible that the load balancer itself is returning 503 error. So, probably with tracing, you can see if the error is coming from BW or LB. One thing is 100 requests/sec, is not same as 100 concurrent requests. Do you mean 100 concurrent requests. In that case, if the intent is to have 100 requests processed simultaneously, you should not be setting maxjobs and flowlimit. For HTTP starters, flowlimit is probably redundant or may not work properly as this can be easily controlled through max/min proc setting. Are these requests involving large payloads?
Few things to try out
- Try to bypass load balancer and hit the bw service directly and see if you see errors
- remove the flowlimit/maxjobs
- If you have more concurrent requests than maxproc, acceptCount will come into picture, but the acceptCount requires some tweaks to TCP settings on some platforms. If the maxproc is limiting, then you should see an error in BW logs. Try increasing maxprocessors assuming that acceptCount is not working on this platform.
- Try to use httpComponents instead of tomcat for the receiver. HttpComponents is better for high concurrency, but its less performant compared to tomcat for lower concurrency. But it may still be in your limits, its just relative observation.
- I am not sure if you are getting few 503 errors or all of them are getting errored out. If its the case, try httpComponents again. There is a workaround if the issue is specific to tomcat.
0
reset
Dear Mathieu,
You are right, we wanted to catch this error or record the timestamp of Tomcat logs to note exactly what time the request reached Tomcat.
This will clearly give us an indication whether it is problem of Tomcat not able to process the load or some nework delay happening in between. There is a hardware loadbalancer sitting between Client's Application Server and Tibco Tomcat server.
Your comments are welcome on this.
BR
Krish
0
reset
Your comments on this.
The answer of the support is correct regarding your question on the 'trace' configuration.
My understanding was that you wanted to catch the 503 error in your BW process that expose the service, not to make more logs. But I might have misunderstood your needs.
regards,
mathieu
0
reset
Dear Mathieu,
Apache Tomcat 503 Service Temporarily Unavailable
We received this response from TIBCO for tracking tomcat server:
-------------------------------------------------------------------------------------------------------------------------------------------
How to configure the debug tracing for tomcatwebserver in BW?
Body: |
We can open the debug tracing for tomcat and our BWengine.
Youcan change the log4j.properties which locatesTIBCO_HOME/bw/BW_VERSION/lib/log4j.properties to enable the tomcat debugtracing. Just append below lines in it and restart BW. Then you can find theTomcat debug information.
log4j.logger.com.tibco.bw.service.binding.bwhttp.tomcat.TomcatServer=DEBUG,tibco_bw_log
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost]=DEBUG,tibco_bw_log
log4j.logger.org.apache.catalina.core=DEBUG,tibco_bw_log
log4j.logger.org.apache.catalina.session=DEBUG,tibco_bw_log
Please becare of these debug tracing properties, they will generate much more log thanbefore.
For BWengine, please add below proerties into the deployed tra file.
Trace.Startup=true
Trace.JC.*=true
Trace.Engine=true
Trace.Task.*=true
Trace.Info.*=true
Trace.Warn.*=true
Trace.Error.*=true
Install Tomcat Service
Trace.Debug.*=true
Tomcat 503 Service Unavailable
bw.plugin.http.server.debug=true
-------------------------------------------------------------------------------------------------------------------------------------------------------
Tomcat 503 Service Temporarily Unavailable
Your comments on this.
BR
Krish