iRawwwN 282 Posted August 22, 2023 (edited) fix the servers u guys its hard to hit Edited August 23, 2023 by iRawwwN 1 Quote Share this post Link to post Share on other sites
BlatMan 711 Posted August 23, 2023 The latency is not the problem. The problem is between APB's servers and the DDoS protection run by Path Network Inc. Also, the in game latency is better described as a server population counter. It's impossibly low when the servers are empty. It slowly goes up as the population grows, but it's always lower than the real latency. There is packet loss or latency jitter on the upload packets from client (our PC) to server (APB District). Sometimes shots do not register, sometimes there's excessive delay, like 2s between shooting and getting a hit marker. Sometimes ghost shot, as in firing client sided but not server sided. Occasionally, the hit registration and firing is perfect. For me, every single time the hit registration was perfect on a US West server, the server went offline within a few hours. My issues are only on US servers. I have no issues on EU servers currently. In-game packet loss always displays 0.00. This counter only counts download packet loss, or server to client. Scanning APB's packets with Wireshark, the incoming packets are in numerical order and appear to be in line with the server tickrate. This means there is most likely no issues with packets going from the server to the client. I had support reach out to me a while ago. They asked to run a trace route. The trace route shows near perfect response time to the server's IP address. I also got no further responses. These are the same symptoms experienced during Reloaded Production's ownership. Back then, the servers were constantly under a DDoS attack. Reloaded would disable the DDoS protection after weekly maintenance. The hit registration was perfect, but the servers would go offline within the same day, sometimes minutes after maintenance. Reloaded would then enable the DDoS protection. The hit registration was terrible, but the servers would stay online. I didn't play much on EU, but I did play the Russian version with little to no issues with hit registration. Support staff created a pinned topic for players to post trace routes. There were no issues with the player's routes, and the problem was never fixed. Running a trace route is pointless for 99% of users. It only shows replies to ICMP packets with their time to live (TTL) value exceeded. APB uses UDP packets to communicate. ICMP can be handled differently than UDP. For example, Path's nodes do not respond to UDP packets with a TTL value of 0. This is most likely part of their DDoS mitigation. ICMP and UDP can be sent to different routers. The router responding to ICMP can be fine, but the one handling UDP can be overloaded. The only way to check is to monitor the game's traffic both client and server sided, with timestamps within the packets to determine if there's extra latency between packets. The in-game /latencytest does not do this. It runs a trace route to a US APB server, checks real latency between client and server, and shows the server's tickrate. The latency shown in /latencytest is always higher than what is displayed in game. Don't get me started on how the servers fail to run at 30tps. That's for another thread. Quote Share this post Link to post Share on other sites
iRawwwN 282 Posted August 23, 2023 (edited) 13 hours ago, BlatMan said: snip the 127ms latency while high is not the issue, its the 35-50+ server ms that is the issue and the fact the server does not auto-restart and/or find new routing sucks balls maybe we need to spend more money on JT and name changes edit; this was on EU fin and EU WF last night... WF was 34 at the start but after like 5 missions was up to 39. matt PLEASE Edited August 23, 2023 by iRawwwN 1 Quote Share this post Link to post Share on other sites
BlatMan 711 Posted August 25, 2023 I tried using Cloudflare WARP to bypass as much of my ISP's routing as possible. I can login, but I can't connect to any district. Is Cloudflare WARP VPN blocked? Quote Share this post Link to post Share on other sites
BlatMan 711 Posted August 26, 2023 (edited) The connection to US East is screwed up internally. This is obviously not routing to a US East location. Did you start the district server in the wrong datacenter? Regardless, that doesn't explain the +70ms variance and these random +200ms latency spikes. District: Missions-Financial-1-US-East IP Address: 205.220.229.40 In-game Latency: 30ms to 90ms Edit. Added Cloudflare traceroute to prove it's not an ISP issue. Obviously traceroutes don't show the whole picture, but they do show that both my ISP and Cloudflare WARP VPN are functionally the same. Both route to gtt.net on the west coast and then to 205.220.229.40. Cloudflare Trace Route: 1 9 ms 12 ms 11 ms 104.28.0.0 2 9 ms 11 ms 12 ms 104.29.106.1 3 20 ms 17 ms 8 ms 162.158.156.2 4 11 ms 11 ms 12 ms ae5.cr2-nyc4.ip4.gtt.net [208.116.240.225] 5 69 ms 70 ms 85 ms ae12.cr4-lax1.ip4.gtt.net [89.149.128.174] 6 * * * Request timed out. 7 72 ms 70 ms 69 ms 205.220.229.40 Home Trace Route: 2 4 ms <1 ms 5 ms lo0-100.CMDNNJ-VFTTP-325.verizon-gni.net [72.82.234.1] 3 11 ms 7 ms 7 ms B4325.NWRKNJ-LCR-22.verizon-gni.net [100.41.3.252] 4 * * * Request timed out. 5 12 ms 13 ms 11 ms customer.alter.net [152.193.0.114] 6 63 ms 64 ms 62 ms ae12.cr4-lax1.ip4.gtt.net [89.149.128.174] 7 * * * Request timed out. 8 63 ms 63 ms 64 ms 205.220.229.40 /latencytest District_Server: Line 5: 26/08/2023 18:59:09,DISTRICT_SERVER,,,70,,Packet loss:,0,CCU:,25,25 Line 17: 26/08/2023 18:59:10,DISTRICT_SERVER,,,70,,Packet loss:,0,CCU:,25,25 Line 26: 26/08/2023 18:59:11,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,25 Line 29: 26/08/2023 18:59:12,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,25 Line 38: 26/08/2023 18:59:13,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,25 Line 43: 26/08/2023 18:59:14,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,25 Line 46: 26/08/2023 18:59:15,DISTRICT_SERVER,,,93,,Packet loss:,0,CCU:,25,25 Line 55: 26/08/2023 18:59:16,DISTRICT_SERVER,,,93,,Packet loss:,0,CCU:,25,25 Line 60: 26/08/2023 18:59:17,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,25 Line 63: 26/08/2023 18:59:18,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,25 Line 72: 26/08/2023 18:59:19,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,25 Line 77: 26/08/2023 18:59:20,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,25 Line 80: 26/08/2023 18:59:21,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,25 Line 89: 26/08/2023 18:59:22,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,26 Line 94: 26/08/2023 18:59:23,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,26 Line 99: 26/08/2023 18:59:24,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,26 Line 106: 26/08/2023 18:59:25,DISTRICT_SERVER,,,93,,Packet loss:,0,CCU:,25,26 Line 111: 26/08/2023 18:59:26,DISTRICT_SERVER,,,93,,Packet loss:,0,CCU:,25,26 Line 115: 26/08/2023 18:59:27,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,26 Line 123: 26/08/2023 18:59:28,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,26 Line 128: 26/08/2023 18:59:29,DISTRICT_SERVER,,,70,,Packet loss:,0,CCU:,25,26 Line 132: 26/08/2023 18:59:30,DISTRICT_SERVER,,,70,,Packet loss:,0,CCU:,25,26 Line 140: 26/08/2023 18:59:31,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,26 Line 145: 26/08/2023 18:59:32,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,26 Line 152: 26/08/2023 18:59:33,DISTRICT_SERVER,,,93,,Packet loss:,0,CCU:,25,26 Line 157: 26/08/2023 18:59:34,DISTRICT_SERVER,,,93,,Packet loss:,0,CCU:,25,26 Line 162: 26/08/2023 18:59:35,DISTRICT_SERVER,,,253,,Packet loss:,0,CCU:,25,26 Line 169: 26/08/2023 18:59:36,DISTRICT_SERVER,,,253,,Packet loss:,0,CCU:,25,26 Line 174: 26/08/2023 18:59:37,DISTRICT_SERVER,,,93,,Packet loss:,0,CCU:,25,26 Line 179: 26/08/2023 18:59:38,DISTRICT_SERVER,,,93,,Packet loss:,0,CCU:,25,26 Line 186: 26/08/2023 18:59:39,DISTRICT_SERVER,,,93,,Packet loss:,0,CCU:,25,26 Line 191: 26/08/2023 18:59:40,DISTRICT_SERVER,,,93,,Packet loss:,0,CCU:,25,26 Line 196: 26/08/2023 18:59:41,DISTRICT_SERVER,,,93,,Packet loss:,0,CCU:,25,26 Line 203: 26/08/2023 18:59:42,DISTRICT_SERVER,,,93,,Packet loss:,0,CCU:,25,26 Line 208: 26/08/2023 18:59:43,DISTRICT_SERVER,,,70,,Packet loss:,0,CCU:,25,26 Line 213: 26/08/2023 18:59:44,DISTRICT_SERVER,,,70,,Packet loss:,0,CCU:,25,26 Line 220: 26/08/2023 18:59:45,DISTRICT_SERVER,,,70,,Packet loss:,0,CCU:,25,26 Line 225: 26/08/2023 18:59:46,DISTRICT_SERVER,,,70,,Packet loss:,0,CCU:,25,26 Line 230: 26/08/2023 18:59:47,DISTRICT_SERVER,,,70,,Packet loss:,0,CCU:,25,27 Line 237: 26/08/2023 18:59:48,DISTRICT_SERVER,,,70,,Packet loss:,0,CCU:,25,27 Line 242: 26/08/2023 18:59:49,DISTRICT_SERVER,,,70,,Packet loss:,0,CCU:,25,27 Line 247: 26/08/2023 18:59:50,DISTRICT_SERVER,,,70,,Packet loss:,0,CCU:,25,27 Line 253: 26/08/2023 18:59:51,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,27 Line 259: 26/08/2023 18:59:52,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,27 Line 264: 26/08/2023 18:59:53,DISTRICT_SERVER,,,70,,Packet loss:,0,CCU:,25,27 Line 271: 26/08/2023 18:59:54,DISTRICT_SERVER,,,70,,Packet loss:,0,CCU:,25,27 Line 276: 26/08/2023 18:59:55,DISTRICT_SERVER,,,66,,Packet loss:,0,CCU:,25,27 Line 281: 26/08/2023 18:59:56,DISTRICT_SERVER,,,66,,Packet loss:,0,CCU:,25,27 Line 288: 26/08/2023 18:59:57,DISTRICT_SERVER,,,62,,Packet loss:,0,CCU:,25,27 Line 293: 26/08/2023 18:59:58,DISTRICT_SERVER,,,62,,Packet loss:,0,CCU:,25,27 Line 298: 26/08/2023 18:59:59,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,27 Line 305: 26/08/2023 19:00:00,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,27 Line 310: 26/08/2023 19:00:01,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,27 Line 315: 26/08/2023 19:00:02,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,27 Line 322: 26/08/2023 19:00:03,DISTRICT_SERVER,,,93,,Packet loss:,0,CCU:,25,27 Line 327: 26/08/2023 19:00:04,DISTRICT_SERVER,,,93,,Packet loss:,0,CCU:,25,27 Line 332: 26/08/2023 19:00:05,DISTRICT_SERVER,,,93,,Packet loss:,0,CCU:,25,27 Line 338: 26/08/2023 19:00:06,DISTRICT_SERVER,,,93,,Packet loss:,0,CCU:,25,27 Line 344: 26/08/2023 19:00:07,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,27 Line 349: 26/08/2023 19:00:08,DISTRICT_SERVER,,,101,,Packet loss:,0,CCU:,25,27 Line 356: 26/08/2023 19:00:09,DISTRICT_SERVER,,,70,,Packet loss:,0,CCU:,25,27 Line 361: 26/08/2023 19:00:10,DISTRICT_SERVER,,,70,,Packet loss:,0,CCU:,25,27 ... Ping to 205.220.229.40 from PC, APB client during /latencytest: Reply from 205.220.229.40: bytes=32 time=66ms TTL=118 Reply from 205.220.229.40: bytes=32 time=74ms TTL=118 Reply from 205.220.229.40: bytes=32 time=60ms TTL=118 Reply from 205.220.229.40: bytes=32 time=64ms TTL=118 Reply from 205.220.229.40: bytes=32 time=60ms TTL=118 Reply from 205.220.229.40: bytes=32 time=62ms TTL=118 Reply from 205.220.229.40: bytes=32 time=66ms TTL=118 Reply from 205.220.229.40: bytes=32 time=62ms TTL=118 Reply from 205.220.229.40: bytes=32 time=65ms TTL=118 Reply from 205.220.229.40: bytes=32 time=66ms TTL=118 Reply from 205.220.229.40: bytes=32 time=66ms TTL=118 Reply from 205.220.229.40: bytes=32 time=61ms TTL=118 Ping to 205.220.229.40 from router during /latencytest: Min: 59ms Max: 75ms Avg: 63ms. Edited August 26, 2023 by BlatMan Added Cloudflare trace route Quote Share this post Link to post Share on other sites
BlatMan 711 Posted August 30, 2023 (edited) Edit: Tested each US East district server. IP address, port, and latency are taken from the /latencytest report. I only ran it for a few minutes. I restarted APB and rejoined Waterfront-1-US-East. The IP address is the same between Financial-1-US-East and Waterfront-1-US-East. There is only one Waterfront US-East and 1 Waterfront US-West, but there are two Financial US-East and two Financial US-West. Automatic district spin up is not working. Fight Club-Ambigton Towers-1-US-East: IP Address / Port: 205.220.227.73:12595 Latency: 31ms to 39ms Fight Club-Baylon Shipping Storage-1-US-East: IP Address / Port: 205.220.227.77:10467 Latency: 31ms to 39ms Missions-Financial-1-US-East: IP Address / Port: 205.220.227.75:10555 Latency: 23ms to 70ms Missions-Financial-2-US-East: IP Address / Port: 205.220.227.71:10563 Latency: 31ms to 39ms Missions-Waterfront-1-US-East: IP Address / Port: 205.220.227.75:12793 Latency: 265ms to 273ms Extra info. Here's my Original Post: It's even worse after maintenance. Did you fire up the servers in China? There's something wrong with your anti-DDoS provider, Path Network Inc, or your servers are starting in the wrong physical location. I don't see any incidents opened: https://status.path.net/history District: Missions-Waterfront-1-US-East IP Address: 205.220.227.75:12778 In-game Latency: 230ms LatencyTest District_Server Latency: 260ms-280ms /latencytest latency: Line 5: 30/08/2023 22:12:22,DISTRICT_SERVER,,,268,,Packet loss:,0,CCU:,19,22 Line 16: 30/08/2023 22:12:23,DISTRICT_SERVER,,,268,,Packet loss:,0,CCU:,19,22 Line 26: 30/08/2023 22:12:24,DISTRICT_SERVER,,,265,,Packet loss:,0,CCU:,19,22 Windows Trace Route (removed IP from first hop): Tracing route to 205.220.227.75 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms XXX.XXX.XX.X 2 2 ms 6 ms 6 ms lo0-100.CMDNNJ-VFTTP-325.verizon-gni.net [72.82.234.1] 3 6 ms 7 ms 7 ms B4325.NWRKNJ-LCR-22.verizon-gni.net [100.41.3.252] 4 * * * Request timed out. 5 14 ms 11 ms 12 ms customer.alter.net [152.193.0.114] 6 13 ms 12 ms 12 ms ae10.cr3-nyc3.ip4.gtt.net [89.149.140.182] 7 * * * Request timed out. 8 14 ms 11 ms 11 ms 205.220.227.75 Trace complete. Edited September 2, 2023 by BlatMan Added Port Information. Fixed typo 1 Quote Share this post Link to post Share on other sites
BlatMan 711 Posted September 4, 2023 I cannot connect to a district server with Cloudflare VPN or with USB tethering on mobile networks. I can login and select my character, but it gets stuck on "Connecting to district." All other network related services work. Web browser and Discord work. Other games work. Just more proof that the network problems are on LO's end. I bet you are blocking legit game traffic with your DDoS protection. Quote Share this post Link to post Share on other sites
