7 Replies - 557 Views - Last Post: 01 June 2017 - 06:13 PM

#1 e_i_pi  Icon User is offline

  • = -1
  • member icon

Reputation: 879
  • View blog
  • Posts: 1,893
  • Joined: 30-January 09

Advice on servers, I think we're getting shafted

Posted 17 May 2017 - 06:14 PM

At my new work (as of 2 months), we're running on an old oooold Debian 6 server through our web hosts, and are pegged to PHP5.4 and OpenSSL0.98 (yes, the heartbleed one). This has obviously raised red flags with me. I have commissioned a new server that has SSD, Ubuntu 16.04, more RAM, and better CPUs. The problem is, when I run PHP pages on the new server it is very slow, 300% slower than the old one. As the old server is production, I can only stress test is so much, as I don't want to interrupt business as usual, so I'm doing testing using the following PHP benchmark script:

https://github.com/odan/benchmark-php

The old server on PHP5.4 as a mod runs the script in around 4s.
The new server on PHP5.6 as a mod runs the script in around 16s.
The new server on PHP5.6 as CGI runs the script in around 16s.
The new server on PHP7.1 as CGI with OpCache enabled runs the script in around 6s.

So, to try to get to the bottom of it, I looked at the CPUs on the server, by running the command line command lscpu:
OLD SERVER
Architecture:          x86_64
CPU op-mode(s):        64-bit
CPU(s):                5
Thread(s) per core:    1
Core(s) per socket:    1
CPU socket(s):         5
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 45
Stepping:              7
CPU MHz:               1999.999
Hypervisor vendor:     Xen
Virtualization type:   para
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              15360K



NEW SERVER
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    4
Core(s) per socket:    1
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 44
Model name:            Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
Stepping:              2
CPU MHz:               2666.846
BogoMIPS:              5333.69
Hypervisor vendor:     Xen
Virtualization type:   para
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              12288K
NUMA node0 CPU(s):     0-3
Flags:                 fpu de tsc msr pae cx8 apic sep cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm epb dtherm arat



I also looked at the memory, using the command line command free -m:
OLD SERVER
             total       used       free     shared    buffers     cached
Mem:          2040       1724        316          0        203        878
-/+ buffers/cache:        642       1398
Swap:         2047         29       2018



NEW SERVER
              total        used        free      shared  buff/cache   available
Mem:           3195         208        2292          62         695        2878
Swap:             0           0           0



It looks to me that the new server is much more powerful than the old server. While it has less CPUs (4 vs 5), the speed is higher (2.67Mhz v 2.00MHz), which should counterbalance that. It also has more memory (though a single PHP thread is limited in memory -
running mem_peak_usage() on each benchmark yields 786432B usage on both servers). The only other difference I can see that may be tangible is that the new server has op modes of 32 and 64 bit, whereas the old server is purely 64 bit.

Can someone help me brainstorm why this new server is slower, and how to gather information to compare the two servers as evenly as possible?

Is This A Good Question/Topic? 0
  • +

Replies To: Advice on servers, I think we're getting shafted

#2 BetaWar  Icon User is offline

  • #include "soul.h"
  • member icon

Reputation: 1462
  • View blog
  • Posts: 8,159
  • Joined: 07-September 06

Re: Advice on servers, I think we're getting shafted

Posted 17 May 2017 - 06:47 PM

Well, the lscpu output for the old server doesn't seem to have the flags line in it, but I do notice that the new server is a VM (you can tell because of the flags line containing hypervisor in it, which Linux only adds if it is running in a VM).

So, if I had to guess based solely off of that, I would say that the hypervisor is overprovisioned, and while you are theoretically more powerful (and you definitely should be based on raw specs) but are being throttled or contending for resources with other VMs on the hypervisor.
Was This Post Helpful? 1
  • +
  • -

#3 ArtificialSoldier  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 1769
  • View blog
  • Posts: 5,640
  • Joined: 15-January 14

Re: Advice on servers, I think we're getting shafted

Posted 17 May 2017 - 06:50 PM

Something is wonky with newserver. 16 seconds for that benchmark (without the MySQL tests, anyway) is ridiculous. The thing about that benchmark is that it is almost exclusively using the CPU. There's no disk I/O, so rule that out. Just for comparison:

[[email protected] ~]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                48
On-line CPU(s) list:   0-47
Thread(s) per core:    2
Core(s) per socket:    12
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz
Stepping:              2
CPU MHz:               1200.000
BogoMIPS:              5004.13
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              30720K
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47
[[email protected] ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:         128658       14730        4314        1633      109612      111302
Swap:         63999        1863       62136



Quote

benchmark
math 0.892
string 0.908
loops 0.035
ifelse 0.046
total 1.881


The CPU cores shouldn't matter, I don't think that a single PHP process will ever use more than one core. You may want to have a terminal window open and run top before you start the benchmark, I doubt you would see more than 100% CPU usage for the process running the benchmark (that would be full utilization of 1 core). If your load average in top is approaching the number of CPU cores then maybe you have other processes running which are tying it up, but you should see those listed in top also.
Was This Post Helpful? 1
  • +
  • -

#4 e_i_pi  Icon User is offline

  • = -1
  • member icon

Reputation: 879
  • View blog
  • Posts: 1,893
  • Joined: 30-January 09

Re: Advice on servers, I think we're getting shafted

Posted 17 May 2017 - 06:59 PM

I've run top on the old server, and htop on the new server. The benchmark script maxes out one CPU, going between about 85%-100% at all times, mostly up at 100%. PHP is single-threaded AFAIK, and while my server admin game isn't strong, I'd expect it to hit just one CPU because of that.

Both are VMs by the way, but I'd say we're the last of the last on the old VM, given they no longer provision Debian 6 VMs, only Debian 7 onwards. For some reason flags doesn't show up on the old server - could be an issue with it being an (old) Debian 6 version of lscpu.

I've put a ticket in with them - it smells to me like the machine our VM is on is overstretched. Thank you BetaWar for the term "hypervisor", that's what I'm looking for. It's no good if I say "your server is slow" without backing it up with lingo. I know it's slow - they know it's slow - if I start bandying about terms like I'm a SysAdmin or NetworkEng, I'm more likely to get traction.
Was This Post Helpful? 0
  • +
  • -

#5 Atli  Icon User is offline

  • Enhance Your Calm
  • member icon

Reputation: 4237
  • View blog
  • Posts: 7,213
  • Joined: 08-June 10

Re: Advice on servers, I think we're getting shafted

Posted 18 May 2017 - 01:33 AM

It's probably SOP for hosting providers to set up new VMs on a crowded/throttled system, hoping nobody notices, and only doing something about it if they get a complaint. Odds are that most of their customers never realize they aren't getting the full power they are paying for.
Was This Post Helpful? 1
  • +
  • -

#6 ArtificialSoldier  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 1769
  • View blog
  • Posts: 5,640
  • Joined: 15-January 14

Re: Advice on servers, I think we're getting shafted

Posted 18 May 2017 - 02:25 PM

Yeah if that's a virtualized machine then there's a lot going on behind the scenes that you can't really test for.

That server I posted is a dedicated server, although we do have a client actively using it, but I increase the loops in that benchmark 10x so I could see things in top and it did use 100% of a core to run that. The benchmark (without MySQL) has no I/O at all, so if it's not performing well then that's a CPU issue.
Was This Post Helpful? 1
  • +
  • -

#7 e_i_pi  Icon User is offline

  • = -1
  • member icon

Reputation: 879
  • View blog
  • Posts: 1,893
  • Joined: 30-January 09

Re: Advice on servers, I think we're getting shafted

Posted 18 May 2017 - 05:02 PM

I agree that hosts likely over-provision and hope clients don't notice. I've been trying to convince the boss to go with a big provider (AWS or Linode have been my suggestions), but I think he's got a long standing working relationship with this other mob, who are a local business. I'm all for supporting local business, but if they can't meet our needs, I don't see why we should hire them. You don't hire a slacker just because they live nearby.
Was This Post Helpful? 0
  • +
  • -

#8 e_i_pi  Icon User is offline

  • = -1
  • member icon

Reputation: 879
  • View blog
  • Posts: 1,893
  • Joined: 30-January 09

Re: Advice on servers, I think we're getting shafted

Posted 01 June 2017 - 06:13 PM

Bump update :)

We've decided to go with AWS instead. Benchmarking on even the micro AWS server (i.e. the free one) shows that it's 2x faster than our current server, and 8x faster than Crucial's "super powerful server" they provided us.

I ran further benchmarks on that server, using sysbench. CPU was never the problem, it's RAM. Check out these stats:
    Memory Read Ops/s
    Linode 2048 - 3.2MM
    AWS micro - 1.4MM
    Crucial Blaze - 0.2MM


So it looks like memory was a massive problem. There's 3GB memory in the machine, but with those speeds I wouldn't be surprised if it's as old as DDR. Who knows what their caper is, but we're ditching them.
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1