TL;DR; The Let’s Encrypt service was down, it wasn’t my fault. Check status here: https://letsencrypt.status.io/

Background

For background, with a Proxmox server you can setup an ACME account, and a Challenge plugin:

The account is simple enough to make, you just give it a name, enter your email address, and agree to Terms of Service, and that’s done. The GoDaddy challenge represents a GoDaddy DNS challenge plugin that I have already setup:

Then, within the Proxmox host, you should be able to order a certificate:

The problem is, that the request kept failing with the following:

Loading ACME account details
Placing ACME order
Order URL: https://acme-v02.api.letsencrypt.org/acme/order/0000000000/000000000000

Getting authorization details from ‘https://acme-v02.api.letsencrypt.org/acme/authz/0000000000/000000000000’
The validation for pm1.acme.net is pending!
[Mon Jul 21 19:41:33 EDT 2025] Adding record
[Mon Jul 21 19:41:44 EDT 2025] Added TXT record ‘azazazazaz-azazazaza-zazazazazazazazazazazaza’ for ‘_acme-challenge.pm1.acme.net’.
Add TXT record: _acme-challenge.pm1.acme.net
Sleeping 120 seconds to wait for TXT record propagation
Triggering validation
Sleeping for 5 seconds
Remove TXT record: _acme-challenge.pm1.acme.net
TASK ERROR: validating challenge ‘https://acme-v02.api.letsencrypt.org/acme/authz/0000000000/000000000000’ failed – status: invalid

I finally decided to check the url that was reported:

https://acme-v02.api.letsencrypt.org/acme/authz/0000000000/000000000000

Once there, I saw json similar to the following:

{
  "identifier": {
    "type": "dns",
    "value": "pm1.acme.net"
  },
  "status": "invalid",
  "expires": "2025-07-28T23:41:30Z",
  "challenges": [
    {
      "type": "dns-01",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall/0000000000/000000000000/gJaD_Q",
      "status": "invalid",
      "validated": "2025-07-21T23:43:44Z",
      "error": {
        "type": "urn:ietf:params:acme:error:serverInternal",
        "detail": "During secondary validation: Secondary validation RPC failed",
        "status": 500
      },
      "token": "azazazazazazazazazazazazazazazazazazazazazaz",
      "validationRecord": [
        {
          "hostname": "pm1.acme.net",
          "addressUsed": ""
        }
      ]
    }
  ]
}

And what sticks out here, is the status “500” in the “error” tag. This looks like a 50x server error. Sure enough, I went to check the status of the Let’s Encrypt services here:

https://letsencrypt.status.io

And there, I see that Let’s Encrypt is experiencing problems:

Luckily RackNerd provides VNC so you can see the boot console, and I saw this:

A start job is running for dev-disk and “1min 30s” is the key. We can see that there is a start job delaying reboots by 90 seconds. Looking into the journalctl data, I found these details:

[root@rn1 ~]# journalctl | grep 355fe974f1.device | grep timed

Mar 25 07:49:38 rockylinux8-minimal systemd[1]: dev-disk-by\x2duuid-2052dea6\x2d1b40\x2d4c49\x2d81e1\x2d4b355fe974f1.device: Job dev-disk-by\x2duuid-2052dea6\x2d1b40\x2d4c49\x2d81e1\x2d4b355fe974f1.device/start timed out.

Most of the articles I came across had to do with a bad or missing partition and using fstab to fix it, or changing DefaultTimeoutStartSec in the system.conf file. But these didn’t fit my issue exactly, and I wanted to know why this particular device (…355fe974f1.device) was the issue, when I had no device with this UID.

I took a look at the grub file:

[root@rn1 ~]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto resume=UUID=2052dea6-1b40-4c49-81e1-4b355fe974f1 net.ifnames=0 biosdevname=0 rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

The resume option was where the bad device ID was comming from. The resume option has to do with hiberntation mode, and since this was a server there should be no hibernation mode. So, I removed the ‘resume’ section from the file.

I then went to run grub-update, but it wasn’t found. Luckily, this is a simple shell script, so I recreated it:

vi /usr/sbin/grub-update

Add to this file:

#!/bin/sh
set -e
exec grub2-mkconfig -o /boot/grub2/grub.cfg "$@"

Then add executable permission to it:

chmod +x /usr/sbin/grub-update

And finally, run the new script, and reboot:

[root@rn1 ~]# /usr/sbin/grub-update
[root@rn1 ~]# reboot

The reboot now takes about 10 to 15 seconds, instead of a minute and a half.

Specifically, the EC2 VM had ssh access controlled via a private key, and not your typical password login. This allows you to restrict access not by giving someone the login and password, but by giving them a specific private key file.

To jump ahead, my mistake was that I placed the key file into a program files folder where special permissions were required. When mRemoteNG was loading that file, the permissions were actually too open, and putty would complain about it without showing a message in mRemoteNG. When I ran the manual ssh attempt, I saw the message:

Again, that’s because of where I placed the private key. If you place it in your documents folder, you will not have this issue. There are 4 key steps to follow here, some of which you may not need:

• Create an AWS Virtual Machine
• Convert the PEM file to PPK
• Setup Putty with the settings
• Setup mRemoteNG to use new Putty setting

Create an AWS Virtual Machine

In the AWS dashboard, select to create a new instance with Launch Instance:

We will select a RedHat Linux instance:

For this demo, select a Free tier instance:

Select Review and Launch and then Launch:

You will be prompted to create a key pair with the following dialog:

Enter a new key pair name (or reuse existing one. We decided on DemoVHost):

This will download a file with the Key pair name. Make sure to save this file somewhere safe, you will need it later:

Click the Launch Instance button. Back in the instances, your new VM should start soon:

In the instance details, make note of the DNS name:

You can now proceed to setup Putty settings.

You can not proceed to Convert the PEM file to PPK.

Convert the PEM file to PPK

You need to convert the PEM file we downloaded earlier into a PPK file.  Wherever you store this file, you need to make sure that you are the only person who has access permissions to it.  That may mean running puttygen as an administrator, and after this process changing the generated ppk file permissions so that only you have access to it (or, you can just store it in your documents folder somewhere).

Run the puttygen program.  I downloaded mine from https://www.putty.org/:

In Putty Generator, select Conversions / Import Key:

Select the PEM file you just downloaded from AWS:

In the next dialog, you can enter a passphrase, or leave it blank (you will get a warning), but click the Save private key button:

You can close the generator now, and proceed to Setup Putty with the settings.

Setup Putty with the settings

In mRemoteNG, select the Tools / Options menu choice:

Select the Advanced option, and click the Launch Putty button:

In Putty, go to the Connection / SSH / Auth item in the left bar, and select the Browse… button, and elect the PPK file that you generated previously:

Go to the Session item on the left panel, enter a new name in the Saved Sessions field, then click Save:

Confirm your new name is in the Saved Sessions area, and click Close.

You can now proceed to Setup mRemoteNG to use new Putty setting.

Setup mRemoteNG to use new Putty setting

In the configuration settings for your mRemoteNG connection, update the following:

• Set Hostname/IP to the DNS name noted earlier.
• Set Username to ec2-user
• Set Putty Session to the new settings you saved previously in putty

When you connect, you will get the normal putty message about the host key not being cached.  Just click Yes:

You should now be connected remotely:

I started to look for a program to download my PBase library of images and galleries, but couldn’t find any options or a feature in PBase to allow me to download all my images. So, I decided to write one in C# using WinForms.

The source code and a pre-built executable are all available on Github at https://github.com/MarioGiannini/MGPBaseDownloader

The user interface is pretty simple, and just contains a text box for the URL of the pbase user root gallery and a folder where to store the downloaded file. I’ll be honest and further state that there is basically zero error handling for things like selecting an invalid (or even blank) URL or destination folder.

The majority of the work is done by the PBaseDownloader class, and the ProcessUrlEx function specifically. The ProcessUrlEx is a recursive function that loads a web page using WebClient and then parses it using HtmlAgilityPack.HtmlDocument methods. When it identifies a thumnail it determines if the thumbnail leads to another gallery or to an image display page, and then handles it accordingly. When the thumbnail refers to another gallery, then ProcessUrlEx calls itself recursively to continue processing that sub-gallery.

The HtmlAgilityPack was added using the Nuget package manager.

Below is some pseudo code that demonstrates downloading the page and parsing it:

using (WebClient Client = new WebClient())
{
   String Page = Client.DownloadString(url);  // Download the web page
   HtmlAgilityPack.HtmlDocument Doc = new HtmlAgilityPack.HtmlDocument();
   Doc.LoadHtml(Page);
   ItemNodes = Doc.DocumentNode.SelectNodes("//td[@class='thumbnail']");
   if (ItemNodes == null)
      Results.AppendText( ">> No images or folders\r\n");
   else
   {
      foreach (HtmlNode Node in ItemNodes)
      {
         ItemLink = ExtractString( Node.InnerHtml, "href=");
      }
   }
}

The example above uses the SelectNodes method to iterate the td elements with a class of ‘thumbnail’ to help iterate the thumbnail elements that link to another gallery or to an image display.

Note: The ExtractString method in the sample code is a method in the PBaseDownloader class of the source code. It simply extracts the quoted portion of a string after an attribute like “href=”.

The pseudo code below demonstrates how to identify a JPG link (from PBase formatting) and download the file:

HtmlNode jpgnode = ImageDoc.DocumentNode.SelectSingleNode("//div[@id='imgdiv']");
if (jpgnode != null)
{
   String jpgLink = ExtractString(jpgnode.InnerHtml, "src=");
   Client.DownloadFile( jpgLink, SaveFile );
}

When iterating though the elements of a page, each element (an image page or a sub gallery) is assigned a simple sequential index, and that index becomes a part of the folder or image name. This is done to help maintain the order of the items as they appear. When the folder or filenames are created, a zero-padded prefix is used to insure identical ordering (despite in Explorer where folders and files are separated into groups). For example:

001-Cross Country
    001.jpg
    001.txt
    002.jpg
    002.txt
002-People
    001.jpg
    001.txt
    002.jpg
    002.txt
    003.jpg
    003.txt

The .txt file corresponding to the JPG files contain the title and EXIF data (if available) that were also parsed from the HTML page. In addition, each gallery folder contains the thumbnail of the folder (Thumbnail.jpg) and the description that the PBase user added for the gallery (Description.txt).

I have not tested it extensively, but it’s working today on the few PBase accounts I’ve tested it with. In time, PBase may change their formatting and this could potentially break the HTML parsing logic. This is a common downside to web-scraping programs.

In general, I frown on the process of web scraping, but sometimes when you want a convenient way to gather your own information from a site, it may be the only way of doing it. Obviously, such a program could be used to go and ‘steal’ data that’s already in the public, and I hope that nobody plans on that. I just thought this was interesting demonstration of some basic operations in web scraping with C#.

Rescued
Macrium saved my bacon by allowing me to go from a dead computer to operational status in about 15 minutes. I pulled the hard drive containing the drive image, put it into a SATA/USB enclosure I had, and used Macrium to launch the drive image as a virtual machine. It is very relieving to see your dead computer come back to life as a virtual machine. As my data was backed up nightly, I was able to continue working on projects with apps and IDEs as needed for the rest of the week, as I awaited a replacement motherboard.

Assisted
As my computer was used for various systems administration and software development with several IDEs, setting up a replacement can easily take 3 or 4 full days. I wanted to purchase the exact same motherboard in hopes that I could just swap it out and regain full operation. Unfortunately, the motherboard was discontinued, so I purchased the most similar motherboard I could from the same manufacturer. I also opted for a larger SSD as an upgrade since I was running low on the original. Macrium assisted by allowing me to clone the old SSD to the new one. Pretty standard stuff really, but Macrium just made it a simple process. With the new SSD installed, and the new motherboard, I booted up everything and was greeted with a fully operational version of my old PC.

Downside
The only real downside I had was that because the PC was using an OEM System Builder version of windows, replacing the motherboard meant I had to activate Windows. And of course, OEM System Builders don’t reactivate on a new computer, only on the first computer that they were activated with. So, I had to purchase another copy of Windows 10 OEM System Builder.

Summary
We all know the importance of backups. But, these Macrium drive images allowed me to go one step further. Thanks to the fact that I have some extra hardware at my disposal (like a spare PC and some SATA/USB enclosures), I was able to go from a dead PC to normal in a few minutes. Granted, it was a virtual machine of my PC, and running slower than normal, but at least it allowed me to get my work done when I needed. I am now a paying member of the Macrium fan club.

Proof of Value: I have a process that uses tesseract to convert a bunch of JPGs into PDF files, then pdfunite to join those into a single PDF.  Working on a single file took 81 seconds, but working with 4 threads (on a 4-core VM), I was able to create 4 identical PDFs in about 104 seconds.  That’s a 400% increase in throughput with only a 28% increase in time. As the processing power and core count increases, we can expect improved throughput with additional threads.

I am going to use the number of cores on the computer to determine how many simultaneous threads my script should be limited to.  This is absolutely a ‘best guess’ scenario, and you may prefer to run two or more times as many cores on the computer.  I am making no attempt to spread out the threads or identify how busy a core is (the OS should be doing that).  In reality, multiple threads are already running on all the cores from different processes.

The script will be run on a cron job, every minute.  Once run, it will run for 50 years, or until the server is shut down.  However, it will use an exclusive lock to make sure that only one instance of the script is running at a time (except for the child threads of course).

The demo script will scan for a set of files, then fork itself (once for each thread) to sleep for several random seconds and then delete the file.

The script starts with a standard header and the declaration of a temp folder for our files:

#!/bin/bash
#define our own tmp folder name, should be unique
tmpdir='/tmp/mtdemo'

Then, it tests to see if we got a command line argument.  If we did, it means the script was invoked with information to do some job in the child thread.  In our example, we’re just sleeping and deleting some flag and data files to simulate doing real work:

if [ "$1" != "" ]; then
    # If we got an argument, we were told to do something. This is where the child works:
    RANDOM=$$
    R=$(($(($RANDOM%8))+2))
    echo "Child thread in slot $2 started with $1.  Simulating processing with sleep $R"
    sleep "$R"
    # clean up the files that control synchronization
    echo "Child thread in slot $2 finished with $1."
    trap "rm -f ~/$1; rm -f $tmpdir/$1; rm -f $tmpdir/slot.$2" exit
 else

This demo code will show you what ‘slot’ the thread is running in, and the file it’s processing, along with start and stop messages.  Note how we’re using trap to clean up any orphaned files in case of an error.

The following code snippet shows how to insure we are running only a single copy of the main file monitor that will invoke a child thread as needed. The logic for identifying a free ‘slot’ and a file to be processed will be located within the do … done block below:

(
    # Wait for lock on /tmp/mtdemo_exclusivelock (fd 200) for 1 second
    flock -x -w 1 200 || exit 1
    end=$(date -ud "50 year" +%s)
    while [[ $(date -u +%s) -le $end ]]
    do
       # If needed, invoke ourselves as a forked process (&) with some parameters:
       if something
          $0 "$key" "$slot"&
       fi
    done
 ) 200>"/tmp/mtdemo_exclusivelock"

The parens will create a code block that will be identified as a file handle of 200 on the file /tmp/mtdemo_exclusivelock.  Inside that block, it uses flock to attempt to lock the file, and if the lock can’t be obtained then exit is invoked, otherwise it proceeds into the block.  The end variable and the while loop create a loop that will execute for 50 years (or until the server shuts down).

In order to identify a free ‘slot’ for threading, we will first get the number of cores with the following:

cpus=$(nproc)

Then, we use files to try and identify free slots.  For example, let’s say we have 4 cores.  We will test to see if the file slot.1 exists.  If it does, then that means a thread is already running on that slot, and we proceed to test for slot.2.  If it doesn’t exist, then we’re going to start a thread with ‘1’ as a parameter, and the child thread will know to create slot.1 at the start of it’s processing, and delete it at the end.  The child process will create and destroy the slot.1 file as a flag.

Here’s the loop that tests for a free slot:

slot=0
for i in $(seq 1 $cpus); do
   if [ ! -f "$tmpdir/slot.$i" ]; then
      slot=$i
   fi
done

At the end of the loop above, if slot is > 0, then a free slot was found.  The bash test would be:

if [ $slot -gt 0 ]; then
   ... Do something
fi

Now, the Do something comment above is going to look for files to process.  For our demo, we’re just going to look for .tmp files in our home folder.  The ‘Processing’ that will be done for each file is the same sleep and file delete with messages discussed above.  This loop shows how we look for .tmp files to ‘process’:

for d in ~/*.tmp ; do
   [ -f "$d" ] || continue
   key=$(basename -- "$d")
   if [ ! -f "$tmpdir/$key" ]; then
      echo "Working $d" > "$tmpdir/slot.$slot"
      echo "Working slot $slot" > "$tmpdir/$key"
      $0 "$key" "$slot"&
      break;
   fi
done

Here’s a key component:  When a file to process is found, the script runs itself in a fork (&) passing to it the name of the file to process.  If no files are found, no fork takes place.

In order to test the script, let’s first make several bogus .tmp files in our home folder (make twice as many as nproc reports, which is 4 in my case):

echo Bogus File > 1.tmp
echo Bogus File > 2.tmp
echo Bogus File > 3.tmp
echo Bogus File > 4.tmp
echo Bogus File > 5.tmp
echo Bogus File > 6.tmp
echo Bogus File > 7.tmp
echo Bogus File > 8.tmp

With the script in the home folder, run it.  The results should look like this:

[root@fpsrv5 ~]# ./demo.sh
Cores found: 4
Child thread in slot 4 started with 1.tmp.  Simulating processing with sleep 9
Child thread in slot 3 started with 2.tmp.  Simulating processing with sleep 3
Child thread in slot 2 started with 3.tmp.  Simulating processing with sleep 4
Child thread in slot 1 started with 4.tmp.  Simulating processing with sleep 4
Child thread in slot 3 finished with 2.tmp.
Child thread in slot 3 started with 5.tmp.  Simulating processing with sleep 7
Child thread in slot 2 finished with 3.tmp.
Child thread in slot 1 finished with 4.tmp.
Child thread in slot 2 started with 6.tmp.  Simulating processing with sleep 7
Child thread in slot 1 started with 7.tmp.  Simulating processing with sleep 5
Child thread in slot 4 finished with 1.tmp.
Child thread in slot 4 started with 8.tmp.  Simulating processing with sleep 4
Child thread in slot 1 finished with 7.tmp.
Child thread in slot 3 finished with 5.tmp.
Child thread in slot 2 finished with 6.tmp.
Child thread in slot 4 finished with 8.tmp.

So we can see the slots 4, 3, 2, and 1 were child threads for files 1.tmp, 2.tmp, 3.tmp, and 4.tmp respectively.  Note how slot 3 started on 2.tmp with a 3 second sleep.  Because of that, it finished first, and you can see that slot 3 then started processing file 5.tmp (with a sleep of 7 seconds).

To complete the practical use of this script, we would remove any of the information ‘echo’ statements we didn’t want, and add it to a cron job so that it would run all the time.

Entire Script

#!/bin/bash
#define our own tmp folder name, should be unique
tmpdir='/tmp/mtdemo'
 
if [ "$1" != "" ]; then
   # If we got an argument, we were told to do something. This is where the child works:
   RANDOM=$$
   R=$(($(($RANDOM%8))+2))
   echo "Child thread in slot $2 started with $1.  Simulating processing with sleep $R"
   sleep "$R"
   # clean up the files that control synchronization
   echo "Child thread in slot $2 finished with $1."
   trap "rm -f ~/$1; rm -f $tmpdir/$1; rm -f $tmpdir/slot.$2" exit
 else
   #execute the following only if we get an exclusive lock.  The first run will hold the lock for 50 years.
   (
      # Wait for lock on  /tmp/mtdemo_exclusivelock (fd 200) for 1 second
      flock -x -w 1 200 || exit 1
 
      #Start by wiping out the synch files, incase we were rebooted mid-job
      rm -r -f "$tmpdir"
      mkdir -p "$tmpdir"
 
      #determine how many processors we have to control thread count
      cpus=$(nproc)
      echo "Cores found: $cpus"
      end=$(date -ud "50 year" +%s)
      while [[ $(date -u +%s) -le $end ]]
      do
         #Locate a free slot for a CPU based on file existance
         slot=0
         for i in $(seq 1 $cpus); do
            if [ ! -f "$tmpdir/slot.$i" ]; then
               slot=$i
            fi
         done
 
         #if we got a free slot, then do we have a file that needs processing?
         if [ $slot -gt 0 ]; then
            for d in ~/*.tmp ; do
               [ -f "$d" ] || continue
               key=$(basename -- "$d")
               if [ ! -f "$tmpdir/$key" ]; then
                  echo "Working $d" > "$tmpdir/slot.$slot"
                  echo "Working slot $slot" > "$tmpdir/$key"
                  $0 "$key" "$slot"&
                  break;
               fi
            done
         fi
      done
   ) 200>"/tmp/mtdemo_exclusivelock"
fi

I increased the per-user limits by editing the /etc/security/limits.conf file and adding the following to the end of the file:

*         hard    nofile      500000 
*         soft    nofile      500000 
root      hard    nofile      500000 
root      soft    nofile      500000 

I rebooted, because I could, and then verified the maximum open file count using the ulimit command:

#su - apache -c 'ulimit -aHS' -s '/bin/bash' | grep 'open files'
open files                      (-n) 500000 

Attempting to submit the files for processing again worked without a problem.

After using yum install ImageMagick, when I tried to convert with ICC files like below;

convert -verbose -density 300 -colorspace rgb SaleCat.pdf[10-20] -profile eciCMYK.icc -profile eciRGB_v2.icc -quality 100 y-%04d.jpg

I got this message:

convert: delegate library support not built-in `SaleCat.pdf' (LCMS) @ warning/profile.c/ProfileImage/565.

I ran the following to see the delegates installed:

convert -list configure | grep -i "delegates"

The results showed the lcms delegate wasn’t setup:

DELEGATES     bzlib fontconfig freetype gs jpeg jng jp2 lzma openexr pango png rsvg tiff x11 xml wmf zlib

I verified that lcms was already installed with yum install lcms2 (it was).

So it appears that I needed to rebuild ImageMagick from source to include the lcms2 delegate. I removed Image magic with “yum remove ImageMagick” first.

I made sure gcc was installed:

yum install gcc

Then downloaded and configured the ImageMagick source:

wget https://imagemagick.org/download/ImageMagick.tar.gz
tar xvfz ImageMagick.tar.gz
cd ImageMagick-*
./configure  --with-lcms=yes

The config should show that lcms was now enabled, but it wasn’t:

LCMS              --with-lcms=yes             no

So I looked at config.log to see the details:

     configure:31839: checking for lcms2 >= 2.0.0
     configure:31846: $PKG_CONFIG --exists --print-errors "lcms2 >= 2.0.0"
     Package lcms2 was not found in the pkg-config search path.
     Perhaps you should add the directory containing `lcms2.pc'
     to the PKG_CONFIG_PATH environment variable
     No package 'lcms2' found

I made sure that lcms2 devel was installed:

yum install lcms2-devel

I tried again to build:

./configure  --with-lcms=yes

And this time the config looked good:

LCMS              --with-lcms=yes             yes

And so I built and installed ImageMagick from the source:

make
make install
make clean

Now I can run convert from /usr/local/bin/convert and process with icc files.

The following articles cover the details of the endeavor:

• Copying ESXi 6.5 VM to ProxMox 5.2
• Increase VM Disk and Consolidate SQL database

Overall, ProxMox is a very slick product.  I’ve used it on and off for various test home labs, and opted to use it at work on Lab computers that no longer support VMWare ESXi hosts.  Installation was pretty painless, though understanding the various storage options got me a bit confused. 

The worst problem I had with ProxMox was that the web-based Console used to access the VM wouldn’t work in IE 11.  I thought there was some major issue as the web-based Console would never connect to the VM.  But once I logged in with Chrome, I could access the VM perfectly fine through the exact same ‘console’ option.

In one experiment, I did go for a high-availability with host-to-host cross replication and switch-over to a single host after powering down one of the hosts.  After some trial and error I got things to work, but it’s not as smooth as VMWares replication features.

Cost
Both ProxMox and vSphere/ESXi are available for free. ESXi free is a crippled version of the product, specifically in the area of templates, backups and High Availability features. You can purchase a perpetual license of vSphere for approximately $5,654, which breaks down to about 4,470 for a perpetual license key, and 1,184 for 1 year of support which covers up to 3 hosts and 3 CPUs per host. By comparison, the ProxMox free version primarily just comes with a nag screen at web login. The ProxMox support subscription varies in price, but they charge between $796 and $74.90 per CPU and per host, which can wind up costing more than VMWare based on your needs.

For example, in the past year I have come to rely on VMWare’s support which has been excellent. Based on the past year of VMWare support, we would need to purchase the $398 ProxMox subscription, which would cost $1,592 per year for our 3 hosts with 4 CPUs. That’s $408 more than the VMWare support.

Conclusion
Both VMWare and ProxMox have handled the situations I threw at them. Both have worked wonderfully, but both have experienced hiccups along the way. None of those hiccups have required a complete system restore, only various changes in settings. The trade-offs come in the form of high availability, backups, and support. Converting a VM from ESXi to ProxMox was a painless endeavor.

Generally speaking, VMWare is the king of the virtualization land. Its price is a bit steep for small organizations, but it may not be that far off from a similar ProxMox installation. Relying on your own technical expertise can save you money.

If I were faced with a dirt-cheap scenario, I would probably go with a free ESXi setup and rely on open source backup software such as Macrium Reflect. If there’s a second hardware host available, I would split the VMs between the 2 hosts, and using scheduled scripting to stop the VMs and cross-copy them between the 2 hosts. Not exactly High Availability, but still workable in case of hardware failure.

Once a budget is introduced, I would still push for VMWare, despite the higher initial investment. The assumption is that there are 2 host machines with cross-replication of VMs for High Availability. The VMWare support has been excellent in our production environment, and I have not used ProxMox support because we don’t have a subscription.

Note that if you are limited to older hardware, then ProxMox may be your only solution. VMWare updates eventually stop supporting older processors. Hardware that works fine with ProxMox or an older VMWare installation may not be usable with a current version of VMWare..

Our solution is VMWare with production support running on 2 cross-replicated servers and all VMs copied to a disaster recovery server. If 1 host physically dies, we can be back up and running in 5 minutes. If 2 hosts die, it can take a few hours, but the disaster recovery machine can be brought online. We also use ProxMox free on 2 lab servers. They allow me to run several low-priority VMs and test VMs and test changes.

Update: Emergency Mode
After completing my experiment, I rebooted my ProxMox server after it had been running for more than 2 months. Except, it kept booting into Emergency mode. Networking was working as I could ping the host server, but not much else.

I went to the console, logged in and manually launched the sshd service, and was able to remote in from my desktop. I then looked at journalctl -xb looking for red flags, and I found red/error reports regarding the tmp volume.

After further review, I found that I had created a logical volume and made a permanant mount line in etc/fstabs, but later removed the logical volume without removing the line from fstabs. Trying to mount the non-existing logical volume was my cause. Once I manually removed that line and rebooted, everything was back to normal.

Preparing the VM on VMWare ESXi
Remove VMWare tools from the VM, through the VM guest OS.

Download and run mergeide.zip from https://pve.proxmox.com/wiki/File:Mergeide.zip

Make sure Atapi.sys, Intelide.sys, Pciide.sys, and Pciidex.sys are in the %SystemRoot%\System32\Drivers folder. If any are missing they can be extracted from %SystemRoot%\Driver Cache\I386\Driver.cab which can be opened in Windows file Exlorer like a directory and then the missing files can be copied out.

Shutdown the VM.

Creating temporary space for the VMDK files on ProxMox host
Because my VM totals about 300GB, I need to allocate enoughspace to store and convert the VMWare VMDK files.  My default install of ProxMox left a 98GB ‘Directory’ type storage named ‘local’ (too small for this process) and a 5TB ‘LVM-Thin’ type storage named ‘local-lvm’.

Use lvcreate to create a new logical volume named tmp, 900GB in size, in the pre-existing pve volume group:

lvcreate -n tmp -L 900G pve

Create a file structure on the logical volume:

mkfs.ext4 /dev/pve/tmp

Add now mount the volume:

mkdir /var/lib/tmp
mount -t ext4 /dev/pve/tmp /var/lib/tmp

Note: If you wanted to make tmp permanent, run the following:

echo '/dev/pve/tmp /var/lib/tmp ext4 defaults 0 2' >> /etc/fstab

NOTE: I added the above line for a permanent mount, but then removed the logical volume without removing the line from fstab. The result was that ProxMox then booted into Emergency mode until I removed that line).

Create the new VM onProxMox
We need to create a new VM with that has the same number of hard drives as the original VMWare VM, but each should have slightly more space allocated.  Our VM started with 3 drives, 20, 165, and 200 GB in size, so we are going to create a ProxMox VM with 21, 166, and 201GB in size.

We opted for similar CPU settings and identical memorysettings for the ProxMox machine as from the VMWare VM.  Once the initial VM was created with a 21GBdrive, we went back to the Hardware tab for the ProxMox VM and added the 166GB and 201GB hard drives.

Make note of your new VM’s ID number, it will be needed in the next step.  Our example will be 102.

Copy and convert theVMDK files
There were several options available to get the VMDK files.  I could use pscp or FTP on my Windows workstation to copy files from the ESXi host to my PC and then again to copy to the ProxMox host.  Instead, I opted to use the scp command on the ProxMox host to copy the files directly.  Make sure that the VM is powered down and snapshots are consolidated before copying.

scp root@ESX1:/vmfs/volumes/518eblah-a4bablah-blah-69ee0a22blah/Anakin/*-flat.vmdk
/var/lib/tmp

Now, with all our VMDK files, we want to convert them into the virtual disks that were created with our new ProxMox VM.  If you list the files in /dev/pve, you should see the virtual disk files (again, our demo uses VM ID 102):

Using the qemu-img tool on the ProxMox host, we will convert the VMDK into a raw file format, and store it directly into the virtual disks already created for the ProxMox VM (once for each virtual disk).  Note that this can take some time for large virtual disks:

root@lab2:/#qemu-img convert /var/lib/tmp/ANAKIN-flat.vmdk -O raw /dev/pve/vm-102-disk-0
root@lab2:/#qemu-img convert /var/lib/tmp/ANAKIN_1-flat.vmdk -O raw /dev/pve/vm-102-disk-1
root@lab2:/#qemu-img convert /var/lib/tmp/ANAKIN_2-flat.vmdk -O raw /dev/pve/vm-102-disk-2

That’s it.  You should now be able to start the VM on the ProxMox host.

Removing Temporary storage
When done, we can remove the temporary storage we created.

Unmount it and then remove it with the following:

umount /var/lib/tmp
lvremove pve/tmp

Make sure that you remove any entries in etc/fstab if you added them for permanent mounting of the logical volume.