Whisk together all pickling ingredients except the onion until the sugar dissolves. Add the onion and let sit at room temperature for 1 hour. Drain, reserving the onion and the liquid. This step can be done up to 2 weeks ahead of time.
Cube potatoes and boil in lightly salted water until fork tender, about 8-10 minutes depending on their size. Drain and let dry slightly.
Meanwhile, peel and cut the hard boiled eggs. Thinly slice half the jalapeño and mince the other half. Slice the celery into bite sized pieces.
In a mixing bowl, combine the mustard, mayonnaise, and 1 tbs pickling juice. Add salt and pepper if desired, as well as an extra pinch of cayenne if you like it spicy!
In a large mixing bowl, combine the softened potatoes, eggs, jalapeño, celery, and onion. Pour in the dressing and gently fold to evenly coat.
Refrigerate at least one hour before serving, but the longer it sits, the more the flavors develop. This can keep in the fridge for a few days.