In data analysis, the tuning parameters are working parameters in our methods that need to be tuned. The main difference from tuning parameters to (ordinary) parameters is that the usual parameters are of research interests or have particular physical meanings (like the mass of sun, the average salary for people living in Pittsburgh). In contrast, tuning parameters are created by the method we use.
For instance, in histogram, a tuning parameter is the bin-size. It is not our research interest but is closely related to our visualization of data.
Something interesting I found is how people in different fields pick the tuning parameters. It turns out that the scientists, engineers and statisticians have difference preferences.
- For scientists, they prefer to pick the tuning parameters according to the knowledge on that parameters, especially through the unit and how it is related to other meaningful quantities.
- For engineers, they always construct an objective function and consider a range of tuning parameters. Then they perform a search for the tuning parameters over the whole range and pick the ones that minimize the objective function.
- For statisticians, we prefer a theoretical analysis on how the tuning parameters are related to our evaluation of estimates (or objective). Then we derive the optimal tuning parameters (possibly be a function of data) based on the theoretical behavior.
However, the above is only the preference. In practice, people use the mixed strategies. For instance, many statisticians use theoretical analysis to find the possible optimal value then select a small range near the optimum and perform a search over the whole range. Similarly, many engineers apply the same way to save time. In scientific study, many researchers also tune the parameters over a scientifically reasonable range.
I’m not saying any method is better than the others. I just find this phenomena very interesting. I think the different preferences come from the difference in value systems across fields.
It is really nice to work with people from different disciplines; you can see the different core values for each field.